Contemporary Cargo Cults
The observations in this article about the use of computers are a bit out of date now (the article was first published in 1996) but the observations about evaluation are still pertinent.Cargo cults supposedly originated in Melanesia about 75 years ago, but the type of thinking which is the foundation of cargo cults has long been characteristic of what we like to consider the most sophisticated society on the face of the earth. In fact, North America is currently ruled by cargo cults.
The cargo cult is founded on a familiar, and popular, bit of fallacious reasoning: post hoc ergo propter hoc. The residents of Papua, Yaliwan, Vanuatu and other places noticed that when the colonial occupiers built wharves and airstrips, the wharves and airstrips were soon visited by ships and airplanes which delivered cargoes of goods. They concluded that the ships and airplanes arrived as a consequence of the building of the wharves and airstrips, so they built their own wharves and airstrips in the expectation of receiving their own cargoes.
This reasoning seems naive to us, since we already know that the correct chain of reasoning is the reverse: the wharves and airstrips were built because ships and airplanes were going to be arriving. However, in dealing with that which is new to us, just as wharves and airstrips were new to Melanesians, we draw exactly the same conclusions about it.
An example of this type of thinking is the shopping mall. The first shopping malls were tremendously profitable. As a result, people concluded that if they too built shopping malls, they too would make tremendous profits. As a result, we soon had a shopping mall on nearly every block. Retail space expanded enormously, and eventually the retail industry collapsed under its own weight. Retail chains closed and downsized, their unsustainable outlets either replaced by dollar stores or left permanently empty.
Today, Wellington Square in London, Ontario, the first shopping mall in this country, bloated to three or four times its original size and given a trendier name, lies dying in the centre of a downtown which it has already killed. On Saturdays the retail staff in the few stores that are open outnumber the customers. In Toronto, the Eaton Centre has transformed the retail neighbourhood around it into a vast bazaar of dollar stores, adult book stores, and fast food franchises.
The most powerful of our cargo cults is the cult of the computer. One cannot deny the obvious benefits that the computer has produced in many organizations. One also cannot deny, though, that many organizations, impressed by the success of computerization elsewhere, plunged madly into computerization as if the mere presence of computers in their offices would be enough to improve productivity. You don't have to travel very far to find a large organization or office in which every desk is adorned with a powerful personal computer, most of the computers unused at any given time, and the rest used only for typing.
On many desks in many organizations the personal computer serves the same function as the cargo cults' wharves and airstrips. The purchase of computers is often little more than a ritual intended to bring productivity and quality to the organization by some magic means.
Even when it is used for undertakings which could not be done more effectively without a computer, the computer often is more of a hindrance than a help, simply because it is considered more as a talisman than as the appliance which it is. For example, computers have been widely used to amass great quantities of data in databases which then prove to be unanalyzable or unreliable. We reason that since other people have increased the efficiency of their organizations by constructing databases, we will also increase the efficiency of our operations if we construct databases. We might as plausibly argue that since Newton had brilliant ideas when he sat under apple trees, we should all sit under apple trees.
The computer can help carry out intelligent plans more efficiently, but it cannot do much with wishful thinking, apart from dressing it up with pretty graphics. Nevertheless, even when wishful thinking goes disastrously awry, people seem to think that everything will turn out all right if they just get the next upgrade.
Another important contemporary cargo cult is the cult of evaluation. The reported success of ISO standards, total quality management, and continuous quality improvement has provoked a rush to evaluate. This, of course, could be an extremely beneficial development. However, the danger is that evaluation will become another sort of cargo cult.
This danger arises because of the strict and artificial separation of functions characteristic of the modern organization. McLuhan (1964) argued plausibly that this form of organization is the result of a long history of technological changes which have permitted power to be extended over greater and greater areas and which have encouraged sequential analysis of social functions.
Whatever the reason, modern organizations in both the private and public sectors are organizations of power rather than of functions (Saul, 1992). Functions are merely manifestations of power, and therefore their performance must be strictly regulated and narrowly assigned. In the modern organization, functions which anyone might perform are reserved for experts. In particular, reasoning is reserved for the managerial classes, as anyone knows who has served in any sort of professional capacity in a bureaucracy.
As a result, evaluation is often conceived of as a management tool or support. For example, Reavy, Littell, Gonda, and O'Neill write that:Timely access by decisionmakers to reliable and valid data about program activities is crucial for effective internal program evaluation. Clearly, then, one of the first tasks of the internal evaluator is to help managers determine what data they need and what data they can get. (1993, p. 102)Just as clearly, the role of the evaluator, according to Reavy, Littell, Gonda, and O'Neill, is not to think or to make decisions, but merely to bring grist to the managerial mills. The facts that decisionmaking is what evaluation is all about, and that professional evaluators are, or at least should be, people with specialist training in decisionmaking, are clearly irrelevant. The syllogism is of extreme clarity and elegance:The reasoning is so clear and so elegant that it seems almost a truism. Bureaucracies are organizations of power, so all intellectual functions are performed by the people at the top. Our conceptions of accountability are so influenced by this conception of bureaucracy that any other assignment of decisionmaking power seems not only inappropriate but unethical to boot.1. Only managers make decisions.
2. Evaluators are not managers.
3. Therefore, evaluators do not make decisions.
McLuhan (1964) argued that electronic technology would threaten this type of organization, because of its decentralizing character. Anyone who is familiar with the struggle of computer services departments against the desktop computer has seen an example of this threat.
Desktop computers allow computer functions to be performed without the careful supervision of the central computer services department, and computer services departments quickly realized that. In many organizations, the use of desktop computers has been so restricted that they are of little use at all.
An example of this restriction is the anorexic computer, which is found in many organizations. This is a desktop computer with a huge hard drive and amazing clock speed, but with next to no software. Typically, the software will consist of a word processor and e-mail. The e-mail software of course lets the computer services department get its fingers into every desktop computer in the joint. Other types of software may be allowed on some computers, if the permission of the computer services department, which regulates the licensing agreements, is obtained. Often the software will be installed but staff not trained to use it. In this way centralized functions like data management remain centralized, and by remaining centralized they remain under the watchful eye of the appropriate bureaucratic satrap.
As a result, the computer, a useful tool, has been converted into a process, called computerization. Computerization often amounts to little more than the dispensing of computers to the faithful as signs of the good intentions of management.
By the reverse process, evaluation, a useful process, has been converted into a tool. Management maintains its monopoly on intellectual functions by persuading evaluation staff that theirs is not to reason why, theirs is but to administer questionnaires.
Of course, evaluation staff have their own reasons for accepting this model of evaluation. If they were to be responsible for their own decisions, they would be in perilous seas indeed. The problem is that their evaluations could be evaluated, and their job security thereby endangered. They therefore gratefully upload responsibility for decisionmaking to the officially designated decisionmakers.
This in fact is one of the chief reasons for resistance to the devolution of responsibility and accountability from managers to staff in those organizations which are trying it. Staff are reluctant to undertake responsibilities when they have no control over the standards by which their exercise of these responsibilities will be judged. Again, modern organizations are organizations of power rather than of function, and devolving a function without devolving the appropriate power along with it is just illogical.
The conception of evaluation as simply the shovelling of data from the front lines of the organization into the managerial suites is doomed to failure. That is not because management has no role to play in evaluation, or because it doesn't play its role well, but because management decisions are inevitably made at a distance, and at a relatively gross level of analysis. For example, when you go to a hospital for surgery, any fears you may have about your operation will not be assuaged by the information that the department performing the surgery undergoes an administrative review every three years. You want to know that your doctor can evaluate and respond appropriately to any circumstance that may arise in the course of your surgery.
Hospitals are examples of organizations where responsibilities for evaluation are more effectively distributed than in most. Physicians do not call up the hospital board to get approval for their diagnoses. Another institution with a similar distribution of evaluative responsibility is the school. A teacher at a public school does not call up the board's testing division every time he or she wants to enter a mark in a student's record. Most importantly, he or she will also maintain a record of information which can be used to justify his or her pedagogical decisions.
Such institutions demonstrate, or at least have the potential for demonstrating, effective models for organizational evaluation. Management has not stripped all evaluative responsibility from employees. Management, of course, performs a crucial role in evaluation, as do the professional staff hired to perform evaluation. However, the activities of management and professional staff are supplemented by day-to-day, minute-to-minute evaluative decisions made by staff. Staff, if questioned, have a way of demonstrating that they are accomplishing what they are supposed to accomplish.
In other organizations, including many non-profits, staff perform functions without making any evaluative decisions whatever. While their mission statements and program goals are ambitious and important, their records contain little or no evidence that they are in fact accomplishing their goals or their mission. If called to account, they will not only be unable to demonstrate their effectiveness, they will also quite often be completely bewildered by the proposition that they do so.
The irony of all this is that by conceiving of evaluation as a process rather than as a support, we actually provide more support for management, and for evaluative staff, than if we conceive of evaluation as a support or tool. Evaluating programs which do not collect the information necessary for their evaluation is not only ineffective but pointless. Evaluation in such conditions is at best a huge irrelevance. At worst, it serves as justification for counterproductive and dangerous, although seemingly logical, management decisions.
Evaluation in such a non-evaluative corporate culture quickly deteriorates into questionnaire research of the How happy are you with the service you received? Very happy-Happy-Neither happy nor unhappy-Unhappy-Very unhappy variety. In non-profit organizations, the question usually becomes How happy are you with the service you received? Very happy-Happy-Ungrateful.
One of the bitter truths of evaluators' lives in such a bureaucracy is that they will know that people's opinions do not predict their behaviour, but must act as if opinions predict behaviour if they want to justify the salaries they're pulling down. Of course, in the end many come to believe that opinions predict behavior.
In a corporate culture of evaluation, management can rely on information other than unscientifically assessed opinions, and evaluative staff can justify their positions without brainwashing themselves or passing their lives in cruel despair. The information is collected daily on the front lines by staff who understand the relevance of the information to what they are doing. Management still manages, and evaluators still evaluate, but they do so with a lot firmer grasp of what is happening in the organization.
If we truly want evaluation to be a support for management, managers and evaluation staff must ensure that staff are trained in determining the types of information that they need to collect, and that their work schedules permit them to collect it. Staff must have some ability to modify the programs they carry out as a result of conclusions they have drawn from the information they collect, and to evaluate the results without fear of immediate cashiering if they make a mistake.
That is less difficult than it sounds. It may sound difficult because many of us are wedded to the traditional conception of the bureaucracy as an organization of power. Letting people at the bottom of an organization make decisions seems not only threatening but immoral.
The traditional conception of course allows managers to continue in the traditional beliefs of cargo cult evaluation. If they are willing to carry their beliefs through to their logical conclusions, all they have to do is order a sign that reads ISO 9001 or whatever and put it up outside. If they're particularly scrupulous, they might even obtain approval to use the designation. Either way, their sign will serve the same function as all those unused airstrips in Melanesia.
McLuhan, Marshall (1964). Understanding media: The extensions of man. New York: McGraw-Hill.
Reavy, Pat, Linda Littell, Gail Gonda, and Irene O'Neill (1993). Evaluation as management support: The role of the evaluator. Canadian Journal of Program Evaluation, 8 (2), Oct./Nov., 95-104
Saul, John Ralston (1992). Voltaire's bastards: The dictatorship of reason in the West. Toronto: Viking.