[Not a spoof!]
“Shamanistic Rituals in Effective Schools*”
“Shamanistic Rituals in Effective Schools*” by Brian Rowan, Senior Research Scientist, Far West Laboratory for Educational Research and Development. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA, April, 1984. Asterisk in title is notation on bottom of title page which states, “Work on this paper was supported by the National Institute of Education, Department of Education, under Contract No. 400–83–003. The contents do not necessarily reflect the view or policies of the Department of Education or the National Institute of Education.” Brian Rowan was involved in Bill Spady’s Far West Lab grant to the Utah State Department of Education to “put OBE in all schools of the nation.”
This paper develops a theoretical perspective for analyzing the non-scientific uses of research in educational policy debates. A central focus is educational researchers’ use of shamanistic rituals to affect organizational health (cf., Miracle, 1982). A number of shamanistic rituals derived from research on “effective” schools are described here, and an analysis demonstrates the circumstances under which these rituals can be used to divine the unknown, cure ills, and control uncertain events.
Background
Miracle (1982) suggested that shamans and applied social scientists perform a number of similar functions in society. Shamans, the powerful medicine men of premodern societies, worked mainly to cure ills, divine the unknown, and control uncertain events, and they performed these functions by using a specialized craft obtained after a long period of formal initiation and training. Similarly, applied social scientists acquire a specialized craft after initiation and training, and they too are called upon to alleviate the vague ills of corporate groups, divine the unknown for organizational strategists, or bring order to the uncertain events that plague institutional affairs.
The analogy raises a number of important issues for applied social science. First and foremost, shamans practice magic, whereas applied social researchers are thought to practice “science.” To liken scientists to magicians raises interesting questions about the relationship of science to pragmatic action. An additional problem is that shamans are but one of the many practitioners of magic in societies, and they can be distinguished from others who employ magic in their rituals, for example, sorcerers, witches and wizards. This observation raises questions about the uses of research in modern policy analysis. If educational “science” functions as magic, who are the shamans, witches, and sorcerers of educational research?
Forms of Pragmatic Action
We begin with the problem of whether applied educational scientists practice magic. A number of anthropologists have observed that magic is used for pragmatic purposes in premodern societies, but that magic is not the only form of pragmatism available to premodern practitioners. For example, both Malinowski (1948) and Evans Pritchard (1965) argued that premodern societies possessed sound technical logics that practitioners could use to successfully accomplish most work tasks. In addition, premodern people were able to sharply distinguish between these working, practical logics and magic. In premodern societies, when tasks were going well, the technical logic of everyday work dominated action. But as uncertainties increased, or as conflict and stress became more problematic, premodern practitioners began to supplement technique with magic. Thus, Malinowski (1948) observed the fishing practices of Trobriand islanders and found that, in the safety of lagoons, practitioners made little use of magic and relied primarily on established technical routines to ensure good fishing. But as activities moved into the more dangerous open seas, magic was increasingly invoked as a supplemental technical aid.
Similar points can be made about the modern educational practitioner’s use of research. It seems clear that schools have an established series of technical routines (Goodlad, 1983). But these practices are not grounded in the highly stylized logics of modern science. Rather, they exist in the more subtle and largely unarticulated logic of teachers and administrators (Jackson, 1968). Although some educational observers have likened this unarticulated logic to magic (e.g., Lortie, 1975), Malinwoski’s (1948) [sic] discussion suggests that it is more appropriate to think of educational research as magic. The educational practitioner appears to make wide use of the subtle and unarticulated logic of schooling, and this logic appears to have the desired technical effect on a large number of students (Hyman, Wright and Reed, 1975). Practitioners make much less use of the stylized “scientific” knowledge of applied social scientists. Indeed, like Malinowski’s Trobrianders, they appear to reserve the use of “science” for those sectors of schooling which are problematic or in “crisis.”
Other arguments also suggest that educational “science” functions much like magic. As Miracle (1982) noted, both applied social scientists and shamans utilize a “force” that derives from an other world (Mauss and Hubert, 1961). Shamans, for example, often travel to other worlds to communicate with spirits or accompany the dead to their supernatural resting places. As a result, they are said to inhabit both the real world and a spirit or supernatural world. Similarly, applied scientists appear to inhabit two distinct worlds, one the “real” world, the other the proverbial “ivory tower.” It is widely recognized that knowledge gained in the ivory tower is not the same as that gained in the “real” world, an observation that endows “scientific” knowledge with a certain otherworldly nature. Thus, like shamans, applied educational scientists inhabit two worlds and practice a craft that has a special legitimacy in social affairs.
Types of Magic
If we perist [sic] in the analogy between educational “science” and magic, it becomes useful to classify various types of magic and magicians. In premodern societies, for example, there were numerous practitioners of magic, including not only shamans, but also various witches, wizards and sorcerers. Distinctions among these practitioners can be made on the basis of their actual magic practices. Wizards and witches often practiced forms of “black magic” that were used as weapons to defend interests or harm enemies, whereas the shaman’s magic was most often employed for benevolent purposes, including the curing of ills. There is also a need to look carefully at the rituals practiced by different groups. For example, shamans often engage in a common “spitting and sucking cure,” but they also use other rituals from their “bag of tricks.”
Educational researchers can also be classified by the types and functions of the rituals they perform. For example, policy analysts sometimes use the rituals of research to confound and weaken political or scientific opponents, a form of research that appears similar to the “black” magic of witches. But there are also research shamans who can be called upon by policy analysts to perform healing rituals. All types of research ritualists select from a common and well-known bag of research tricks, although in recent years there has been a rise of ritual specialists who exclusively work either qualitative or quantitative magic on policy audiences.
Shamanism and School Effectiveness Research
In this paper, we limit attention to a single type of research ritualist—the research shaman—and to a few related magic tricks used within a narrow policy domain. Our interest is in describing research rituals that heal and revitalize sectors of education and not in research that fans controversy, inflicts harm on ideological enemies, or demoralizes existing constituencies in a policy domain. Moreover, the analysis will be narrowed to a few research rituals used in one policy domain to better illustrate how research shamans operate.
Shamanism and Crisis
It is commonly observed that working practitioners in education remain detached from, even ignorant of, the findings and applications of applied research. Yet this observation is not entirely true. Educational policy makers and their research ritualists continue to generate research, and this research continues to play a role in certain sectors of educational practice. Thus, a question emerges: in what sectors of educational institutions are the rituals of research shamanism most utilized?
Anthropological studies suggest some answers to this question. It has been argued that magic assumes its highest importance in institutional sectors plagued by three conditions: (a) high levels of technical uncertainty; (b) structural cleavages that create great stress among social groups; and (c) social disorganization that creates problematic mood states among participants (Malinowski, 1925; Gluckman, 1952; Wallace, 1956). The argument here is that research shamanism is most valued in sectors of education that contain these characteristics. Thus, research in education is most numerous in areas where there is high technical uncertainty (do schools/programs/teachers make a difference to educational outcomes?). The rituals of research also take on great importance in areas where there is conflict among social groups (are new educational initiatives needed to redress past social inequities?). And finally, research is increasingly directed at problems related to disorganization and dissatisfaction in institutional sectors of education (are urban/high schools better or worse than in the past?).
Research on Effective Schools
Research on effective schools has its origins in these problems. The research deals with a sector of educational institutions—the instructional core—which has long been the subject of uncertainty, conflict, and pessimism, and where the use of myth and ritual has been common (Meyer and Rowan, 1977; 1978). What is distinctive about “effective schools” research, in contrast to much past scientific work, is that it has taken a shamanistic approach to the problems of schooling. It has not fanned the flames of discontent and uncertainty like previous scholarly work (e.g., Coleman et al., 1966; Averch et al., 1972; Jencks et al., 1972), but instead has held out hope that the pervasive ills of modern urban schooling can be cured.
Edmonds (1979a), the most powerful of all effective schools shamans before his untimely death, seemed accutely [sic] aware of the need for healing in modern educational institutions, and a careful reading of his works reveals his strategy for effecting a cure for the problems confronting urban education. He argued that research must be used to counter the pessimistic view that schools have weak effects on student outcomes, and that as this occurred, practitioners could attain new expectation states that facilitated, rather than hindered, the achievement of disadvantaged children (see, especially, Edmonds, 1978; 1979b). Thus, Edmonds saw that “science” could be used to confront the conflicts, uncertainties, and problematic mood states afflicting modern schooling.
That Edmonds’ [sic] approach possessed a special “force” in educational policy arenas is indisputable. Like the revitalization movements that swept the great plains during the period of indian [sic] decline (Wallace, 1966), the rituals of effective schools research diffused widely and rapidly. They were adopted by other shamans, who brought them to state departments of education and local school systems, and there these rituals were used as the cornerstone of ambitious revitalization ceremonials (see, e.g., Ogden et al., 1982; Shoemaker, 1982; Clark and McCarthy, 1983).
It is worth noting that the perspective being developed here does not necessarily imply that these shamanistic rituals are hoaxes. Indeed, just as many modern medical practitioners have come to recognize the wisdom and efficacy of shamans, there is at least some reason to think that the arguments of effective schools proponents possess some scientific merit (see, e.g., Rowan, Bossert and Dwyer, 1983). Nevertheless, for the moment, it is useful to suspend our empirical curiousity [sic] about whether these initiatives really “work,” [sic] and to examine instead some of the concrete ritual practices that characterize this new educational movement.
Important Shamanistic Rituals
It has already been suggested that shamanistic rituals are designed to cure ills, divine the unknown, and control uncertain events. In this section of the paper, three prominent effective schools rituals are discussed and their relationship to the central functions of magic are illustrated.
Curing Ills with Literature Reviews
We begin with one of the most common shamanistic rituals in the effective schools movement, the glowing literature review that promises relief from the currently pervasive sense that educational institutions are in poor organizational health. Miller’s (1983: 1) review illustrates the general form of this ritual: “Not so long ago the conventional wisdom regarding American schools was that ‘schools do not make a difference.’ ...Yet today... the message of... research is primarily postive [sic] and upbeat: schools can make a difference” (Miller, 1983: 1).
A closer look illustrates the consistent dramatic form used by reviewers to affect the promise of a cure. First, the authors contrast the dismal tradition of school effects research with “more recent” and more positive studies of effective schools. This is followed by the citation of a host of previously unpublished and obscure studies which are often nothing more than other positive literature reviews. The final step is a grandiose concluding statement, which most often calls on practitioners to adopt the new discoveries.
We speculate that these rituals have their most dramatic effect on naïve individuals who have little time or inclination to follow-up footnotes or read works cited in the text, or on those who have little tolerance for the ambiguity that marks true scientific debate. Lacking a systematic understanding of the scientific pros and cons of effective schools research, naïve individuals are left only with the powerful and appealing rhetoric of the reviewers. Thus it is that research on effective schools has come to be seen as a “cure” for educational ills the less it has been published in scholarly journals and the more it has been disseminated in practitioner magazines. The experiences shaman knows to avoid the scrutiny of scholars, for this can raise objections to the “scientific” basis of ritual claims and divert attention away from the appealing rhetoric. Instead, the shaman cultivates the practitioner who needs a simple and appealing formula.
Divining the Unknown Using Outliers
While the literature review ritual can be observed equally well by both qualitative and quantitative specialists, a second ritual, designed to divine the unknown, is the exclusive domain of quantitative ritualists. The ritual uses residuals from a regression analysis to identify “effective” schools and to contrast them with “ineffective” schools. The purpose is to divine an answer to two nagging questions in school effectiveness research: which are the effective schools in a system and what are these schools doing that makes them different?
The techniques involved in this ritual have been described before (see, Rowan et al., 1983). A regression equation predicting school achievement from school socioeconomic composition is tested, and errors of prediction are calculated. The errors (or residuals) are used to identify “effective” and “ineffective” schools and form samples for contrasted groups studies. The ritual almost always strongly supports the rhetorical posture of the ritual literature review. Since predictor variables never account for all of the variance in school-level achievement, an analysis of residuals will always demonstrate that schools differ in achievement even after controlling for socioeconomic composition. Thus any experienced shaman can find “effective” schools. Second, if a shaman asks a large number of questions, a number of structural and cultural differences between effective and ineffective schools can be found. Thus, the outliers ritual not only identifies the previously unrecognized “effective” schools, it also reveals for the first time why these schools attain effectiveness.
From a magician’s standpoint, this ritual’s power can be increased in a number of ways. First, the worse the specification of the initial regression model, the more persuasive the ritual. For example, by failing to include all measures of school socioeconomic composition, a shaman can increase the residual achievement differences between schools. This, in turn, enhances claims that “effective” schools make a difference to achievement. Moreover, to the extent that school characteristics are correlated to omitted socioeconomic predictors, misspecification [sic] enhances the liklihood [sic] that differences in school characteristics will be found between “effective” and “ineffective” groups of schools. Thus, the worse the initial regression model, the more powerful the shamanistic ritual.
A related tactic is to use aggregate models. By using schools rather than individuals as the unit of analysis, proportions of variance in achievement explained by school management and culture are increased. In between-school analyses, schools can be seen to account for nearly 30% of the variance in achievement. But in between-individual analyses, this is reduced to about 5%. Thus, effective schools ritualists have been able to inflate their claims of school effects through a simple aggregation trick (see Alexander and Griffin, 1976).
The experienced shaman also avoids certain practices. For example, it is wise not to repeat the residuals ritual in the same population, for this highlights the low correlation of residuals over time and raises questions about measurement reliability. It is much wiser to demonstrate reliability by using the conventional, and cross-sectional, “split/half” procedure of psychometricians (see, Forsythe, 1973). Similarly, after a few performances of the residuals ritual and the associated contrasted group study, it becomes possible to ignore problems of validation. Thus, as time moves on, the wise shaman avoids achievement data and the residuals ritual entirely, and instead assesses schools on the degree to which their structures match those of previously identified “effective” schools.
Controlling Uncertainty through Measurement
A final shamanistic ritual in the effective schools movement requires the shaman to have advanced training in the art of psychometrics. The ritual is particularly suited to application in urban or low performing school systems where successful instructional outcomes among disadvantaged students are highly uncertain but where mobilized publics demand immediate demonstrations of success. The uncertainties faced by practitioners in this situation can easily be alleviated by what scholars have begun to call “curriculum alignment.”
This ritual begins with an analysis of what is actually being taught in schools. The shaman conducting the ritual assembles a group of local practitioners and together they list instructional objectives for each grade level. The next step is to find achievement tests that ask questions related to these objectives. To the extent that test items matching local objectives are found, either in commerically [sic] prepared tests or in locally constructed ones, and to the extent that these items are used in achievement testing rather than the haphazard collection of items contained in most commerically [sic] prepared tests, the curriculum and testing systems of the local school are said to be “aligned.”
Since it is known that at least some variance in student achievement is a function of students [sic] opportunity to learn what is tested in criterion measures (Cooley and Leinhardt, 1980), the alignment ritual can have immediate effects on perceptions of effectiveness. For example, a school system moving from an unaligned commercially prepared achievement test to an aligned one can expect that it will score higher on national norms than before. But this increased “effectiveness” does not occur because students are learning more or different things. In the typical alignment ceremony, only test items—not instruction—are changed. Nevertheless, while student learning remains unchanged, alignment allows students to practice criterion measures and achieve higher test scores, thus giving them an advantage over comparable students in unaligned school systems.
An even more powerful demonstration of instructional effectiveness can be achieved if shamans avoid the standard psychometric practice of designing norm-referenced achievement tests and move instead toward criterion-referenced tests. As Popham and Husek (1969) discussed, the typical norm-referenced achievement test eliminates items that nearly all students in a population can answer correctly, since norm-referenced tests are designed to produce between-student variance in achievement scores. But if one neglects this practice and allows items that almost everyone can answer correctly to be included in achievement tests, a larger number of students will appear to be performing more successfully in their academics.
Thus, the art of measurement can be used as an aid to shamanism, espcially [sic] in urban schools plagued by the uncertainties of student performance. Student variability in performance can be reduced, and relative performance increased, not by changing instructional objectives or practices, but simply by changing tests and testing procedures.
Conclusion
The analysis of specific shamanistic rituals in the effective schools movement raises a number of important questions about the relationship of applied science to pragmatic action. Most importantly, it suggests that future studies of “science” as magic are needed. There is a need to begin to chart other rituals used by applied scientists to disarm enemies, cure ills, and divine the unknown. Moreover, there is a need to study the conditions under which these magical practices spread through practitioner populations. Using this perspective, much of the literature on organizational change and applied research can be rewritten from an institutional perspective (Meyer and Rowan, 1977).
At the same time, there is a need to carefully analyze the science of magic. There can be little doubt that Malinowski’s (1948: 50) observations about premodern magic will ring true for many observers of current applied research in education:...when the sociologist approaches the study of magic... he finds to his disappointment an entirely sober, prosaic, even clumsy art, enacted for purely practical reasons, governed by crude and shallow beliefs, carried out in a simple and monotonous technique.
Yet this “clumsy” art sometimes achieves great effects in practitioner communities and may even have some empirical merit, and this raises the appealing promise that applied social scientists can someday develop shamanistic rituals that empirically “work.”
References
Alexander, K. and L. Griffin. School district effects on academic achievement: a reconsideration. American Sociological Review, 1976, 41, 144–151.
Averch, H.A. et al. How effective is schooling? A critical review of research. Englewood Cliffs, N.J.: Educational Technology Publications, 1972.
Coleman, J.S. et al. Equality of educational opportunity. Washington, D.C., U.S. Department of Health, Education and Welfare, 1966.
Cooley, W.W. and G. Leinhardt. The instructional dimensions study. Educational Evaluation and Policy Analysis, 1980. 2, 1–26.
Clark, T.A. and D. McCarthy. School improvement in New York City: the evolution of a project. Educational Researcher, 1983, 12. 17–24.
Edmonds, R. A discussion of the literature and issues related to effective schooling. St. Louis: CEMREL, Inc., 1978 [sic].
Edmonds, R. Effective schools for the urban poor. Educational Leadership, 1979a, 37, 15–24.
Edmonds, R. A conversation with Ron Edmonds. Educational Leadership, 1979b, 37, 12–15.
Evans Pritchard, E. Theories of primitive religion. London: Cambridge Press, 1965.
Forsythe, R.A. Some empirical results related to the stability of performance indicators in Dyer’s student change model of an educational system. Journal of Educational Measurement, 1973, 10, 7–12.
Gluckman, M. Rituals of rebellion in S.E. Africa. London: Oxford Press, 1954.
Goodlad, J.I. A Place called school. New York: McGraw Hill, 1983.
Hyman, H.H., C. Wright and C. Reed. The enduring effects of education. Chicago: University of Chicago Press, 1975.
Jackson, P.W. Life in classrooms. New York: Holt, Rinehart and Winston, 1968.
Jencks, C.L. et al. Inequality: a reassessment of the effects of family and schooling in America. New York: Basic Books, 1972.
Lortie D. Schoolteacher. Chicago: University of Chicago Press, 1975.
Malinowski, B. Magic, science and religion. Glencoe: Free Press, 1948.
Mauss M. and H. Hubert. On magic and the unknown. In, Parsons, T. et al (eds.). Theories of society, II. Glencoe: Free Press, 1961.
Meyer, J. and B. Rowan. Formal structure as myth and ceremony. American Journal of Sociology, 1977, 83, 340–363.
Meyer, J. and B. Rowan. The structure of educational organizations. In M. Meyer et al, Environments and organizations. San Francisco: Jossey Bass.
Miller, S. A history of effective schools research: A critical review. Paper presented at the annual meeting of the American Educational research [sic] Association, Montreal, April, 1983.
Miracle, A.W. The making of shamans and applied anthropologists. Practicing Anthropology, 1982, 5, 18–19.
Ogden, E., W. Fowler and D. Kunz. A study of strategies to increase student achievement in low achieving schools. Paper presented at the annual meeting of the American Educational Research Association, New York, March, 1982.
Popham, W. and H. Husek. Implications of criterion referenced measurement. Journal of Educational Measurement, 1969, 6, 1–9.
Rowan, B., S. Bossert and D. Dwyer. Research on effective schools: a cautionary note. Educational Researcher, 1983, 12, 24–31.
Shoemaker, J. What are we learning? Evaluating the Connecticut school effectiveness project. Paper presented at the annual meeting of the American Educational Research Association, New York, March, 1982.
Wallace, A. Revitalization movements. American Anthropologist, 1956, 58, 264–281.
No comments:
Post a Comment