This article has been retrieved
times since February 21, 2000
This article has been retrieved
Education Policy Analysis Archives | ||
Volume 8 Number 12 |
February 21, 2000 |
ISSN 1068-2341 |
|
Editor: Gene V Glass, College of Education Arizona State University
Copyright 2000, the
EDUCATION POLICY ANALYSIS ARCHIVES. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education. |
Top-Down, Routinized Reform in Low-Income, Rural Schools:
|
|
This article has Since 1991, the National Science Foundation has funded fifty-nine state, urban, and rural systemic initiatives. The purpose of the initiatives is to promote achievement in math, science, and technology among all students, and to encourage schools and communities to secure the resources needed to maintain such outcomes. The Appalachian Rural Systemic Initiative (ARSI) is a six-state consortium which focuses these efforts on low-income, rural schools. The primary means of accomplishing ARSI's aims is a one-day-one-school site visit, called a Program Improvement Review, done by an ARSI math or science expert. The centrally important Program Improvement Reviews, however, seem to be premised on unsubstantiated assumptions as to the static, easy-to-understand, easy-to-evaluate nature of educational achievement in rural Appalachian schools. As a result, the Reviews resemble exercises in early-twentieth century scientific management, and are unlikely to enhance achievement in science or math. Consequently, even if there is merit to the commonsense human capital approach to economic growth and development on which systemic initiatives are tacitly premised, this first- person account makes a case that desired payoffs are unlikely to follow from the work of ARSI. |
|
Efforts to promote economic development and eliminate poverty through investment in public education have a long history in the U.S. (See, for example, Bowles and Gintis, 1976; Kaestle, 1983; Perkinson, 1995; Spring, 1997; McMurrer and Sawhill, 1998). In recent years, such efforts have included special attention to elementary and secondary schooling in science, math, and technology (Ashton and Sung, 1997; Senate Committee on Labor and Human Resources, 1997). This emphasis is premised on the assumption that in an increasingly science-based, technology-intensive world, the economic well-being—perhaps even the simple survival—of individuals and entire societies requires ever-higher levels of pure and applied scientific and mathematical knowledge (Shapiro and Varian, 1998; National Council of Teachers of Mathematics, 1998; Reich, 1992). The National Science Foundation's Systemic InitiativesIn line with this straightforward human capital theoretic point of view, since 1991 the National Science Foundation has funded fifty-nine state, urban, and rural systemic initiatives (National Science Foundation, 1999). The purpose of each systemic initiative is to promote education in math, science, and technology (National Science Foundation, 1994a).Published research on the initiatives is hard to find, and evaluation reports are not available. The origin of the term "systemic initiative" remains unclear. NSF's recent request for proposals for "systemic initiative research" provides no insight as to the meaning of the concept (NSF, 1998). The terminology may follow, however, from NSF's judgment that education involves entire communities (Shields, 1997). At its best, in this view, education in math and science focuses on everyday applications in communities where schools are located (National Science Foundation, 1994b). The communities themselves, in a reciprocal process, benefit from development of a technologically literate workforce (Consortium for Policy Research in Education, 1995). NSF's Appalachian Rural Systemic Initiative (ARSI)The Appalachian Rural Systemic Initiative, or ARSI, is a six-state consortium, covering all of the Appalachian region of the U.S. (Harmon and Blanton, 1997). Consistent with NSF's intent, ARSI's ambitious objective is to facilitate educational change in economically disadvantaged rural schools resulting in high achievement for all students in mathematics and science (National Science Foundation, 1997). This is to be accompanied by development of community resources to sustain educational improvements (Brown, 1996).Program Improvement ReviewThe primary means of accomplishing ARSI's aims is the Program Improvement Review. Done by ARSI experts, typically retired teachers, the purpose of a Review is to identify strengths and weaknesses in schools' math and science programs and make recommendations for improvement. ARSI experts, thereby, are charged with helping low-income, rural schools make students more productively employable in a science-based, technology-intensive world. In doing this, ARSI experts aim to contribute to production of the human capital needed for the economic and social development of low-income rural areas.Will ARSI Promote Economic Development in Appalachia?The uncomplicated human capital perspective on which ARSI is premised begs an important policy question. Specifically, can educational reform be used to drive a growth and development strategy whereby the availability of well educated prospective employees attracts employment- creating investments? A tenable alternative holds that economic development is a necessary prerequisite for effective educational change (Bickel and Spatig, 1999).For present purposes, however, we will put this reservation aside and address a more manageable question: If the education-and-development assumptions on which ARSI is premised were undeniably correct, would ARSI accomplish its objectives? A First Person AccountThe following account is written from the vantage point of one who was first an ARSI expert-aspirant, then an ARSI expert writing his first Program Improvement Review, and finally an ARSI dropout. The descriptions of "shadowing," of neutral-site instruction, of report preparation, and of rejection of the ARSI model are based on work done as part of the process of bringing ARSI to West Virginia under the auspices of the regional university with which the paper's authors are affiliated. Participation in this endeavor leads to the following inferences:ARSI experts construe the process of educational achievement as a thoroughly understood, relatively simple mechanism manifest in static indicatiors of school effectiveness.The remainder of this article is devoted to clarifying these inferences based on a first-hand account of ARSI at work. Throughout, one important message seems clear: ARSI's Program Improvement Reviews in low-income, rural schools are unlikely to enhance science and math achievement or promote economic growth and development. We attribute this unfortunate set of circumstances to specious assumptions as to the existence of a taken-for- granted, science-based rationale for the top-down routinization and streamlining of educational evaluation and practice. As a result, even if the commonsense human capital framework on which systemic initiatives are based were valid, ARSI's work would not facilitate their application. |
A Checklist-Guided AuditThe Program Improvement Review takes the form of a one- day, one-expert school visit, yielding a checklist-guided audit, resulting in degree-of-compliance scores ranging from 1 to 5 on approximately seventy Likert items. The check list is called a "Consistency Rating Summary."For example, when evaluating a math program, the first general heading is "Curriculum, " subsuming ten Likert items, the first being "1.1 The math curriculum is written and used in planning the instructional program." The remaining general headings are "Instruction," "Thinking Processes," "Equity and Diversity," "School Climate, " "Relevance" or "Connections," "Training and Development," and "Financial and Material Resources." The total number of items varies slightly depending on the discipline, math or science, the grade level, and the state which provides the educational policy setting for the review. Minor variations in the wording of the general headings and individual items are geared to these same factors. For example, under "Instruction," the first item used in evaluating math programs in West Virginia elementary schools reads as follows: "2.1 Teachers use WV IGO's to guide their instructional practices." "WV IGO's" refers to state-mandated "Instructional Goals and Objectives," around which high-profile state achievement tests are organized. Likert item scores are used to gauge specific strengths and weaknesses in a school's math or science program. Strengths reflect consistency with the ARSI model embedded in the "Consistency Rating Summary." Weaknesses reflect departures from the model. In practice, far more attention is given to weaknesses than to strengths. In spite of the importance of the Consistency Rating Summary, the source of its ten headings and seventy items is not identified. Are they research-based? Are they reasonable inferences based on years of teaching experience? Are they established principles in math and science education? Is their appeal based on face validity among ARSI experts? Do they represent an identifiable educational philosophy or pedagogical model? Participants are not told. Literature is nowhere to be found. NSF StandardsNSF has promulgated a detailed set of National Science Education Standards (National Research Council, 1996). In the course of conversation and training with ARSI experts, however, these are never mentioned. If the experts are aware of NSF Standards, they do not disclose this. If NSF Standards are a source for the Consistency Rating Summary, participants are not told. The absence of descriptive, evaluative, or any other sort of literature concerning the Summary is again conspicuous. ARSI experts occasionally make off-handed references to "constructivism," and they are fond of invoking the notion "hands-on." One might reasonably surmise, therefore, that these ideas, though they typically remain vague, are included in construction of the Consistency Rating Summary and the way it is scored. In the absence of pertinent literature, however, this remains merely plausible conjecture.State MandatesARSI experts often refer to state mandates, such as West Virginia's Instructional Goals and Objectives, mentioned above, and the Kentucky Core Content for Assessment. Whatever the merit of these state-level mandates, their substance appears to have been another influence in construction of the Consistency Rating Summary, and affects the way it is applied. The heading emblazoned at the top of the Consistency Rating Summary may vary with the state in which it is being used, as in "KERActeristics of a Good Mathematics Program" used in Kentucky, or the "West Virginia Program Improvement Review Consistency Rating Summary for Mathematics."Beyond these tentative inferences, however, no rationale for the instrument is provided. One is left with the impression that the Consistency Rating Summary may very well have been the product of brain-storming sessions. The outcome is an instrument which appears to be vaguely current and topically correct, but which, as an evaluation tool, is of uncertain value. Consistency Rating Summary ValidationSimilarly, the technical properties of the Consistency Rating Summary as a measurement tool are not reported, and may not have been investigated. Given organization of the instrument into ten sections, each subsuming six to ten items, one might reasonably surmise that a factor analysis would reveal ten identifiable subscales. If this is the case, however, results are not available. The same is true for routine reliability coefficients. In short, the psychometric properties of the instrument seem not to be known. The possibility that discussion of such properties might be pertinent, even essential, is not acknowledged by ARSI experts.Reporting on a Program Improvement ReviewThe final report, usually written overnight and presented the next day, is organized around the same ten general headings and seventy Likert items. Since much more attention is given to weaknesses than to strengths, most reports do not address all general headings or all items, but only those deemed deficient.Recommendations for change appear throughout the report. A recommendation pertaining to "Relevance", meaning "[relating] mathematical knowledge to students' goals and interests," for a middle school located in a low-income, rural district in West Virginia's southern coal fields reads as follows: "Make a concerted effort to display positive, engaging images of mathematics throughout the school environment, paying particular attention to highlighting student work that is creative (not just correct) . . ." [Emphasis in the original.] Becoming an ARSI ExpertTraining in doing the Program Improvement Review, including scoring the Consistency Rating Summary, usually begins with "shadowing," accompanying an ARSI math or science expert who is doing a Review. ARSI experts also provide training at neutral sites, relying heavily on videos prepared to meet their specific instructional needs. Limited role-playing is used as a means of readying prospective experts to present Program Improvement Review findings to school personnel.Training is informal, with little or no direct instruction. Instead, the ARSI experts serve as models during shadowing, and provide illustrative opportunities to apply the ARSI model during training sessions. Total training time varies, usually ranging from two to three days. An experienced ARSI expert may also participate in the first Program Improvement Review done by a just-trained expert. "Shadowing" in Chemistry 8-B: Deficient InstructionTo illustrate our claim that ARSI Program Improvement Reviews are unlikely to enhance achievement, we begin with a brief case study of shadowing. Two ARSI expert-aspirants, assisting in bringing ARSI to West Virginia under the auspices of the university which employs them, are observing the in-school work of an ARSI science expert at a small, rural, low-income elementary school in eastern Kentucky.We first attend a chemistry class. The three of us open the front door to the classroom without knocking, walk to the rear without speaking, and sit in side-by-side desks, while the class goes on about us. Students seem uninterested in our intrusion. The teacher seems unconcerned, and she makes no effort to acknowledge our presence. Even though this elementary school goes through grade 8, chemistry, rather than, say, general science, seems out of place, too advanced for an elementary school. The class, moreover, is referred to as Chemistry 8-B. This, we learn, means that chemistry students are grouped or tracked, with the ostensibly more capable students located in section 8-A. Nevertheless, the approximately twenty-five students in section 8-B seem quite capable themselves. The teacher is reviewing chemical bonding, referring to positive and negative valences, what they mean with regard to the make-up of individual atoms, and how they govern the way different elements combine to form molecules. She makes occasional reference to a periodic table displayed within easy reach on the wall near the front of the room. Desks are organized in traditional fashion, arranged in rows, all facing forward. The teacher's desk is in the front of the room in the middle, turned toward the students. The teacher stands slightly to the left of her desk facing the students and occasionally turning to the board or, less often, to the periodic table. The presentation, too, is traditional, relying largely on lecture and board work, with questions and responses to teachers' queries from students. The teacher speaks fairly rapidly. The substance of the class is in no sense trivialized to match the ostensibly limited capabilities of lower track students. The material covered is high school chemistry, much as I remember it from the eleventh grade. The teacher, though, seems smarter and more articulate, explaining things more clearly than I remember mine doing decades ago. Her high expectations for students are genuinely taken for granted. None of the students stands out as a stellar performer or favorite. The teacher's high expectations seem to apply equally to everyone. The truly remarkable things about the class are the students' responses. All white, about half male and half female, they seem genuinely engaged. They attend single- mindedly to the teacher's presentation. The students, manifestly, are putting all their time on task. Not just any task, but the conceptually difficult, even esoteric task at hand. The teacher asks questions fairly often. Answers are quickly forthcoming, spoken thoughtfully, usually confidently, without the formality of hand-raising. Students' questions are immediately acknowledged and answered in a business-like, though not unsympathetic fashion. The teacher, a woman of about thirty who seems obviously to enjoy what she is doing, tries various means of explaining the same difficult ideas, sometimes complementing her oral presentation with additional board work. Students don't talk among themselves. Two girls on the teacher's right near the front of the room are an exception, but as they whisper, they look toward the chalkboard, and one points to a diagram that the teacher had drawn earlier, illustrating the bonding of sodium and chlorine. A male student near the rear of the room on the teacher's left has a persistent problem with understanding her explanation of positive and negative valences. He makes his difficulty conversationally evident: "Yeah, but I still don't get it. The signs are the opposite . . ."He makes his point, in the same conversational fashion, more than once: "I still don't get it. Why isn't it negative . . .?"The teacher explains again, varying her choice of words. She gives no evidence of impatience. She addresses the questioning student in a matter-of-fact, even collegial fashion. She moves on, still holding students' attention, and doing so effortlessly. She presents material with relaxed enthusiasm born of genuine interest. There is no exaggerated affect or undue dramatization as she continues with a traditional presentation of conceptually sophisticated material. The puzzled student on the teacher's left remains confused about positive and negative valences, though the precise nature of his misunderstanding is still not quite clear. He remains engaged, however, and raises the issue yet again, without evidence of embarrassment or anxiety. The teacher stops and thinks, looks at her diagrams on the board, seems not to know what else to say. A male student sitting to the immediate left of his confused colleague responds spontaneously and matter-off- factly: "I think I see . . . try this."I cannot hear what is said. After a brief exchange between the two students, the puzzled one addresses the teacher: "If sodium is short an electron and it adds one, why isn't it negative?"Implied in this question is a complementary query about chlorine: if chlorine has an extra electron and it gives one to sodium, why isn't chlorine positive? The source of the student's confusion is now clear. The +1 valence of sodium is determined in its free state, before it combines with chlorine to form table salt. The fact that it takes an electron from chlorine—in effect adds a negatively charged particle—does not make it negative. The fact that it has a place for an electron that it adds to its outer ring, however, does make it positive. And conversely with chlorine. The nature of the difficulty having finally been clarified, the teacher is able to dispel the formerly puzzled students' misunderstanding. He is satisfied. The teacher and students continue in the same matter-of-fact but engaged manner which has prevailed from the beginning. One way to usefully characterize their approach and the nature of the affect which accompanies it might very well be "professional." As an observer, I was stunned. How did the teacher manage to hold the attention and active interest of this B- level—or any level—eighth grade—or any grade—chemistry class—or any class—throughout an entire period in which she discussed, in traditional lecture form, chemical bonding? In a low-income, rural, K-8 elementary school in eastern Kentucky or anywhere else! Here, as best I could determine, was science being taught and learned about as well as could be done. Since the aim of ARSI is to promote high achievement in math and science in low-income, rural schools, this, perhaps, was a model, though one that might prove difficult to codify. NSF's National Science Education Standards, which may or may not be known to ARSI experts, are intended to enable educators to judge whether particular actions will serve the vision of a scientifically literate society (National Research Council, 1996). The actions of this teacher and her students emphatically did just this. Or so it seemed to me. |
An Off-the-Cuff Evaluation of Chemistry 8-BAt the behest of the ARSI science expert, the three of us who had been observing left before the class was over. We had been in Chemistry 8-B for about twenty-five minutes. Going out the door at the front of the room, I said to the teacher:"We're leavin' ‘cause we can't understand this stuff."The teacher stopped in mid-lecture, looked at me while I was speaking, and an expression of uncertainty left her face as she smiled. She gestured toward her students and said with confident pride: "They can understand it!"As the three of us walked to the next class, the ARSI science expert, striding purposefully, leading the way, offered the following judgments: "They didn't understand a word she said." His tone was contemptuous. "She was way, way over their heads."The other observer was non-committal, as if taking in what was being said but still processing it, neither concurring nor disagreeing. There was a brief silence as we walked. Then I said, laughingly, "for what it's worth, she teaches just like I do, when I'm having a really good day." Neither the ARSI science expert nor the other observer acknowledged my comment. The ARSI expert led us into the next classroom. I felt sort of silly. Not because no one had acknowledged my response, but because I had felt the need to cover it with self-deprecating laughter. It was clear that the ARSI expert had definite preconceptions as to what eighth graders could and could not handle. His conclusion that the students in 8-B chemistry had no idea what the teacher was talking about seemed wildly at odds with what I had seen and heard in the classroom. Even the puzzled student eventually understood, and he did so with the help of another student. His confusion, moreover, bespoke an understandable, even imaginative, failure to see the specific terminological conventions which were being employed. In a real sense, his confusion about terminological conventions actually reflected a clear understanding of the chemical bonding process itself. The teacher's method of presenting the material was traditional, to be sure. The students participated freely, however, without fear and without required hand-raising. The teacher-student, and student-student exchanges were conversational and matter-of-factly animated. Students helped each other. The Irrelevance of StudentsFor the ARSI expert, however, this relaxed, informal, traditional approach was inevitably ineffective. It was abundantly clear that the living presence of students in the classroom was not essential to his judgments. He seemed not to notice them, their engagement, or the informed nature of their exchanges with the teacher and with each other. The expert attended only to the teacher, her traditionally limited use of few instructional materials, and the dearth of wall posters.One Best Way to Teach ScienceThe ARSI expert clearly judged himself to be in a position to evaluate any science teacher's performance without benefit of observing or otherwise evaluating student responses, to which he seemed oblivious. In this instance, he purported to know in a matter of minutes that the teacher was clue-less, and that students would not learn. Traditional lecture was bad. Absence of manipulatives was worse. "You can use them to build molecules," he assured us."That's what she was trying to do, but it's something you have to get your hands on. There weren't even any [manipulatives] in the room." |
Thin DescriptionThe ARSI science experts' dismissive, almost angry assessment of the teacher's effectiveness bespoke a willingness to generalize from very limited information. His assumption, clearly, was that twenty-five minutes of haphazardly selected, barged-in-on class time enabled him to produce an accurate typification of the teacher's performance and students' consequent achievement.His harsh judgments, moreover, seemed inconsistent with NSF's position that science teaching and inquiry can be effectively done in a variety of ways (National Research Council, 1996; also see National Science Teachers Association, 1998). But once again, the connection between NSF and ARSI may or may not entail a shared understanding about teaching science and math. NSF standards may or may not be known to ARSI experts. In any case, the experts do not mention them. "Shadowing" in a Program Improvement Review PresentationThree weeks after the visit to the eastern Kentucky K-8 elementary school, I was again involved as an observer. I was paired with the same ARSI expert-aspirant, shadowing another ARSI expert in another small, low-income, rural elementary school in eastern Kentucky. During an hour-long, late morning meeting, the ARSI expert presented his previous-day's findings to the school's principal and six teachers. The ARSI expert began with a weak, almost apologetic grin:"This isn't as bad as it looks. There are a lot of 1's, 2's, and 3's, but this can be fixed . . . a lot of it . . ." [His voice trailed off.]Criteria used in selecting the six teachers present at the meeting were not specified. They and the principal, however, remained silent as the ARSI expert went over his largely unfavorable report. "There's no evidence of the importance of math. They come away thinking it's just what they do in school."The principal, in spite of the beating her school was taking, looked confident and even eager throughout, as if to say "we're professional educators sharing information. There's nothing personal about this. We're glad to hear from outside experts, and we'll benefit from it. Please go on." The teachers seemed affectively disengaged but dutifully attentive. They betrayed no emotion. They seemed to neither accept nor reject the ARSI expert's account. Teachers' Informal ChallengeAfter the report was presented, with only a few words of perfunctory discussion, we went to lunch in the school's cafeteria. By chance, I stood in the serving line with two of the teachers who had attended the meeting. Female, white, in their late-forties to mid-fifties, the teachers pleasantly initiated a conversation by asking where I was from. We talked briefly about West Virginia and work I had done in a rural county there. I likened that to what was being done by ARSI in their school. This was followed by a brief what-do-we-say-now sort of silence.By way of keeping the conversation going, I added that the West Virginia project had been a long one. One of the teachers asked how long. I replied that it had gone on for three years, relying heavily on repeated focus groups with a broad range of stakeholders, and on literally hundreds of visits to the three schools involved. The teachers became more animated and emphatic. Speaking of the ARSI expert's report of instructional omissions and other deficiencies, they commented: "We do a lot of that stuff, but we don't do it all the time. He was only here for one day, for a few hours . . ."Clearly, in this low-income, rural elementary school in eastern Kentucky, teachers were challenging the assumption that ARSI experts' one-day site visits enable them to understand a school's math or science instruction. This assumption, nevertheless, tacitly under girds all ARSI Program Improvement Reviews. In retrospect, it seems obvious that I invited this challenge from the two teachers with my mention of a three- year project in West Virginia. At the time, however, I was just awkwardly trying to hold up my end of a conversation. Moreover, the teachers' responses seemed genuine, something they were waiting for a chance to say. Perhaps I had given them a deserved rhetorical opportunity, rather than a naked invitation to engage in a defensive, self-serving polemic. Training in Fixing Deficiencies: Conservation of MomentumIn addition to shadowing, the training of ARSI expert- aspirants includes neutral-site instruction offered by ARSI experts. As an example, ten ARSI expert-aspirants and a handful of interested onlookers met with an ARSI math and science expert at the small-city headquarters of a West Virginia regional education agency. It was the ARSI expert's aim to continue with the introduction of expert- aspirants to the ARSI approach to evaluating education in science and math. |
ARSI Training VideosA retired teacher, the expert relied largely on a series of videos intended to provide opportunities to illustrate the ARSI ethos in use. During one of the longer and more purposeful videos, a white female teacher in her late twenties is seen reviewing the concept "conservation of momentum" with her high school physics class. There are approximately twenty students, all of them are white, about evenly divided between males and females. Is this a functioning classroom, or something staged by ARSI to aid in the production of new ARSI science and math experts? We are not told, and no one asks.The students in the video are more or less attentive. The teacher's presentation is brief and seems to lack focus, perhaps because the video begins near the end of her explanation, immediately following an exercise with manipulatives. Oddly, there is no teacher's introduction to the video itself. It just starts. Whether or not this is an ARSI-staged video, the absence of an introduction, an explanation to students as to why the video is being shown, is disconcerting. After all, we are supposed to be engaged in the evaluation of instruction. Maybe the ARSI expert will use the teacher's failure to introduce her video as a painfully obvious illustration of the wrong way to do things, such as use audio-visual aids in explaining conservation of momentum. The video is devoted entirely to cars crashing. Cars crashing into each other, cars crashing into telephone poles, cars careening off guard rails and rolling onto their roofs, cars going off the road and landing in ditches . . . . It is reminiscent of a demolition derby, but without a winner. It is not immediately evident to me that the video actually does illustrate conservation of momentum. The ARSI expert says nothing. The only sounds in our room, as in the classroom on the video, are made by crashing cars. As we watch the students watching the video, they seem, for the most part, unmoved. The camera catches two male students sitting together laughing at one, seemingly unexceptional collision. The crashes, presumably, were staged. All the cars are from the middle and late ‘70's. The video is repetitious, it seems too long, there is no narrative, just wreck after wreck, one looking more or less like another. Finally it occurs to me that conservation of momentum, as best I can remember from twelfth grade physics, is manifest in the cars' tendency to continue moving even after they run into something solid. Though this recollection, in retrospect, seems embarrassingly obvious, is it safe to assume that the students on the video made the same inference? After all, their teacher, much as our ARSI expert, provided no commentary. Is this an example of constructivism, of students constructing their own physical knowledge? When the conservation of momentum video is over, our video is over, too. If there was an in-class discussion of what the students had just seen, we didn't get to hear it. Employment of the video seems part of a badly disjointed instructional process. Perhaps the point of all this has been self-evident to the other ARSI expert-aspirants. I am, however, surrounded by nine other adults, all involved in education in one way or another, some with backgrounds in science, but more from administration or higher education. I wonder how many know what conservation of momentum means. Even now, I'm not sure that I do. For all I know, my aforementioned recollection from twelfth grade physics was in error. After all, I may have confused conservation of momentum with "objects in motion tend to stay in motion . . .", or something like that. I wonder how many of the others see the pertinence of a video of serial collisions to understanding conservation of momentum. Were they able to recall or construct their own physical knowledge? Or is this video as bad an instructional tool as it seems to be? The ARSI expert has very little to say about the serial-collision video. For a moment, he seems at a loss. He passes up the opportunity to fault the teacher for not providing an introduction. He says nothing about the absence of a debriefing. Then, belatedly, he calls our attention to the fact that two male students had laughed: "You could see their interest. They weren't just being passive."The expert says nothing more about the video. He has concluded, as far as I can tell, that it demonstrated students' engagement in the process of acquiring a clearer, deeper understanding of "conservation of momentum." Perhaps we really have seen the construction of physical knowledge. My colleagues and I are silent. In truth, the serial collision video seemed like a silly caricature of instruction with audio-visual aids, how to misuse them rather than use them. But the ARSI expert gives no evidence of sharing this view. Training in Fixing Deficiencies: Getting "Down and Dirty"In another instructional video, a white female teacher in her early thirties is standing in front of a class of elementary school students. We are not told the grade, but the children appear to be eight or nine years old. Once again, all the students are white. The classroom is organized in traditional fashion, with individual desks in rows and the teacher standing at the front of the room, her back to the chalkboard. The teacher has said only a few words, the point of her class has not yet become evident, when the ARSI expert interrupts while the video continues to run. He speaks emphatically and with excitement:"Look at her! Look at her clothes! She prepared for this!"In truth, I saw nothing distinctive about the teacher's clothing or appearance. She was dressed modestly, wearing an open jacket with lapels, a white blouse which buttoned at the neck, a just-below-the-knees skirt, and shoes with medium heels. Her clothing was well-suited to working as, say, a bank teller, a receptionist in a family dentist's office, or a casework supervisor in a state social welfare agency. Her hair was cut short, but not extremely so. It was neatly combed, but not stylishly done. She wore make- up, but there was nothing ostentatious or extraordinary about it. She looked like the girl next door, grown up and working for a modest living. But the ARSI expert did not see it that way. The fact that the teacher was presentable counted against her: "She can't get down-and-dirty dressed like that."These observations, coupled with his surmise that the teacher had come prepared to appear on a video, seemed to imbue the ARSI expert with a sense of discovery. His response to the video suggested that, perhaps, he had not seen it before. He was looking for something instructive, and quickly found it in the teacher's appearance, which still seemed unexceptional. He judged a teacher's work as inevitably involving getting "down-and-dirty." Suitable clothes, I concluded, would have been faded jeans, a sweatshirt with holes worn in the elbows, and grass-stained tennis shoes. Why suitable attire for an elementary school teacher should take this form remained a mystery to me, just as the nature of "getting down and dirty" and why it was a pedagogical essential remained unexplained. None of the prospective experts spoke. I saw two give obligatory grins at the "she can't get down-and-dirty dressed like that" judgment. Otherwise, the group was impenetrably difficult to read. Was the lesson clear? Did participants accept it? Did anyone find this informative? Did the ARSI expert know that NSF National Science Education Standards do not include a dress code? Was he aware that teachers' attire is often an issue in rural Appalachian schools because they sometimes dress too informally (Austin, 2000)? Is this what it means to become an ARSI expert? ARSI in West VirginiaARSI's first Program Improvement Review in West Virginia was done in mid-March of 1999. This was also the first time I worked as an ARSI expert. The same was true of my shadowing partner, who was serving as coordinator of our three-school review. Though newly-minted as an ARSI expert, he had long experience in grant writing, program development, and administration of ground-up educational change efforts. Early in his career, he had taught high school science.Adaptation or AdoptionThis Review, moreover, was to be different from those we had seen in Kentucky. It involved three schools rather than one. The schools, an elementary school, a middle school, and a high school, are in close geographical proximity to each other, situated in a low-income, rural district in the state's southern coal fields.In addition, while the ARSI Program Improvement Review was being used as a point of departure, it was not a governing model. The Consistency Rating Summary, replete with Likert items, was still there, but as only one source of information in preparation of a report which was to be tentative, formative, and qualitative. Rather than one-expert school visits, as in Kentucky, there were four evaluators for each school. Most members of each team were newly-minted ARSI experts, who also had training and experience in a variety of pertinent disciplines, including assessment, math education, program evaluation, and administration. Recommendations for improvement were to be made only after discussing the final report with a variety of local stakeholders from the three schools. Stakeholders would participate in the process of actually producing the recommendations. Synthesizing a Final ReportMy task was to synthesize a final report. The Consistency Rating Summary would have left little to synthesize, but its place was not central in West Virginia, as it had been in Kentucky. The materials for synthesis were submitted in manila folders, eleven-by-seventeen envelopes, three-ring notebooks, translucent zip-lock packets, and paper-clipped pages. Consistency Rating Summaries prepared by ARSI experts were included. The Summaries, however, were mixed in with field notes, handwritten reminders, and miscellaneous jottings on single sheets of paper. In addition, each teacher at each school had completed a Consistency Rating Summary, and these, too, had been included.Some ARSI experts' Summaries had conspicuous marginal notes and some did not. Summaries for the same school included and excluded different headings and items. Some included experts' names and some did not. Some had a formal, finished appearance, while others looked like preliminary worksheets. In spite of our plan to make production of recommendations a collaborative effort with stakeholders, a few Summaries included recommendations. All tolled, however, the material did not resemble the output of the sort of mechanically routinized process of thin description we had seen in Kentucky. A Formative Systemic ReportSince our Review involved three schools, a systemic report seemed in order. Furthermore, even though the schools were at three different levels, dramatic cross- school commonalities in traditional educational philosophy and old fashioned, no-nonsense practice made a single report seem fitting. The flexibly formative nature of the process was emphasized in the report's opening paragraphs under the heading "Informed Interpretation from Multiple Perspectives.""A good deal of what we have to say, moreover, is subject to good-faith interpretation and re- interpretation by stakeholders . . ."Similarly, use of the Consistency Rating Summary was placed in context, subsumed by "Judicious use of a Quantitative Rating Summary": ". . . but one source of information for making formative judgments. Its . . . scores . . . merely summarize some of the information used in making our essentially qualitative judgments." A First DraftThe report characterized the math program in each of the schools as traditional, and noted that all adult stakeholders, teachers, administrators, and parents, preferred it that way. Parents were unaware of alternatives. Even some of the teachers were unfamiliar with current terminology and practice. When a newly-minted ARSI expert used the term "rubric," an elementary teacher asked what rubric meant.The schools were autonomous to a fault. Though constituting a rudimentary feeder system, teachers and administrators had no cross-school contact. Insofar as their math curricula were cumulatively compatible, it was due to state and district requirements, and adherence to the same traditional ethos and practices. The report went on for twenty-seven double-spaced pages, addressing topics such as "Avoidance of Innovation," "Cautious Selectivity," "Exclusion of Exploration" Innovations Come and Go," "Traditional Parental Roles," "School-to-School Isolation," and "Staff Development and Teacher Traditionalism." The concluding sections re- emphasized the importance of understanding the report as interpretative and subject to legitimate challenge by stakeholders. Readers were reminded that formulation of recommendations was to be a collaborative effort. "Their Nickel"When I gave this determinedly formative report to my former shadowing partner, still coordinating this first West Virginia Review, his response took me by surprise. Noting the absence of a "Consistency Rating Summary," he said, "it's their nickel." In short, whatever liberties we took with the ARSI model, this remained an ARSI endeavor. ARSI was establishing itself in West Virginia under the institutional auspices of our regional university, and some ARSI expectations had to be met. In response, I used the diverse, unstandardized information which had been submitted, and tried to synthesize a set of defensible Likert item scores for the three-school system. Having attached this to the narrative, I thought the job was done. The coordinator agreed. He submitted a copy to West Virginia's first ARSI Collaborative Director, and scheduled a meeting.Meeting with ARSI OfficialsThe meeting with the ARSI Collaborative Director and an associate began amicably. They had read the report, and they listened with what appeared to be friendly interest as we explained our plans to meet with stakeholders to collaboratively produce recommendations for change. I characterized the approach to Program Improvement Reviews in Kentucky as "take-it-or-leave-it," "expert-centered," "prematurely codified," "top-down," and "quick-and-dirty." The evolving West Virginia approach, by sharp contrast, was "flexible," "client-centered," "qualitatively formative," and "collaborative."The Director responded by noting that there was only one Consistency Rating Summary for three schools. I replied: "Right. Like we said in the report, we took a systemic approach. It made sense, especially since the schools are so much alike."The Director responded that there were no recommendations. I referred again to the report, noting that the recommendations were to be produced collaboratively with school-level stakeholders. The Director, still smiling, shook her head. She said: "The reports are standard. We need Summary scores for each school, and recommendations for each."I responded that I had seen take-it-or-leave-it reports, loaded with misguided Likert-item claims to precision, done all too quickly during shadowing. They were the sorts of reports, I added, that later sat on shelves gathering dust, because stakeholders were not involved in their production. The Director replied: "I'm sorry if that was your experience."The assistant shook her head and murmured unintelligibly. I returned to my characterization of what I had seen in Kentucky, including again "take-it-or-leave-it," "prematurely codified," and "quick-and-dirty." The Director responded: "But that's just your opinion."My shadowing partner intervened. He asserted that he had not expected to do Program Improvement Reviews exactly as they were done in Kentucky. He was especially concerned about formulating recommendations without collaboration with local stakeholders. "They need to be involved in this process. They need a sense of ownership. Otherwise, the report will never be implemented."The ARSI Collaborative Director was not persuaded. She said little, remained unflappable, and would not budge: ARSI Program Improvement Reviews were standard. I asked: "What did you think of the text of the report?"The Director and her assistant both nodded approval. Then the assistant added: "It was long. People are busy . . ." (Followed by a conciliatory, partly muffled chuckle.)I asked: "What's missing from the report as it is now?"The Director repeated that Consistency Rating Summaries and recommendations for each school were essential parts of any ARSI Program Improvement Review report. These, in fact, as submitted by the ARSI experts, are the report. I responded: "So I just clip the three reports together? It's a clerical job?!"By this time, I had lost my composure, while the Director had retained hers. I left the room, acknowledging that ARSI would get the kind of report it wanted. That was the end of my involvement with the Appalachian Rural Systemic Initiative. In RetrospectIt is worth noting that, until our Program Improvement Review, ARSI had kept a low profile in West Virginia. Unknown to me was an earlier series of three meetings with West Virginia educators hosted by ARSI representatives, one of whom is now the ARSI Collaborative Director. According to a participant, a former math teacher who is currently a professor of education and a co-author of this article, the meetings were held January through April of 1998. Her unsolicited invitation to attend described the first meeting as intended to explore "the development of a self-assessment instrument . . . to aid counties in:
A math and science expert from ARSI's Kentucky program presented his state's version of the Consistency Rating Summary for review. Abandoning the "should we do this at all in West Virginia?" attention-getter, ARSI representatives instructed participants to examine each item on the Summary according to unspecified national standards and state-mandated West Virginia regulations. In a February 10, 1998, e-mail from the West Virginia Collaborative Director, participants were told they should decide if Consistency Rating Summary items "should be kept as is, deleted, or modified. You may also generate additional questions." Latitude, yes, but narrowly circumscribed by production of Likert items. Reservations as to whether and how Program Improvement Reviews should be done in West Virginia had been dispelled. The Kentucky model, giving pride of place to the Consistency Rating Summary and check-list guided audits, had been adopted. ConclusionWhen we first thought about writing this article, our main concern was that there is nothing rural about the work of the Appalachian Rural Systemic Initiative. That remains emphatically true. For ARSI, a-school-is-a-school- is-a-school. We concluded, however, that even more important than ARSI's failure to find anything consequentially distinctive about rural education was the characterization we offered at the beginning:ARSI experts construe the process of educational achievement as a thoroughly understood, relatively simple mechanism manifest in static indicators of school effectiveness.How has this come about? ARSI's approach appears to be premised on two assumptions, one following from the other. First, and fundamentally, educational achievement is so thoroughly understood that its practice and evaluation can and should be routinized and streamlined. Second, and as an obvious corollary, well-informed routinization assures not only enhanced achievement, but provides opportunities for cost cutting, as well. If these assumptions, especially the first, were true, educational reform, in low-income rural schools or elsewhere, could be inexpensively programmed and monitored. Schools would become models of efficiency and effectiveness. If the assumptions are false, however, science-based routinization gives way to quick-and-dirty dogmatism, producing contemporary caricatures of early-twentieth century scientific management. As a result, as we have sought to make clear, ARSI's Program Improvement Reviews in low-income, rural schools—or any schools—seem unlikely to enhance achievement in science and math. There may be merit to the claim that high levels of achievement in science and math are far more important today, for individuals and entire societies, than ever before. There may be merit to the claim that the commonsense human capital assumptions which tacitly undergird NSF's systemic initiatives are valid. Whatever the value of these claims, however, it seems unlikely that interventions made by the Appalachian Rural Systemic Initiative will facilitate educational reform and promote economic growth and development. Even if the education-and- development assumptions on which ARSI is premised were undeniably correct, this first-person account casts depressingly serious doubt on ARSI's ability to accomplish its objectives. ReferencesAustin, L. (2000) Some Schools Trying Faculty Dress Codes. Huntington [WV] Herald Dispatch, January: 1A and 9A. Bickel, R. and Spatig, L. (1999) Early Achievement Gains and Poverty-Linked Social Distress: The Case of Post-Head Start Transition. Journal of Social Distress and the Homeless. 8: 241-254. Bowles, S. and Gintis, H. (1976) Schooling in Capitalist America. New York: Basic Books. Brown, D. (1996) Systemic Change and the Role of School Boards. New York: American Educational Research Association Annual Meeting. Consortium for Policy Research in Education (1995) Reforming Science, Mathematics, and Technology Education: NSF's State Systemic Initiatives. New Brunswick, New Jersey: Consortium for Policy Research in Education, Rutgers University. Harmon, H. and Blanton, R. (1997) Strategies for Improving Math and Science Achievement in Rural Appalachia. Tucson, Arizona: National Rural Education Association Annual Convention. Kaestle, C. (1983) Pillars of the Republic. New York: Hill and Wang. McMurrer, D. and Sawhill, I. (1998) Getting Ahead. Washington, D.C.: The Urban Institute. National Council of Teachers of Mathematics (1998) Principles and Standards for School Mathematics. Reston Virginia: National Council of Teachers of Mathematics. National Research Council (1996) National Science Education Standards. Washington, D.C.: National Academy Press. National Science Foundation (1994a) Foundation for the Future. Washington, D.C.: National Science Foundation Directorate for Education and Human Resources. National Science Foundation (1994b) Building the System: Making Science Education Work. Washington, D.C.: National Science Foundation, Directorate for Education and Human Resources. National Science Foundation (1997) Rural Systemic Initiatives in Science, Mathematics, and Technology Education. Washington, D.C.: National Science Foundation, Directorate for Education and Human Resources. National Science Foundation (1998) Systemic Initiative Research. Washington, D.C.: National Science Foundation, Directorate for Education and Human Resources, URL http://www.nsf.gov/pubs/1998/nsf98140/nsf98140.htm National Science Foundation (1999) HER Projects. Washington, D.C.: National Science Foundation, Directorate for Education and Human Resources, URL http://www. ehr.nsf.gov/project. asp National Science Teachers Association (1998) The National Science Education Standards: A Vision for the Improvement of Science Teaching and Learning. Arlington, Virginia: National Science Teachers Association, URL http://www.natt.org/handbook/nses.htm Perkinson, H. (1995) The Imperfect Panacea. New York: McGraw-Hill. Reich, R.( 1992) The Work of Nations. New York: Vintage Books. Senate Committee on Labor and Human Resources (1997) Innovative Workforce Development Initiatives. Washington, D.C.: Congress of the United States, United States Senate. Shapiro, C and Varian, H. (1998) Information Rules. Cambridge, Massachusetts: Harvard Business School Press. Shields, P. (1997) Evaluation of the National Science Foundation's Statewide Systemic Initiatives (SSI) Program: First Year Report. Menlo Park, California: SRI International. Spring, J. (1997) The American School: 1642 to 1996. New York: McGraw-Hill. About the Authors
College of Education and Human Services
Robert Bickel
Robert Bickel is a Professor of Advanced Educational Studies at Marshall University. His recent research is concerned with school size as a variable which moderates the relationship between social class and measured achievement, and with contextual factors which occasion the at-risk designation.
Terry Tomasek
Teresa Hardman Eagle
|
Copyright 2000 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-0211. (602-965-9644). The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu . EPAA Editorial Board
|
