This article has been retrieved times since May 1, 1996.

Education Policy Analysis Archives
Volume 4 Number 7	April 4, 1996	ISSN 1068-2341

---

A peer-reviewed scholarly electronic journal. Editor: Gene V Glass,Glass@ASU.EDU. College of Education, Arizona State University,Tempe AZ 85287-2411 Copyright 1996, the EDUCATION POLICY ANALYSIS ARCHIVES.Permission is hereby granted to copy any article provided that EDU POLICY ANALYSIS ARCHIVES is credited and copies are not sold.

---

Respecting the Evidence:
The Achievement Crisis Remains Real

Lawrence C. Stedman
State University of New York-Binghamton

stedman@binghamton.edu

Abstract: Wherein Stedman answers Berliner and Biddle's reply to his review of The Manufactured Crisis.

"It ain't so much the things we don't know that get us into trouble. It's the things we know that just ain't so." .....Artemus Ward

In his engaging book, HOW WE KNOW WHAT ISN'T SO, the social psychologist Thomas Gilovich offers marvelous insights into the origins of human misconceptions. The problem, he finds, is not irrationality but flawed rationality--the very reasoning mechanisms that help us make sense of reality also lead to questionable beliefs. These include

• the tendency to seek confirmatory information,
• the excessive impact of confirmatory information, and
• the tendency to evaluate evidence in a biased manner.

He explains that "We humans seem to be extremely good at generating ideas, theories, and explanations that have the ring of plausibility. We may be relatively deficient, however, in evaluating and testing our ideas once they are formed" (p. 59).

In a fascinating insight, he notes that people "place a premium on being rational and cognitively consistent" and so rather than simply disregard evidence, they "subtly and carefully 'massage' the evidence to make it consistent with their expectations" (p. 53).

This leads to the illusion of objectivity:

Although people consider their beliefs to be closely tied to relevant evidence, they are generally unaware that the same evidence could be looked at differently, or that there is other, equally pertinent evidence to consider (p. 80).

One fundamental mechanism that gets us into particular trouble is what Gilovich calls "optional stopping":

When the initial evidence supports our preferences, we are generally satisfied and terminate our search; when the initial evidence is hostile, however, we often dig deeper, hoping to find more comforting information, or to uncover reasons to believe that the original evidence was flawed (p. 82).

Or, as he puts it more directly:

I have argued that people often resist the challenge of information that is inconsistent with their beliefs not by ignoring it, but by subjecting it to particularly intense scrutiny (p. 62).

For complex issues, such as the condition of U.S. education and achievement, the desire for consistency outweighs the willingness to respect ambiguity.

For nearly all complex issues, the evidence is fraught with ambiguity and open to alternative interpretation. One way that our desires or preferences serve to resolve these ambiguities in our favor is by keeping our investigative engines running until we uncover information that permits a conclusion that we find comforting (p. 83).

Gilovich has captured well the fundamental failing of the MANUFACTURED CRISIS. Whether Berliner and Biddle are discussing the "myths" about achievement and schools, the power of right- wing disinformation, or the contrast between neoconservative and progressive reforms, they repeatedly offer a one-sided treatment of the evidence. With few exceptions, they accept at face value any information that supports their viewpoint, while they dissect and reinterpret any information that challenges it.

The purpose of academic training and scholarship is to rise above such flawed rationality; to learn how to critically analyze the evidence that supports your own favored arguments--and to treat fairly the evidence that contradicts it. It is also a matter of learning to accept the complexity and ambiguity of evidence--and to fairly present that.

Unfortunately, Berliner and Biddle failed to do this--either in their book or in their response to me. They have even gone beyond the flawed rationality Gilovich describes. They ignored or dismissed entire areas of relevant evidence--such as the extensive data on students' low levels of achievement and knowledge--and, in selectively presenting other evidence--such as the data on test score trends--they winnowed out only that which supported their viewpoint and discarded the rest. In several cases, they have even directly misrepresented the actual data.

What's worse is that they are now resorting to sweeping, disrespectful condemnations of those who disagree with their arguments and point out the limitations of their evidence. They characterize the various critiques of their book as "distorted portrayals and outright lies"; they labeled my analysis a "diatribe" and as "disingenuous" and filled with "lacunae, misrepresentations, and trivialities". They have impugned the motives of reviewers and, in my case, even attributed positions to me that I have never taken.

This, too, is understandable, however. As Gilovich points out, the psychologist Robert Abelson argued that "beliefs are like possessions" and that, consequently, people are "possessive and protective" of them and react defensively when their limitations are pointed out (pp. 86-87). The motivational determinants of belief are particularly powerful. As Sir Francis Bacon put it in the NOVUM ORGANUM, "Man prefers to believe what he prefers to be true" (Gilovich, 1991, p. 75).

THE PURPOSE OF MY EPAA REVIEW

Berliner and Biddle were upset that my review focused on their treatment of the achievement evidence. That was its purpose and should have been immediately obvious from the introductory paragraphs.

Why did I focus on the achievement evidence? Because it underpins their basic argument about a manufactured crisis. They claim that U.S. students and schools are actually doing well and that the evidence to the contrary and beliefs in a crisis have been manufactured by right-wing school critics and administrations. Having already produced a general review of their book back in November in the WASHINGTON POST (one I am sure they must have seen) (Stedman, 1995), I felt it imperative to discuss, at length, in an academic forum, the details of how they treated the evidence on student achievement. EDUCATION WEEK also had devoted a full-page general story about their book back in September (Viadero, 1995).

It should be noted at the outset that even Berliner and Biddle considered such evidence so central to their argument that they spent several chapters trying to explode the myths about the current condition of schools and achievement.

Contrary to their repeated claim in their response, I never stated that "their book" was based on four sweeping claims, but rather that their achievement analysis was. Nevertheless, the review was supposed to have contained the following two introductory sentences, which could have eliminated much of their consternation.

This review is focused on the achievement analysis portion of the MANUFACTURED CRISIS. My more general review of the book can be found in the Education Review section of the Washington Post, Sunday, November 5, 1995, pages 16-17.

OVERVIEW: THE MAJOR FAILINGS OF THEIR ACHIEVEMENT ANALYSIS

The actual evidence on student achievement is crucial to their argument. It directly addresses their claim that U.S. students are achieving well and that the educational crisis has been "manufactured". Instead of systematically reviewing the evidence, they selected a few pieces of data on each topic and reinterpreted them to suit their argument. They concentrated on trends (mostly stable) but ignored levels of achievement (mostly low).

Let me be clear at the outset. I believe that right-wing forces have been attacking the public schools and EXPLOITING the evidence, but there is also extensive, credible evidence that there is a real achievement crisis, something Berliner and Biddle continue to deny. They have still never dealt directly with the actual evidence about low achievement.

Their response to my review repeats and reinforces the book's major failings in its treatment of the achievement evidence. Here's what they did (or did not do) in their analysis.

1. They ignored a large and growing body of research which shows that student achievement has been weak for several decades. Our high school students lack important knowledge in history, civics, geography, and English; they have done poorly in mathematics and science and few write well. The evidence is overwhelming that the achievement crisis is real. In the next section, I report on the latest National Assessment of Educational Progress (NAEP) results.
2. They analyzed the test score decline in a misleading fashion. Although they rightfully criticized the myth of a RECENT general achievement decline, they ignored the 1970s decline and failed to present any of the contradictory evidence from the 1980s. They clearly overstated the case when they claimed "only ONE test, the SAT" ever suggested a decline (p. 35 emphasis original).

   Worse, they then overreached and tried to cast current achievement in an historically positive light. Without the needed evidence, they claimed that this generation of students achieves "substantially" higher than previous ones on "virtually all" commercial standardized tests--a contention that is directly refuted by the major reviews of historical trends on such tests.

   In their response, they compounded their error by arguing that then-and-now studies--including MY review of such research--support such a sweeping contention. They claimed that "almost all" then-and-now studies showed improvement when, in fact, many studies showed no change, several showed declines, and the ones showing improvement typically involved small gains (Stedman & Kaestle, 1991b). They did not mention that such studies have been fraught with problems. They also have never acknowledged that achievement on NAEP HIGH SCHOOL science and civics tests remains lower than in the past, below their 1969 levels.

3. They tried to claim that U.S. failure in international assessments is a "myth", but it is actually partly true. Although our younger students have done well in reading, our older students have done quite poorly in secondary school math and the high school sciences. Here, again, they overreached by claiming that U.S. schools "stack up very well" in the international comparisons. For one thing, they argued that curriculum differences were a major cause of the international achievement differences, but they based this on only one study of outdated 8th grade math data from 1981- 82, and the data did not support their claim. This was a thin reed on which to characterize the standing of the U.S., and even if it had been true, it is still disturbing news for it means the U.S. curricula and programs are not up to international standards. More recent studies also do not support their assertions. In the 1991 IAEP math study, our 8th graders lagged well behind those in nearly all other countries, and this was true even when algebra curricular differences were accounted for.
4. They systematically misrepresented major research studies and data on U.S. achievement.

   a) They graphed standardized test score trends from a study by Linn, Graue, and Sanders (1990), but somehow dropped the very tests and grade levels which included declines! Worse, they offered these data as definitive proof of improving achievement, when in fact, Linn, Graue, and Sanders pointedly remarked that the results were "equivocal" and noted that part of the gains were caused by districts' repeated use of the same tests rather than by genuine improvement. The 1980s back-to- basics movement also helped to artificially raise scores by frequent testing and skill-drill approaches (Stedman & Kaestle, 1991a). In their response, they claimed that the omitted data supports their original claims when, in fact, much of it contradicts them. The data were also outdated, coming from the late 1970s through mid-1980s, and thus are not even relevant to their claims about current students or recent improvement. In a later section, I will discuss their continued mischaracterizations of this study and its data.

   b) They graphed international math scores from a study by Westbury (1992), but somehow left out his 12th grade comparison where the U.S. did poorly! Worse, they disregarded Westbury's caution and improperly compared our elite 8th grade algebra students to the AVERAGE Japanese student. Westbury actually used the top 20%. They claimed it proved that with a COMPARABLE curriculum our students do well in math, but never mentioned that our students spent far more time on algebra (61% vs. 26%), covered more test items, and were one grade older.

   c) They claimed the international assessments have improperly compared the broad mass of U.S. students to an overseas elite attending high-status high schools, but this is old criticism from the early international studies, and it was only partly true even back then. In the early IEA math studies, for example, researchers deliberately sampled college-bound students who were taking math in their senior year of high school--in the U.S. this was an elite group of only 18% of our students; in the second IEA math study, it was only 13%, a similar percentage to that in other countries (Stedman, 1994a).

   d) They attributed the SAT decline to demographic changes in test takers, yet never reviewed the evidence which shows this explains much, but not all, of the decline. They also used AVERAGE SAT scores to claim minority student performance gains, but this masked minority VERBAL declines in the late 1970s and late 1980s.

These are serious, major failings (not molehills) which directly undermine their argument and impugn their credibility as scholars. It is little wonder that they chose to attack me personally rather than deal forthrightly with the evidence.

I have divided this response into several sections:

THE NEW ACHIEVEMENT EVIDENCE--a review of the 1994 NAEP findings which shows that students continue to display serious weaknesses in their knowledge and skills.

THE EVIDENCE AND THEIR RESPONSE--a direct response to their arguments in their reply, organized around their four sweeping claims about U.S. achievement which they continue to support.

THE MANUFACTURED CRISIS REVISITED--a look at several major areas of errors and misrepresentation that were not covered in my original review, in particular their claims of high levels of parental satisfaction with local schools. Here again, they were so intent on fitting the data to their argument, that they distorted the evidence. It turns out that only about a quarter of public school parents rate their oldest child's school an A, while about half of them rate their community's schools C through Fail!

PROGRESSIVE REFORMS AND THE RIGHT-WING AGENDA--an endorsement of much of their reform agenda, coupled with an analysis of their one-sided presentation of a national right- wing agenda, which again demonstrates their Procrustean handling of evidence. In particular, I discuss their treatment of the Sandia Report, which they claimed provided a valid look at the achievement evidence and which they allege was suppressed by the Bush administration.

THE NEW ACHIEVEMENT EVIDENCE

Students are struggling. The depth of the achievement problem is strongly borne out by the latest round of NAEP studies of reading, history, and geography achievement. Performance is reported for basic, proficient, and advanced levels. In 1994, substantial portions of students did not even make the basic level while a majority failed to achieve the proficient level in each subject at each grade level tested: 4th, 8th, and 12th. I also review the results from NAEP's 1992 assessment of writing portfolios, which revealed that little classroom writing is of high quality.

1994 HIGH SCHOOL SENIORS' ACHIEVEMENT

I concentrate here on the data for high school seniors because they provide the best overall assessment of K-12 performance. In reading, a quarter of our seniors failed to reach even the basic level (Williams, Reese, Campbell, Mazzeo, & Phillips, 1995, p. 15). Only about one-third demonstrated reading proficiency (or better). In geography, about a third were below the basic level, while only about a quarter displayed proficiency (or better) (Williams, Reese, Lazer, & Shakrani, 1995, p. 16). History showed the worst results. Over half the seniors were below the basic level and only 11% made the proficient level or higher (Williams, Lazer, Reese, & Carr, 1995, p. 19).

These levels were set by NAEP's independent policy-making body-- the National Assessment Governing Board with "contributions from a wide variety of educators, business and government leaders, and interested citizens" (Williams, Reese, Lazer, & Shakrani, 1995, p. 3).

The reader should recognize that the results are based on the judgments of panels, approved by the Governing Board, of what advanced, proficient, and basic students should know and be able to do in each subject assessed (p. 9).

Concerns have been raised about the construction and interpretation of these levels (Stedman, 1993), and this latest series of NAEP report cards clearly labels them as "developmental" (Williams, Reese, Lazer, & Shakrani, 1995, p. 3). Nevertheless, both the Commissioner of the National Center for Education Statistics and the National Assessment Governing Board believe the levels are "useful and valuable" in reporting on student achievement.

Fortunately, NAEP has returned to their practice of making public sets of test items used in the assessments. This allows educators and the public to appraise the items and evaluate student knowledge directly. The test themselves are quite rich, combining constructed response questions with multiple choice ones. In geography, for example, 60% of the testing time was devoted to constructed response items. The geography and history tests offer a rich panoply of maps, graphs, photographs, cartoons, paintings, and magazine covers. A look at individual items avoids the scaling problems and reveals that many students have serious deficiencies in basic knowledge and skills.

1994 GEOGRAPHY RESULTS

Let's consider the geography results first (Williams, Reese, Lazer, & Shakrani, 1995). Less than half the seniors knew that slavery was a major reason many Caribbean people are of West African descent (p. 63). Only about a third recognized a description of a rain forest and could identify a country that had one. Only about a quarter could identify three or more of the following on a map--the Pyrenees Mountains, the Japanese Archipelago, the Mediterranean Sea, and the Persian Gulf. (And this was after the Persian Gulf War!) Only 10% could interpret a simple bar chart of predicted hydrocarbon emissions and give a reason for the trends displayed.

Relatively stronger results were found for identifying four world cities as major religious centers (76%), identifying shaded countries on a world map as belonging to OPEC (65%), and deciphering interpreting tabular data about two countries (53%- 67%). Still, it should be noted that one-fourth to over one- third of the students had problems with such items.

1994 HISTORY RESULTS

In history, the results were also disturbing (Williams, Lazer, Reese, & Carr, 1995). Only about half of the high school seniors (55%) knew that cotton trade was a main reason Great Britain leaned toward the Confederacy during the Civil War. The other choices were British plantation owners held slaves, most British immigrants lived in the South, and British politicians wanted to conquer the U.S.

Less than half of seniors could identify the purpose of the Monroe Doctrine (41%), date a newspaper report about the Civil War destruction of Charleston (41%), or realized that preventing the spread of communism dominated U.S. foreign policy in the post- war period (47%).

Less than half (47%) could interpret an 1876 magazine cover depicting the "Indian problem" even though general statements were permitted about attitudes or events. Only a third were able to identify a consequence of Nat Turner's slave rebellion (tighter controls on slaves). Only a quarter know that the Camp David accords promoted peace between Egypt and the U.S. (Other choices were the Soviet Union and China; Palestinians and Jordanians; North Korea and the U.S.). Only 15% were able to interpret a simple cartoon showing the long, winding road necessary to spiritually fulfill the civil rights law after enactment.

There were several strong spots. Over 80% properly interpreted two paintings of George Washington as reflecting the glorification of political figures and the use of religious symbols and, in what was hardly a surprising result, 88% knew that the computer rather than the typewriter, superconductor, or radio produced the greatest change in how people worked between 1960 and 1990.

OTHER RECENT NAEP EVIDENCE ABOUT STUDENT PERFORMANCE

Writing is another area that is important particularly given the connection between critical thinking skills and written expression. In 1992, NAEP conducted the first national assessment of writing PORTFOLIOS gathered from classrooms across the country. Such an approach avoids the artificiality and time pressures of using a national sit-down test to judge writing ability. The findings were troubling. Olson (1995) reported that "the best writing that students produce as part of their classroom work is still not very good."

Only between 4-12% of the 8th graders achieved high marks (5 or 6) on the six-category evaluation scale. One-fourth to almost one-half received low marks (1 or 2), depending on whether informative or narrative tasks were being considered. Gary W. Phillips, the associate commissioner at the National Center for Education Statistics, concluded that, "The moral of the story is that the writing is not very good in the nation. Even the best is mediocre."

This may be a bit harsh, however, given that there were writing samples achieving the highest ratings. The portfolio assessment methodology also needs to be systematically and independently evaluated. No doubt, problems will be found that could require some adjustments to the results (up or down). In the meantime, though, the findings suggest there is a serious writing problem and mirror those of the traditional set-task writing assessments that NAEP has conducted, including the one in 1992 (Applebee et al., 1994). High school students have struggled over the years. In 1992, only about 2% to 23% produced "elaborated or better" writing, with the weakest performance on persuasive tasks (Applebee et al., 1994, p. 5). (These are averages across four tasks of each type: persuasive, narrative, and informative. On one informative task, students did much better, 46%; on another much poorer, 6%.) The percentages who produced "developed or better" responses was better but still troubling--only around 16% to half of the students performed acceptably. On most tasks, most students' writing was undeveloped or minimally developed. This mirrors their inadequate writing in prior NAEP assessements (Applebee et al., 1990, p.107; see Stedman, 1993 for information about earlier results and scoring methods.) The good news is that most students have done well with basic mechanics--spelling, grammar, and punctuation--and so additional WHOLE-class drill and practice in these areas is not warranted.

NAEP also did a follow-up analysis of the 1992 reading assessment in which they explored student performance on different kinds of test questions (Olson, 1995). They found a marked drop-off in student understanding and proficiency as the questions became more open-ended and required more elaborated responses. At the three grade levels (4, 8, and 12), performance fell from around two-thirds correct on multiple-choice problems, to slightly above half on short, constructed answer questions, and then to only one-fourth to around a third on questions requiring an extended response.

All of which has important implications as we move toward more authentic assessment. We will most certainly find initially that student performance is even worse than what has been revealed by the more straight-forward, multiple-choice recall testing that has been done primarily so far.

In math, NAEP analysts have determined that "less than half (of high school seniors) appeared to have a firm grasp of seventh- grade content" and only 5 percent "attained a level of performance characterized by algebra and geometry--when most have had some coursework in these subjects" (Mullis et al., 1991b, p. 80). Although high school students have done well on basic operations such as adding whole numbers and reading a line graph (90%+), many have trouble even with simple problems involving fractions, decimals, and percents (Mullis et al., 1991, pp. 302- 309). In 1990, for example, 34% of 17-year-olds could not find the area of a rectangle, given a diagram and the length of two sides (Mullis et al., 1991a, p. 306). Math educators who reviewed the NAEP data in the late 1980s determined that students "exhibit serious gaps in their knowledge and are learning a number of concepts and skills at a superficial level" (Carpenter et al., 1988, pp. 40-41). They concluded that "students' achievement at all age levels shows major deficiencies." Although there have been some modest gains in math achievement in the 1990s, their general conclusions are still appropriate today.

By the way, the NAEP findings I have presented do NOT include dropouts; overall high school student achievement is, therefore, likely to be even worse than this evidence indicates. When we combine these recent results with those from the past several decades, we have a serious cause for concern. (See Stedman, 1993, for a review of this evidence and a discussion of its strengths and limitations.)

BERLINER & BIDDLE'S REJOINDERS

Instead of reviewing this extensive and troubling evidence about low achievement, Berliner and Biddle offered a series of rejoinders in their book about unrealistic standards, our students' focus on breadth of experience, and the nature of the tests. As I explained in my review, however, the achievement standards are realistic (they might even have been set too low), knowledge is an important part of our students' experience, the achievement problems are not an artifact of psychometric scaling, and the tests incorporated real-world tasks and knowledge.

In their response, they took much the same approach. First, they wrote that the "standards against which America's schools are to be judged and found wanting are arbitrary and can be made up as one goes along". Historically, this is untrue. The major studies have not used "arbitrary" or "made up" standards; they have relied strongly on school- and curriculum-based measures-- the textbooks that are most widely used, teacher consensus about what is important to be tested, citizen panels on what students should know and be able to do. Most people would certainly expect high school seniors to have mastered 7th grade math and basic social studies, but they have not.

Berliner and Biddle then suggested that those of us who are concerned about academic achievement are "school bashers" and "standardized test enthusiasts". (I, for one, am neither!) They label the solid evidence that U.S. general knowledge and academic achievement have been low for decades as "Nonsense!" and "ludicrous" (see Stedman, 1993 for a review of the evidence). That is the level of their argumentation--dismissive and mocking, without ever examining the actual evidence. Their primary argument about historically low achievement was the following:

We find it ludicrous that anyone should claim that "academic and general knowledge have been at low levels for decades" in this country. If this were actually true, how on earth did our nation ever manage to win World War II, send astronauts to the moon, create a plethora of new pharmaceuticals, and invent the transistor and virtually all the computer technology now used world wide? For that matter, how did we achieve the world's highest rate of industrial productivity, and establish ourselves as this century's dominant super-power? "Low levels" of academic and general knowledge? What nonsense!

Let's examine this argument. These accomplishments did not depend upon the MASS of U.S. students and adults being well- informed and knowledgeable. Instead, they exemplify the prowess of the military-industrial complex in post-war America, the skills of a narrow technical elite, and the inventiveness of a single individual or group of individuals.

It took a Jonas Salk to develop the polio vaccine, for example. The transistor was invented by John Bardeen, Walter Brattain, and William Shockley. (This is the same Shockley who later espoused racially-charged ideas about intelligence being genetically determined.) The micro-computer revolution can be largely credited to three school dropouts--Steve Jobs and Steve Wozniak who developed the Apple II computer and Bill Gates who founded Microsoft.

In other words, such accomplishments have readily existed alongside low levels of knowledge and achievement in the general population. Our citizens' lack of knowledge of civics, history, geography, and literature, for example, had little bearing on our winning World War II or getting a man to the moon. (Let us also be careful lest we believe that it is only Americans who have discovered pharmaceuticals or that only a U.S. education was involved. Penicillin, for example, was developed by Alexander Fleming, but he was a Scottish biologist. Streptomycin was discovered by the American Selman Waksman, but he was born in Russia in 1888. The oral form of the polio vaccine was developed by the Polish-American Edward Sabin, born in 1906. Many of these discoverers, therefore, were educated well before World War II, long before the decades of low achievement that I was talking about!)

I find it curious that Berliner and Biddle have unwittingly embraced here a Human Capital view of economic productivity and military-corporate power, a view that they critique at great lengths in their book! According to their new argument, students' general knowledge and academic achievement have been the keys to U.S. economic and technical accomplishment!

In their book, however, Berliner and Biddle gave only a passing nod to the importance of knowledge and cultural heritage- -even for social and civic reasons. Yet it is important that students be well informed about the key events, people, issues, literary works, and social struggles that have shaped our multicultural society. Such information matters--it helps us as voters, workers, readers, newswatchers, and community members. In a society torn by debates over immigration and affirmative action, we all should be alarmed by how little our students know of world cultures and how poorly informed they are about our country's tortured racial history.

The low levels of achievement also are unimpressive results for 12 years of schooling. The tests do measure much of what is being taught in our schools and show we are not succeeding in our efforts. This is the heart of the achievement crisis. A complex, democratic society needs a well-read and knowledgeable citizenry and yet the evidence shows we are not accomplishing this.

THE EVIDENCE AND BERLINER & BIDDLE'S RESPONSE

SWEEPING CLAIM #1: "TODAY'S STUDENTS ARE OUT-ACHIEVING THEIR PARENTS SUBSTANTIALLY"

Their treatment of the achievement evidence continues to be one-sided. In their response, they wrote that "we were actually quite cautious in what we claimed about the achievements of students and their parents." That claim contrasts strikingly with what they actually stated in their book about standardized test trends. They claimed that "virtually all of them would show that today's students are out-achieving their parents substantially" (p. 33). Not some of them, but virtually all of them. Not somewhat outperforming, but substantially. As I noted in my review, they did not present the evidence needed to support this sweeping generational claim; they failed to discuss the many reviews of historical trends that refute it.

They then had the amazing chutzpah to cite my own research on then-and-now studies to try to prove their claim. Note first, that they did not cite this research in their book, but are only bringing it in now, after the fact. Next, notice what they claimed I found:

Additionally, when one looks at more than 20 "then" and "now studies of student achievement--reviewed previously by Stedman himself in his studies of literacy in the U. S.!-- almost all the results show that the students taking the test "now" outscore the students that took the test "then."

They claim that "almost all the results" showed improvement. In fact, of the 13 local then-and-now studies done through the 1960s, seven showed no real change, including two that showed declines. Two of three then-and-now studies done in the 1970s showed declines relative to earlier students. Overall, across the century, more studies had gains than declines, but the gains were small and many trends were stable. The studies also suffered from a variety of flaws. Here's how Carl Kaestle and I (1991b) actually summarized our findings:

If one takes age into account, more of the tests showed gains than declines, whereas many others showed approximately equal performance rates. But few of the studies were nationally representative. And the magnitude of the changes, up or down, was usually half a school year or less--a shift that can easily be attributed to the margin or error caused by the problems we have described (p. 89).

We then concluded:

Our educated guess is that schoolchildren of the same age and socioeconomic status have been performing at similar levels throughout most of the twentieth century (we consider the 1970s in detail in Chapter 4). But we also caution that then-and-now studies are fraught with design and interpretation problems; reliance upon them to support arguments about literacy trends is unjustified (p. 89).

This illustrates well their treatment of evidence--a misrepresentation of findings and other scholars' research, a continued effort to fit the evidence to their argument, and a failure to acknowledge the complexity and problems with the data.

Note as well that they completely disregarded one of the major conclusions of our literacy research. By focusing on trends, they again ignored the findings about the levels or depth of the achievement and illiteracy problems. We wrote:

Does this mean that things are rosy on the literacy front? Certainly not. The functional-literacy tests showed that a substantial portion of the population, from 20 to 30 percent, has difficulty coping with common reading tasks and materials. The job literacy measures, for all their limitations, show that there are substantial mismatches between many workers' literacy skills and the reading demands of their jobs. Even if schools are performing about as well as they have in the past, they have never excelled at educating minorities and the poor or at teaching higher-order skills (p. 128).

As I pointed out in my review, Berliner and Biddle selectively presented evidence on recent trends in commercial test scores, specifically data from a study by Linn, Graue, and Sanders. Remarkably, in presenting the data, they omitted the very grades and tests that showed declines and only graphed those that showed gains! They also never mentioned that the researchers had determined that the test increases were partly caused by districts' repeated use of the same tests rather than by genuine improvement.

Their explanation of their selectivity is a curious one--and should have been presented in their book, not after the fact now! First, as to their omission of SRA data--which showed reading and math declines in several grades--they argued that the SRA data are "complex and mixed, and we judged that they required too much explanation to warrant their inclusion in a book designed for general readers". That is both unscholarly and an insult to readers. They were, in fact, able to describe the data in a only few sentences in their response. It would have been easy for them to have included an extra bar in their graph covering the SRA data. They ponder: "What on earth would readers have gained had we displayed these data in TMC?" That is the nub of their problematic treatment of evidence. Readers would have gotten an honest and more complete look at the elementary school data. SRA reading scores, for example, declined in 5 of the 8 elementary school grades!

Their characterization of the data also varies, depending upon whether they are trying to support their case or discredit other researcher's positions. In their book (p. 31), for example, they described annual gains of 2 percentile points on commercial tests as "large"; yet in their footnote in their response, when they are trying to discount the significance of the SRA data, they labeled annual reading declines of 1.5 percentile points as "tiny"! (Note as well that half the "gains" they did graph were under 1.5 points!)

Second, as to their omission of high school data--which also showed some declines and where gains were less impressive--they now explain that they omitted them because high school students show less growth in academic subjects--yet wasn't that worth presenting?--and that Linn, Graue, and Sanders did not include CTBS and ITBS high school data. This is a weak excuse. A scholar interested in presenting a thorough picture would have gotten the CTBS high school data, while ITBS doesn't even go the high school level! (Riverside Publishing uses the ITED for high school students.) Furthermore, why not present the data that was at hand?

The difficulty may have been that results would not have fit their thesis as well. On the MAT, reading scores were up in 9th grade, but they declined in grades 10, 11, and 12. On the SRA, grades 11 and 12 showed declines in both reading and math. Overall, the CAT and Stanford showed annual gains of only around 1 percentile point in reading and math, much less than the elementary school scores. Given such mixed evidence, it is misleading, therefore, for them to claim that the "high school data SUPPORT our assertions" (emphasis original!).

Their characterizations of specific high school data were also questionable--as to the MAT math scores, they wrote "ALL four high school grades provided evidence of increased scores in mathematics" (emphasis original) when in fact, 9th graders showed no change! As to MAT reading scores, they wrote, "The MAT reading tests generated mixed data for these four grades: scores were up in two grades, but scores were down in two others". As we have noted, however, scores in the last three grades 10-12 actually declined, by -.7, -.4, and -.7. Such repeated errors lead one to distrust their analysis. It should be noted, as well, that they never informed the reader that they were graphing only elementary school data--instead, they presented it as if were generally representative of student achievement, when it was not.

Finally, they still have not acknowledged that K-8 test score increases should not be simply equated with improvement in achievement. The Lake WoeBeGone phenomenon of repeated test administrations and teaching-to-the test is too well- established to be ignored. Furthermore, the 1980s back-to-basics movement also helped to artificially raise students' scores by emphasizing frequent testing and skill-drill approaches (Stedman & Kaestle, 1991a). Berliner and Biddle's conclusion, however, continues their overall sweeping characterization of the data and this study: "So, student achievement is UP on commercial tests, and that is exactly what we concluded."

One final note--this evidence is outdated, so it does not support claims about current achievement trends! The renorming data covered the period from the late 1970s through the mid 1980s. The CAT test, for example, came from a 1978-1985 renorming. The CTBS data came from a 1987 renorming. The data, therefore, is not recent, but refers to trends from over a decade ago!

SWEEPING CLAIM #2: ONLY THE SAT EVER SHOWED A DECLINE

Berliner and Biddle were right to challenge the mythology that we are currently in a massive, general decline. We are not. But they went well beyond that in their own assertions. They wrote, "The two of us know of only ONE test, the SAT, that ever suggested such a decline" (p. 35). That is a sweeping claim and one that is unsupported by the evidence.

As I pointed out in my review, many major tests showed declines, particularly in the 1970s and at the high school level. These declines electrified portions of the legislative, educational, and public communities--they led to major investigations, including the College Board's ON FURTHER EXAMINATION (Wirtz, 1977). While conservative critics may have exaggerated their significance, the declines did occur and to claim otherwise misleads readers. Unfortunately, they did not discuss this evidence in their response--or explain their claim.

Scholars have a responsibility to present the full story, particularly contradictory evidence. Although trends have generally been stable, there are important exceptions. Berliner and Biddle never mentioned in their book that high school students' NAEP science and civics scores remain below their 1969 level, that high school reading scores fell in the late 1980s on several tests, and that the SRA tests showed reading and math declines at several grades.

Their attempt now to discredit this evidence is curious. I noted that HIGH SCHOOL students' NAEP science and civics scores had declined substantially in the 1969-1976 period. They tried to challenge this with RECENT data from 9- and 13-year olds! That was hardly relevant to my original comment. High school students' scores are also a more important indicator of performance as they reflect the entire K-12 experience.

I also noted that high school students' civics scores slipped in the late 1980s, something they took issue with. In NAEP's report, THE CIVICS REPORT CARD, however, analysts noted "Seventeen-year-olds participating in the 1988 assessment performed significantly less well than their counterparts assessed in either 1976 or 1982" (Anderson et al., 1990, p. 13).

And my judgment about science trends is not "simply wrong!" as they gleefully exclaimed. I stated that HIGH SCHOOL students' science scores "fell during the 1970s and have only partly rebounded". They even presented the data that bears me out in their response--17-year-olds had a scale score of 305 in 1969 (not 1970) and it dropped steadily to 283 by 1982--this was a substantial drop of about a half a standard deviation. By 1992, it had recovered to 294, or only about half the way back.

There was also some slippage in reading and writing scores, particularly for younger students. 9-year-olds dropped six scale points in NAEP reading achievement between 1980 and 1990 while 8th graders dropped 10 scale points in writing proficiency between 1984 and 1990. The latest reading assessment showed that 4th graders had dropped a minor three scale points between 1992 and 1994, while 12th graders had dropped five (Williams, Reese, Campbell, Mazzeo, & Phillips, 1995, p.7).

Berliner and Biddle argued that "Stedman's interpretation of the data is once again wrong! He sees a decline in reading scores when he should be seeing remarkable consistency of scores over time." This is far-fetched. I am no supporter of the decline thesis--as they well know--and stated so quite clearly in my review. In my general review of achievement trends (Stedman, 1993), which I cited in support of my comments, I wrote:

I begin with literacy because it undergirds academic performance and is a perennial concern of educators. Here, a picture is worth a thousand words (see Figure 1). The picture for NAEP writing performance is similar to that for reading: both have remained basically stable for more than two decades (p. 216).

They also claimed that I ignored the accomplishments of schools "in the face of escalating social problems", yet in my EPAA review of their book, I wrote:

Given changing school populations and societal conditions, generally stable scores are still a remarkable accomplishment for U.S. schools. This is an important message that the public needs to hear.

Such severe distortions and misrepresentations do them no credit.

THE SAT DECLINE

Finally, there is the SAT decline itself. Here again, they attributed to me a position I did not take. They know I am no fan of the SAT; I have described it as an "irrelevant measure" of educational quality and national achievement (Stedman, 1994b). Others disagree, however, and so it remains of interest. Indeed, its national prominence is one reason they dealt with it. My concern again is their unscholarly and one-sided treatment of the evidence. The first problem was that they attributed the SAT decline to demographic changes in test takers, such as increases in minority students, yet never reviewed the research!

The major investigations have concluded that the SAT decline was not entirely compositional (Stedman, 1993; Stedman & Kaestle, 1991). The tremendous rise in minority test-takers, for example, cannot explain the large decline in WHITE students' SAT scores during the 1960s and 1970s. During one stretch, the pool of test takers did not expand, yet scores still declined. This suggests that, to some extent, there was a real decline in performance.

The most comprehensive analysis of the demographic changes-- the College Board's special Advisory Panel study published in 1977 (Wirtz, 1977)--concluded that much of the 1960s decline, from 2/3rds to 3/4ths, but a smaller part of the 1970s decline, up to 30%, was due to demographic changes in test takers. (They reviewed a vast array of demographic indicators.) If one considers the additional effects of age (students were getting younger) and birth order (younger siblings score more poorly), up to one-half of the 1970s decline may have been due to compositional changes. The Advisory Panel attributed the remaining portion to an UNDETERMINED combination of school and societal factors.

They may have misgivings about such research, but it was incumbent upon them to acknowledge its existence.

Curiously, in spite of their misgivings about SAT scores themselves, they chose to use them to claim that minority students gained in achievement in recent decades. They even went so far as to present a bar graph of SAT scores by minority groups to document their claim. The problem, as I pointed out, was that they used AVERAGE SAT scores which masked minority verbal declines in the late 1970s and late 1980s (Stedman, 1994b). Here again, I find it remarkable that when an error is pointed out, they do not discuss the evidence pertaining to it. Instead, they again attributed a position to me that I have never taken--that the SAT is as meaningful a barometer as NAEP. Why can they not gracefully acknowledge contradictory evidence or their errors?

(It should also be noted that they essentially set up something of a straw man argument about the decline in their book. Several of the leading conservative critics have NOT focused on the decline for some time--these educators recognize and have acknowledged that scores recently have been stable. The so-called "myth" is no longer one in certain quarters.)

SWEEPING CLAIM #3: U.S. STUDENTS "STACK UP VERY WELL" IN INTERNATIONAL COMPARISONS

The first problem here is that the so-called "myth" of U.S. international failure is actually partly true. U.S. international performance has been dismal in secondary school mathematics and poor in several high school sciences. As I explained in my major review of the international assessments, these are real results and not an artifact caused by sampling or curricular-test bias (Stedman, 1994a). Berliner and Biddle, however, do not accept ANY evidence that shows U.S. achievement in a negative light.

The second problem is that they failed to review and summarize the findings about U.S. achievement from the major international assessments. This would have led readers to a very different conclusion about the current state of U.S. international performance. As I noted in my review, our students have "done well in reading and elementary school science, middling to poor in geography and secondary school science, and last or near-last in mathematics." That is a fair and balanced characterization of the international findings and shows that critics who make sweeping claims about a GENERAL U.S. failure are mistaken, but so are reviewers such as Berliner and Biddle who try to cast the international findings only in a positive light.

Curiously, they now write that they decided against presenting these findings because the international validity problems are so great. Yet this did not prevent them from making sweeping claims about the findings such as "Many, perhaps most, of the studies' results were generated by differences in curricula" (p. 63). A more scholarly approach, particularly for the general public, would have been to have presented the overall findings and then discussed their strengths and limitations. Nor did they present any counter-arguments or counter-evidence to their sweeping assertions about validity (I review their claims below; see also Stedman, 1994a).

The third problem is that Berliner and Biddle went well beyond challenging the mythology of a general U.S. international failure and reinterpreted selective evidence into a highly positive, one- sided view. They wrote that "American schools stack up very well" (p. 63), the international evidence "confirms impressive strengths of American education" (p. 64), and when opportunities to learn are considered, "American students' school achievement looks quite similar to that of students from other countries" (p. 58). Such sweeping contentions would not have been supportable by a general review of the international research.

WESTBURY STUDY AND THE PRINCIPLE OF CONTROL REVISITED

One of their most egregious examples of reinterpreting evidence was their handling of Westbury's (1992) study, which was their major piece of "evidence" about curricular opportunities-to- learn. Comparing U.S. algebra students to the average Japanese student, however, violated their own research precept--the Principle of Control. As they put it,

to estimate the true effect of a factor using survey data one MUST control, in the analysis, for the effects of other crucial factors that can affect the relationship. Trained data analysts are very aware of this principle--indeed, it one of the first things taught in courses on statistics (p. 159).

Clearly, U.S. students who take algebra in the 8th grade are a unique, elite group with marked advantages in college expectations, math interest, parental support, social class, and academic ethic. Consequently, one cannot tell how much of their achievement reflects the effects of their curriculum and how much their background advantages. The comparison is, therefore, inappropriate and unwarranted and was specifically cautioned against by Westbury himself (1992).

Furthermore, our algebra students actually had a more focused algebra program--they had spent 61% of their time on it compared to only 26% for Japanese students. They also had covered more test items and were one grade older. So even the curricula--or opportunities to learn--were not similar as Berliner and Biddle asserted. (They also labeled the data as "achievement scores" when in fact it was only algebra scores.)

In general, Berliner and Biddle argued that 8th grade math comparisons have been unfair because, unlike students in other countries, most of OUR students do not take algebra in the 8th grade. Algebra items, however, make up only part of the international tests, and the results are virtually the same whether they are included or not. In the 1991 IAEP-2 math study, for example, the U.S. still would have scored BELOW the international average and trailed the leading countries by 16 to 18 percentage points (Lapointe, Mead, & Askew, 1992, pp. 39, 146).

Their response to me was baffling: "Somehow Stedman takes this simple demonstration of the effects of differences in curricula and opportunity-to-learn and converts it into a series of assertions that we did not make in TMC and do not believe."

As discussed, this was anything but a "simple demonstration" of curriculum differences; in fact, it was quite flawed. Furthermore, I have to ask: What "series of assertions"? I simply discussed Westbury's actual methods and findings that pertained to THEIR opportunity-to-learn claim and noted that they failed to discuss the 12th grade results which showed U.S. students at a serious mathematical disadvantage--even after curricular differences had been taken into account! As I discussed in my review of the international assessments (Stedman, 1994a), curriculum differences and opportunity-to-learn can only explain part of the U.S. international achievement deficiency. Furthermore, the lack of U.S. curriculum coverage, particularly in mathematics, often reflects our less demanding and weaker academic program, and so does not excuse our low achievement.

By the way, Berliner and Biddle also violated the Principle of Control in their public vs. private schools graph--p. 123--when they showed that public school students who take advanced math courses slightly outperformed private school students. This does NOT prove, however, as they asserted, that the public-private difference is simply a matter of curriculum--the public school advanced math takers are a select, elite group. Here again, they failed to disentangle curriculum and class effects. Furthermore, although their graph came from AFT research reported by Albert Shanker (1991, p. 10), they never mentioned that he concluded that both sectors were achieving poorly! (Although I agree with their general point that the private vs. public school achievement gap has been overblown, I wouldn't characterize the gaps as generally "small" as they did--in the 1990s NAEP comparisons they have often been substantial, but probably not that much more than would be expected given that private schools have a more upscale student body. I also think that Shanker's conclusion is an intriguing one that is well worth exploring further.)

VALIDITY AND SAMPLING BIAS IN THE INTERNATIONAL ASSESSMENTS

Finally, they offered a series of arguments about the appropriateness and validity of the international assessments which are not supportable. In the first one, Berliner and Biddle are caught in a Catch-22. They argue that the international tests have not measured "the unique values and strengths of American education", including "creativity, initiative, and independence of thought in students"--yet at the same time, their book criticizes today's schools for lacking these very features. They are clearly concerned that neoconservative strategies, such as work intensification and national standards, are dominating schooling and propose numerous progressive alternatives (cooperative learning, project method, etc.) designed to rectify the situation and enhance creativity and initiative.

There is also a certain hubris in asserting that "American" education is "uniquely" focused on such things. As I noted, Japanese elementary students have rich curricular and extra- curricular activities--calligraphy, sewing, hands-on math and science activities, group problem-solving, electronics, dance, musical training, play, reading, physical exercise, cooperative learning, school jobs, etc. Without explanation, however, they labeled this as one of my "stranger" assertions! Furthermore, our breath of focus hardly excuses our low levels of achievement and knowledge--our schools, parents, and policy makers all clearly value high levels of achievement.

They also argued that sampling bias is a major problem for the international assessments, claiming that the assessments compare the broad mass of U.S. high school students to select samples in high-status high schools overseas (p. 54). Others have claimed similarly that our average student was compared to an elite, university-bound group of European students. This is an old criticism, however, emerging out of the first round of IEA international assessments in 1964 and 1970-71. Even then, the severity of the sampling problem varied by country and subject. In mathematics, the assessment deliberately sampled seniors who were taking math as part of a college-preparatory sequence. This narrowed the U.S. selection to college-bound students (only 18%) and thus avoided an unwarranted mass-to-elite comparison.

Their claim is even less applicable to the second international IEA math study, where many countries had 12th grade math enrollment rates similar to that of the U.S. (which was only 13%). Furthermore, most of these countries outperformed the U.S. by a considerable margin (Stedman, 1994a). Even some of the countries with higher enrollment rates matched or outperformed the United States. Hungary, for example, scored about the same as the small U.S. elite in several areas even though it enrolled half its students! In the second international science study in the mid-1980s, the U.S. actually had more selective 12th grade enrollments than most countries and still achieved more poorly in chemistry, physics, and biology. (Their example of a Japanese teacher's comments about sampling problems is a red herring. It has nothing to do with the major international assessments--IEA or ETS's IAEP.)

Critics have made too much of the variations in high school enrollments. Most of the assessments have involved 9- to 14-year- olds, ages when education is compulsory in developed countries and nearly 100% of the students are represented. Unfortunately, these are also the ages where the U.S. has struggled in several subjects.

On another point. I, too, am concerned about the newsmedia's inadequate coverage of the international assessments, but that does not prove that U.S. schools "stack up very well".

One of the worst features of Berliner and Biddle's response is that they repeatedly retreat from or even misrepresent their own position! As to variability, they now claim:

Stedman asserts that we had argued that overall variability in achievement among students should be greater in our country, but we did not argue for such an effect.

Yet, there's what they wrote in their book:

Together these two problems [disparities in student wealth and inequities in funding] mean that scholastic achievements will vary far more in the United States than in other countries (p. 58).

and

To state this issue succinctly, the achievement of students from American schools is a LOT more variable than is students achievement from elsewhere (p. 58, emphasis original).

As I noted, the evidence does not bear out this sweeping contention. In fact, the 1991 IAEP math and science studies showed our variability was similar to that of other nations and less than that of Taiwan and Korea, the leading performers.

I have no trouble with the implication of the states-to-nation comparison they presented. Clearly, there are enormous regional variations in U.S. achievement and it is always useful to look at disaggregations of data for other patterns. What I was concerned about was their failure to inform the reader that this comparison had been labeled "experimental" and was technically problematic. (Contrary to their assertions in their response, they did not report in their book the details of the data or how the comparison was conducted!) Furthermore, when even our best state scores (those from a few typically high-scoring mid-Western states), are only at the AVERAGE level of Taiwan and Korea, we have cause for concern. Both aggregated and disaggregated scores indicate a serious problem in mathematics.

Finally, although minority and low-income students achieve relatively poorly, that remains insufficient to explain our generally low achievement. As I explained, the math deficit is not simply a minority student problem. In 1992, only 30% of WHITE U.S. 8th graders demonstrated NAEP math proficiency while over a quarter did not even make the basic level. Nor are our problems due to low-achievers. Even our top half have not kept pace internationally in math and science (Stedman, 1994a). Why do their "minds boggle" over such straight-forward explanations?

Instead of dealing with this evidence, they twisted my explanation into an argument that I claimed the low scores of minority students had no impact on average scores! Which is, of course, ridiculous. The point is that a major math problem and gap remains even when one looks at (disaggregates) other portions of the data--such as white students and the top half. It is also worth noting that, with the same demographics, U.S. reading scores are quite strong internationally.

Berliner and Biddle should have admitted that they selectively reviewed the international evidence, presenting only a couple of scattered pieces that supported their viewpoint. I invite readers to read my comprehensive analysis of the international assessments, in which I report the major findings and discuss the assessments' strengths and weaknesses (Stedman, 1994a).

SWEEPING CLAIM #4: THE EDUCATIONAL CRISIS IS MANUFACTURED

In addition, Stedman asserts that we made another "sweeping claim," that "the general education crisis is [merely] a right-wing fabrication," although he provides no citation to justify this charge. Again, this misrepresents what we wrote.

This is remarkable. This claim of theirs--that the general education crisis is not real and was manufactured by right-wing forces--is one of the central arguments of their entire book.

My review, however, was not focused on their political assertions but rather on their claim that the achievement crisis is a myth. Hence, my title "The Achievement Crisis is Real" and my extensive review of the achievement evidence in my section, "Low Achievement".

Let me be clear. I believe that right-wing forces have been attacking the public schools and EXPLOITING the evidence (and have been aided by a mix of social forces), but there is also extensive, credible evidence that there is a real achievement crisis, something Berliner and Biddle continue to deny. They have still never dealt directly with the actual evidence about low achievement.

Nevertheless, let us consider their charge. Note here that they had to add the word "merely" to my quote before discussing it. Does my statement really misrepresent what they wrote?

Let's quote and cite them from several places. First, begin with the title: THE MANUFACTURED CRISIS. Manufactured? By whom? Well, as they stated in their response "right-wing ideologues gained access to the White House with the election of Ronald Reagan, and in our book we detailed their influence on White House education policy." Here's how they explained the manufactured crisis and the lack of real evidence:

We began our book by noting that throughout most of the Reagan and Bush years, the White House led an unprecedented and energetic attack on America's public schools, making extravagant and false claims about the supposed failures of those schools, and arguing that those claims were backed by "evidence." . . .

No such White House attack on public education had ever before appeared in American history--indeed, even in the depths of the Nixon years the White House had not told such lies about our schools. Since the attack was well organized and was led by such powerful persons-- and since its charges were shortly to be echoed in other broadsides by leading industrialists and media pundits--its false claims have been accepted by many, many Americans. And these falsehoods have generated a host of poor policy decisions that have damaged the lives of hard-working educators and innocent students. In our book we labeled this attack "The Manufactured Crisis".

Ironically, they claimed that I was the one that was reducing complex realities to a "political slogan"!

In the introduction to their book, they point quite clearly to "organized malevolence" and "nasty lies," and alleged that "government officials and their allies were ignoring, suppressing, and distorting evidence" (xi). In their chapter 4, "Why Now?", they laid out their case that right-wing forces have manufactured the crisis, and titled various sections "The Entitlement of Reactionary Voices", "The Far Right", "The Religious Right", "The Neoconservatives" and "School-basing and Governmental Scapegoating". They argued that,

Early in the 1970s, however, a number of wealthy people with sharply reactionary ideas began to work together to promote a right-wing agenda in America (p. 133).

. . . these foundations have undertaken various activities to "sell" reactionary views: funding right- wing student newspapers, internships, and endowed chairs for right-wing spokespersons on American campuses. . . lobbying for reactionary programs and ideologues in the federal Congress (p. 133).

They were quite clear in arguing that the "Manufactured Crisis was not merely an accidental set of events or a product of impersonal social forces" (p. 9) but involved a "serious campaign by identifiable persons to sell Americans the false idea their public schools were failing and that because of this failure the nation was at peril."

They themselves, therefore, have made it quite clear that they believe that the achievement crisis was a right-wing fabrication.

THE MANUFACTURED CRISIS REVISITED

In my review, I only touched the tip of the iceberg as far as their errors and distorted evidence went. One of the most egregious examples of misleading and selective presentation was their handling of opinion data on schools. It is worth exploring at length for it is both a crucial piece of evidence and argumentation and illustrates how they select confirmatory evidence and ignore disconfirmatory.

PARENTAL (DIS)SATISFACTION WITH THE SCHOOLS

In a compelling comparison, Berliner and Biddle pointed out that opinion about the national status of education, which was supposedly influenced by the conservative assault, is negative, and then claimed that parents' judgments of their community's and children's schools, which were supposedly based on local information, are quite positive. Here we have an important piece of evidence that goes right to the heart of their argument about a manufactured crisis. Berliner and Biddle argued that the negative opinions about national conditions are "stereotypic" reflecting "rumors" and "bad portrayals" in the "popular press" and are, in essence, manufactured by right-wing neoconservative critics, whereas the positive opinions about local schools are based on "personal experience, direct observation, informed judgment, and discussions with others" (p. 112). In particular, parents of school-age children will have "first-hand, direct knowledge" and their opinions are "more likely to reflect reality." Thus, according to this argument, our schools are actually in good shape because that's what parents and local opinion says.

At one level, this is a very curious argument for them to be making given their interest in sweeping educational changes. If it were true, it spells disaster for their own reform agenda. It would mean that parents are quite satisfied with what is going in their local schools and there would be little justification for progressive reforms.

Before reviewing the actual data, let us consider a different perspective on why opinion about local schools might be more positive. Andrew Coulson (1994) makes an intriguing counter- argument--namely, that citizens are better informed about the national condition of education than they are about the local one. Every few years, for example, the National Assessment of Educational Progress reports on students' knowledge and skills in major academic areas--history, civics, geography, reading, mathematics, writing, etc.--and the findings are widely distributed in the media. It could well be that, if parents had the same kind of detailed achievement information about local students' knowledge and performance, they would be just as critical of their local schools.

I think Coulson is on to something. Few parents ever visit classrooms, particularly at the high school level, or shadow students throughout a day; few have ever actually observed what goes on inside the schools. Few districts routinely gather and report to the local media and community information about what students know and can do. In most communities, there is no systematic testing and reporting of high school students' knowledge in the key academic subjects. (I am referring here to curriculum-based exams in Algebra II, English Literature, U.S. History, Civics, Spanish 2, etc. and not generic, commercial standardized tests of reading, math, and social studies that are sometimes reported.)

If the results on such exams were regularly reported, and if parents routinely spent time in classrooms during the day, judgments of local schools could well be more negative. (Similarly, if parents were familiar with the many ethnographies of school conditions that were produced over the past decade, they might be decidedly more critical of their local schools.)

My primary concern here, though, is with the actual evidence and how Berliner and Biddle presented it. For over 25 years, Gallup and Phi Delta Kappa have surveyed the educational opinions of a national representative sample of adults, including public school parents. They have repeatedly asked respondents to rate the schools on the A, B, C, D, and Fail grading scale.

Berliner and Biddle used this data to claim that public school parents are "well satisfied with their schools" and "rate them highly" (p. 114). But, in presenting the data, they combined A and B ratings, which thus inflated the positive ratings, and omitted grades of C entirely! Their graph of parental opinion was an unusual one, therefore, in that it contrasted A/B ratings with D/F ratings and left out Cs entirely (p. 113). The result was a skewed comparison. (Their graph also contained a error-- what they labeled as the adult sample's opinion of local schools was actually that from respondents with no children in schools!)

Contrary to their selective approach, I here present tables of the 1993 results complete with each of the grades, A through Fail, so that readers can inspect them (Elam, Rose, & Gallup, 1993). The first table gives the ratings by all respondents, the second gives the ratings of public school parents.

1993 RATINGS--ALL RESPONDENTS

	A	B	C	D	Fail	Don't know
Nation's Public Schools	2	17	48	17	4	12
Public Schools in this community	10	37	31	11	4	7

1993 RATINGS--PUBLIC SCHOOL PARENTS

	A	B	C	D	Fail	Don't know
Nation's Public Schools	3	16	49	17	4	11
Public Schools in this community	12	44	28	12	4	<.5
School your oldest child attends (did not specify public or private)	27	45	18	5	2	3

So what do we find? Public school parents certainly do rate local schools more highly than national ones--fewer Cs, Ds, and Fails, and more As and Bs. But look closely at the data. Only about a QUARTER of public school parents rate their oldest child's school an A, which is hardly a ringing endorsement. A quarter apparently have serious concerns about it, rating it C through Fail. (By 1995, this percentage had grown to over a third; see Elam & Rose, 1995). Furthermore, almost half the public school parents (44%) in 1993 expressed some displeasure with their community's schools, rating them C through Fail. (By 1995, this figure had grown to exactly half.)

Nonpublic school parents' responses were particularly revealing as the next table shows. They were quite critical of their local public schools. About 2/3 rated them C through Fail. Although one might argue that they are less familiar with the public schools, one could conversely argue that the reason they became private school parents is because they know all too well what local schools are like.

1993 RATINGS--NONPUBLIC SCHOOL PARENTS

	A	B	C	D	Fail	Don't know
Nation's Public Schools	6	9	48	15	12	10
Public Schools in this community	5	32	41	9	11	2

All this data hardly suggests that "American parents" are "well satisfied with their local schools" as Berliner and Biddle argued (p. 114).

Berliner and Biddle compounded the distortions by then claiming that

What is amazing is that this high level of parental satisfaction with their local schools is growing and is actually HIGHER today than it was seven years ago (p. 112).

Although "satisfaction" (As & Bs) grew in the late 1980s, ratings in the 1990s leveled off. In fact, 1993 ratings were a point lower than those of 1991, and 1992 ratings were a point lower than those of 1986. By 1995, ratings had fallen back to 1986 levels (Elam & Rose, 1995).

In any event, how do they explain these trends? They don't bother to. A conservative critic might argue, however, that the reason satisfaction grew in the 1980s was because schools went back to the basics, raised standards, improved discipline, etc., but this interpretation is not considered by Berliner and Biddle. Interestingly, the increases in parental satisfaction took place in the aftermath of reforms generated by A NATION AT RISK. Was this a reflection of real improvement? Or of national activity and publicity influencing local opinions?

WHO IS SITTING IN JUDGMENT?

Berliner and Biddle condemn several prominent educators for mistrusting positive parental opinion about their local schools. They wrote:

Who are Doyle, Ravitch, Finn, and Stevenson to tell them they are wrong? (p. 114)

In effect, these critics have proclaimed themselves part of an elite who, for the good of the nation, will be pleased to tell other Americans what they are to believe and how they are to act (p. 114).

But isn't that exactly what Berliner and Biddle have done in their 414 page book as they lay out a progressive reform agenda and critique the conservative approach, one that turns out to have much parental support?

Why do Berliner and Biddle only respect--and present-- parental opinion when it suits them and not respect it--or discuss it--in other areas? The PDK/Gallup opinion study that Berliner and Biddle relied on (Elam, Rose, & Gallup, 1993) reported that the overwhelming majority of respondents have favored, for a long time, national achievement goals and standards, requiring a standard exam to get a high school diploma, and using national tests to compare communities' achievement.

Other parental opinions also ran counter to their (and my!) preferred approach. In 1993, two-thirds of PUBLIC SCHOOL PARENTS favored English immersion for language minority students or even instruction at parents' expenses over bilingual education. Half supported longer school years. In 1995, three-fourths of public school parents favored a constitutional amendment to allow prayers to be spoken in public schools (Elam & Rose, 1995). These results were similar to those from 1984. Most preferred a moment of silence for silent prayer or contemplation rather than spoken prayer.

The 1995 poll also shows that parents continue to strongly support national exams and standards. Over 80% of public school parents support higher standards in the major academic subjects for promotion and for graduation. About 60% favor them even if it meant "significantly fewer students would graduate". About three-fourths even favor setting standards for kindergarten through 3rd grade. About two-thirds of public school parents favor using standardized, NATIONAL exams for promotion in THEIR OWN community schools.

Such parental opinions do not simply reflect the national conservative hegemony that emerged in the last decade during the Reagan and Bush administrations. Although support for such measures as national testing grew a bit in the 1980s, it has a long history (Elam, Rose, & Gallup, 1993). Way back in 1970, people were advocating NATIONAL tests to measure their community's achievement and, even in the mid-1970s, most were advocating that all students be required to pass a standard exam to receive a high school diploma--and this was well before the conservative onslaught occurred that Berliner and Biddle labeled the MANUFACTURED CRISIS. So the issue of parental opinion is a complex one.

This past year, the Phi Delta Kappa/Gallup poll explored the reasons parents rated their local schools higher than the nation's (Elam & Rose, 1995). Their answers were striking and challenge Berliner and Biddle's complacency about academic achievement. Given a list of 11 possible reasons, Elam and Rose reported that the parents made a "significant number-one choice: THE LOCAL SCHOOLS PLACE MORE EMPHASIS ON HIGH ACADEMIC ACHIEVEMENT" (p. 43, emphasis original). So, if Berliner and Biddle are right that local parents are in the know about their public schools, then they should also respect their opinions about emphasizing academic achievement.

One limitation of this finding, however, is that the parents generally agreed with each of the choices they were offered--with one notable exception, that their children's schools were better because they had more to spend per pupil. That exception has relevance for the next section.

PROBLEMS IN LAKE WOEBEGONE

If parents truly were satisfied with their schools, it would undermine Berliner and Biddle's case for reform. So they had to find some support in the data for their reform agenda. PDK/Gallup asked respondents an open-ended question: "What do you think are the biggest problems with which the public schools of this community must deal?" Here's how Berliner and Biddle characterized the findings:

In fact, the biggest complaint that American parents indicated in the 1993 Gallup poll was that their local schools were not supported adequately. This complaint took precedence over their concerns about drug abuse, lack of discipline, fighting, violence, gangs, and a host of other real and imagined problems (p. 114).

This neatly fits the basic argument Berliner and Biddle are advancing, but is truly misleading. THERE WAS NO CONSENSUS IN PARENTAL OPINIONS ABOUT SCHOOL PROBLEMS. A lack of proper financial support was the most often mentioned problem, but ONLY 24% of the public school parents cited that. THE VAST MAJORITY CITED OTHER PROBLEMS. It is unclear that funding took "precedence" over other problems. Respondents were not asked to rank problems. Almost half of them (43%) were concerned about issues of order and behavior--15% cited discipline, 14% drugs, and 14% fighting, violence, and gangs.

I find it curious that they would label some of the problems "imagined". Why were they suddenly discounting certain parental opinions, given that it is supposedly informed opinion? (Interestingly, 10% of the public school parents reported they had no idea what the biggest problems were.) They didn't mention that those without children in school responded similarly to public school parents, which further undermines their argument about locally-informed opinions.

It is likely that 1991-1993 concerns over finances were partly influenced by national happenings--the 1992 Bush-Clinton election campaign that focused in part of support for education and Jonathan Kozol's book SAVAGE INEQUALITIES--rather than simply the "reality" of the local situation. The survey itself may also have played a part in inducing financial concerns in that there was a series of questions about educational expenditures--equal funding, the impact of money, support for poor communities, etc. (One hopes that those questions came after the question about biggest problems.)

By 1995, the mention of financial support had dropped in half to only 12%. A lack of discipline was mentioned just about as often (11%). Had local conditions changed so dramatically? Had schools suddenly received adequate funding? Or, had the national debate shifted?

Berliner and Biddle identified the opinions about problems as those of "parents"--but it was actually parents with children currently in the public schools. Parents of nonpublic students made different, and quite intriguing, comments about their community's public schools. A lack of proper financial support was NOT the problem they most often mentioned in 1993 (or 1995). Instead, they were most often concerned about a lack of discipline in the local schools (19%), the standards and quality of education (18%), and fighting, violence, and gangs (17%). Although Berliner and Biddle ignored them, their opinions about local schools are worth listening to as they were the ones who decided to remove their children from those schools--or not put them there in the first place.

Opinions about public schools and reform, I believe, reflect a complex, highly tangled interaction of parental experience with local schools, the spirited national debate over educational reform, and a growing conservative hegemony.

Instead of recognizing these complex influences on parental opinion, instead of respecting the opinions of all parents, it was far simpler for them to set up false dichotomies--parents vs. nonparents, national illusions vs. local realities, and manufactured crisis vs. high satisfaction.

In the end, the "problem" became those without children. Berliner and Biddle commented about public school parents:

The major problem they face is trying to persuade those who do not have children in the schools to agree to pay their share of school taxes (p. 114).

Such a sweeping comment flies in the face of the very survey they were reporting on (Elam, Rose &, Gallup, 1993). Two-thirds of the respondents WITHOUT children in school said they would be willing pay more taxes to improve the quality of public schools in poorer states and communities. That figure closely matches the 71% of public school parents who said they'd be willing. 59% of those without children in school said they'd be willing to pay more federal taxes to improve inner-city schools, just about the same as the 62% of public school parents. Those without children in school also gave similar responses as to the local schools' biggest problems--although a lack of proper financial support was first, drugs, discipline, and violence together garnered the lion's share of the concerns.

Berliner and Biddle then concluded their discussion of parental opinion with:

Perhaps it is time for citizens without children to join parents and go into the schools to see for themselves what is actually happening there (p. 114).

Perhaps it is time for both groups (along with educational researchers) to do just that!

The main point I am making in this section is that opinions about local schools are nowhere near as strong as Berliner and Biddle argue--one can hardly describe it as "remarkable degree of consumer satisfaction" (p. 113) when half the public school parents are rating their community's schools C through Fail. What it suggests to me is that there is a deep well of dissatisfaction that could be enlisted in a movement toward progressive reform. But we must understand and respect the fact that public school parents have many conservative ideas about schooling and reform, shaped by national forces (and conservative propaganda) but grounded as well in local experiences.

PROGRESSIVE REFORMS AND THE RIGHT-WING AGENDA

There should be little question that I basically agree with Berliner and Biddle's reform mission. As I wrote in the WASHINGTON POST review,

Berliner and Biddle offer a welcome critique of the neoconservative agenda--privatization, national testing, gifted programs, and work intensification. They forcefully document the social problems plaguing our schools--from economic stagnation to poverty--and provide a useful compendium of alternative reform strategies--small schools, authentic assessment, equitable funding, and community involvement.

As a progressive educator, therefore, I'm sympathetic to their concerns. The ascendancy of the political right is troubling and could harm public education greatly. We do need to overhaul school financing systems and do more for low-income rural and urban students. We do need to critically examine neoconservative reform strategies and aggressively promote progressive alternatives.

Ultimately, though, the book suffers from being one-sided. While right-wing "organized malevolence" and government suppression of evidence make for good reading, they do not mean the educational crisis is a myth.

Berliner and Biddle were so intent, for example, on branding the major 1980s reform reports as ideologically conservative, that they even tarred thoughtful critiques of the schools by progressive educators. Their list of reports, for example, that were supposedly products of conservative ideologies and Human Capital theories included A PLACE CALLED SCHOOL by John Goodlad and HORACE'S COMPROMISE by Ted Sizer (p. 140).

THE SUPPRESSION OF THE SANDIA REPORT

They were more on target when they described how the conservative political agenda shaped the Department of Education's WHAT WORKS? reports and how self-interested budget considerations may have led NSF to stand by a flawed study predicting a national shortfall of scientists (pp. 162-164). But then they went further and, without evidence, suggested that NSF stood its ground because the Reagan administration was interested in helping industrialists (p. 165).

In a more dramatic tale, they also alleged the Bush administration suppressed a major study of education-- the Sandia Report--because it contradicted official claims about the poor state of education, and would have set the achievement record straight (pp. 165-168). This story is an important one because the report formed the basis of several well-known articles challenging the notion of an educational crisis (see, e.g., Bracey, 1991; 1992) and Berliner and Biddle extolled its virtues (pp. 26, 354).

The report was rife with errors, however, which helped delay its publication and they overlooked its substantial shortcomings-- sloppy analysis of the SAT and international data and omission of key achievement data (Stedman, 1994b).

The allegation of suppression is a serious one and potentially libelous. Berliner and Biddle had an obligation to furnish the evidence for such charges IN their book and, in the interest of fairness, present alternative interpretations of the events-- particularly giving the viewpoints of those charged with suppressing. This they did not do. They simply alleged that administration officials subjected the report to "unprecedented" NCES and NSF reviews, yet it seems that the reports' authors were involved in requesting the reviews. In 1993, one of the authors, Robert Huelskamp, wrote that, "As our work unfolded in the spring of 1991, WE SUBJECTED a draft to peer review with the U.S. Department of Education, the National Science Foundation, and other researchers (most notably Gerald Bracey)" (Huelskamp, 1993, p. 719, emphasis added).

It has struck many observers as reasonable that a report on education created by Department of Energy analysts--not by educators--should be reviewed by education researchers at the National Center for Education Statistics, people who would be more conversant with the data. Berliner and Biddle offered no evidence that such a review was unprecedented (nor did the source they relied on--Tanner, 1993); indeed a major Energy report on the general condition of K-12 public schooling was itself something unprecedented. As one of its authors noted, it was a departure from previous efforts that had focused on analyses of postsecondary education and the training of scientists and mathematicians (Huelskamp, 1993, pp. 718-719).

Berliner and Biddle also wrote that "the report itself eventually appeared in the JOURNAL OF EDUCATIONAL RESEARCH--without fanfare, without even a listing of its authors!" (p. 159). In fact, Huelskamp (1993) first published a version of the report in PHI DELTA KAPPAN, one of the largest circulating educational journals, and informed readers that the "full report will be published in the May/June issue of the JOURNAL OF EDUCATIONAL RESEARCH" (p. 719). Furthermore, the entire issue of JER was devoted to the report and its front cover listed the authors' names--C.C. Carson, R. M. Huelskamp, and T.D. Woddall--in bold print!

Even though it took time for the final report to be released, its ideas were widely circulated much earlier. The authors themselves distributed drafts of the report even before the summer 1991 NCES and NSF reviews were completed (Miller, 1991, p. 32). Gerald Bracey (1991) used them as the basis of his first annual report on the condition of education that appeared in PHI DELTA KAPPAN back in 1991, an article that received widespread publicity, and he later credited them with helping change conservative critics' views of the achievement decline (Bracey, 1992). The report's authors also testified to Congress in the summer of 1991 and the printed testimony, including a synopsis of the report, was readily available (HEARINGS ON THE STATE OF EDUCATION, 1991).

To be sure, the entire episode is quite controversial. Miller (1991) reported that unnamed sources contended the authors were worried about possible reprisals (funding cut-offs), a GAO audit was conducted, several politically-charged statements were revised out of the draft, etc. Several sources did charge that the report was being buried because it conflicted with Bush administration educational policy and that the Congressional testimony was needed to get the message out. Administration officials countered that the report was delayed because it was undergoing an expert review process.

Whether it was suppressed, buried, delayed, or legitimately subjected to additional reviews (or several of the above!), such actions do not mean that the report's findings were valid and should be accepted. Berliner and Biddle claimed that NCES and NSF reviewers "dutifully detected trivial 'flaws' " (p. 167) ), but like Tanner (1993), they did not present the reviewers' findings or what was concluded about the nature and extent of the flaws. In fact, the reviewers raised serious, fundamental questions about the quality of the report, its data handling, and its conclusions.

(Tanner argued that the reviewers were opinionated and provided one example where some reviewer had unprofessionally written "Nuts" next to a passage on a Sandia draft (p. 292)-but a blunt opinion hardly invalidates what many reviewers found or what the summaries of the reviews concluded).

In his summary of NCES's review, Emerson Elliott (1991), the commissioner of NCES, described the problems as follows:

The report appears to be highly selective in the information it presents. Information that is widely known and understood is not presented, and the data shown are consistently supportive of a picture of U.S. education in a positive light. This could give rise to criticisms that the report is a biased presentation instead of the "balanced" presentation that has been claimed.

. . . the trends in educational performance among U.S. students are complex and not well-represented in this analysis. The oversimplification leads to simplistic interpretations.

In many places in the report the findings and interpretations are not supported by the data presented.

. . . the results of the science examinations in the NAEP are provided. The assertion is made that the trends shown are consistent with the results of exams in other subject matter areas. This is not the case, as demonstrated in numerous analyses of NAEP and other achievement data.

The discussion of international comparisons on test scores reflects this problem as well. Many other international comparisons have been made, and some of the issues identified in the issue discussion on p. 94 have been addressed in studies. These findings should have been included for a more balanced discussion of U.S. student performance.

A longitudinal component over the course of a year permitted comparison of what students were actually taught during a year and how they performed on those test items. The U.S. performance, unfortunately, was rather dismal.

He concluded that the report contains:

assertions that contradict what we know well from broadly grounded research conducted over a number of years with repeated replications using different databases

misinterpretations of the data presented

inappropriate policy conclusions [and]

conclusions not well founded in the information presented.

The NSF review determined that "the report rests on a partial and flawed analysis" and that its conclusions are "not adequately supported" (House, 1991). The NSF reviewers (several not just one as Berliner and Biddle suggested) found "several major flaws" typified by a "lack of understanding of the data series used" and "unresolved conflicting interpretations" (House, 1991). They noted there were "dozens of flaws" and gave many examples, including the Sandia analysts' sweeping claim there wasn't ANY NAEP test that showed declines and their failure to recognize students' low achievement levels on the tests.

My own review concluded that the report was

generally right about steady trends, but that it is seriously flawed by errors in analysis, insufficient evidence, mischaracterizations of the international data, and a failure to consider the evidence that U.S. students are performing at low levels. In spite of its findings, fundamental school reform is still warranted (Stedman, 1994b).

Interested readers can find a detailed treatment of the report's strengths and limitations in Stedman (1994b).

SHAPING A PROGRESSIVE REFORM AGENDA

Berliner and Biddle also characterized the present national agenda as right-wing and neoconservative, but it was developed across the political and educational spectrum--by governors of both parties, teacher union leaders, and state school superintendents. While right-leaning, it contains a complex mixture of reforms. Even the national Goals 2000 program includes such long-time progressive objectives as parental participation and ensuring children come to school ready to learn.

Let me be clear. I have no doubt that right-wing forces have organized an assault on the public schools; that conservative school critics exploited the evidence and exaggerated the decline. I was, for example, an early critic of the NATION AT RISK for misusing data, exaggerating the decline, and ignoring equity issues (Stedman & Smith, 1983; see also Stedman & Kaestle, 1985). But just as conservative critics were wrong to argue that we were in a massive decline and needed to return to traditional schooling, so too, progressives such as Berliner and Biddle are now wrong to suggest that our schools are achieving well and that concerns about students' levels of knowledge are unfounded.

As I explained in my WASHINGTON POST review (Stedman, 1995), progressives should be willing to admit that achievement is low. But that does not mean embracing a conservative agenda or calling for the U.S. to be #1 in the world in math and science, as the nation's Goals 2000 program does. Nor does it mean calling for the schools to go back to old-fashioned, regimented teaching. The existing curriculum is already too facts-based and memory- driven and is not working. As I wrote in the POST review:

An historical perspective helps here. Conservatives often blame the decline of excellence on 1960s liberalism, but students' achievement and general knowledge were low even in the 1940s and 1950s--a clear indication traditional practices have never been very successful. Such persistent failure strengthens the case for a sweeping, progressive restructuring of schools.

Berliner and Biddle, therefore, missed a great opportunity to strengthen their own case for progressive reform. By combining the progressives' call for cooperative learning and rich curricula along with the conservatives' emphasis on high levels of knowledge, we would be far more likely to develop reflective, well-informed students. (Note as well that thoughtful conservatives are also calling for innovative teaching methods, an engaging, challenging curriculum, and an end to tracking.) A far more compelling case for reform could be made--and one that could garner more universal support--when we explain that traditional methods have failed and that even children of the middle-class are often not mastering important academic knowledge.

I invite readers to compare my analyses of the condition of educational achievement with theirs (see bibliography). Judge for yourselves who has produced the balanced, careful treatment of the data; who is willing to acknowledge the complexity of the data and achievement patterns, and who is working hard at understanding the evidence rather than trying to fit it into one neat, pat story. Although we should be concerned about the growing influence of right-wing politics, let us also respect the evidence; the achievement crisis remains real and the need for fundamental school reform remains great.

References

Anderson, L., Jenkins, L., Leming, J., MacDonald, W., Mullis, I., Turner, M., & Wooster, J. (1990). THE CIVICS REPORT CARD. Princeton, N.J.: Educational Testing Service.

Applebee, A. N., Langer, J. A., Mullis, I. V. S., & Jenkins, L. B. (1990). THE WRITING REPORT CARD, 1984-1988. Princeton, N.J.: NAEP.

Applebee, A., Langer, J., Mullis, I., Latham, A., & Gentile, C. (1994). NAEP 1992 WRITING REPORT CARD. Washington, D.C.: National Center for Education Statistics.

Berliner, D., & Biddle, B. (1995). THE MANUFACTURED CRISIS: MYTHS, FRAUD, AND THE ATTACK ON AMERICA'S PUBLIC SCHOOLS. New York: Addison-Wesley.

Berliner, D., & Biddle, B. (1996). Making molehills out of molehills: Reply to Lawrence Stedman's review of THE MANUFACTURED CRISIS. EDUCATION POLICY ANALYSIS ARCHIVES, 4(3). http://seamonkey.ed.asu.edu/epaa/

Bracey, G. (1991). Why can't they be like we were? PHI DELTA KAPPAN (October), 105-117.

Bracey, G. (1992). The second Bracey report on the condition of public education. PHI DELTA KAPPAN (October), 104-117.

Carpenter, T. P., Lindquist, M. M., Brown, C. A., Kouba, V. L., Silver, E. A., & Swafford, J. O. (1988). Results of the fourth NAEP assessment of mathematics. ARITHMETIC TEACHER (December), 38-41.

Carson, C. C., Huelskamp, R. M., & Woodall, R. D. (1993). Perspectives on Education in America. THE JOURNAL OF EDUCATIONAL RESEARCH, 86 (May/June), 259-310.

Coulson, A. J. (1994). A Response to John Covaleskie. EDUCATION POLICY ANALYSIS ARCHIVES, 2(12). http://seamonkey.ed.asu.edu/epaa/

Elam, S., & Rose, L. (1993). The 27th annual Phi Delta Kappa/Gallup Poll of the Public's attitudes toward the public schools. PHI DELTA KAPPAN, 77(1), 41-56.

Elam, S., Rose, L., & Gallup, A. (1993). The 25th annual Phi Delta Kappa/Gallup Poll of the Public's attitudes toward the public schools. PHI DELTA KAPPAN, 75(2), 137-152.

Elliott, E. (1991). Review of the Sandia National Laboratory report on education. Letter to Richard E. Stephens, Associate Director for University and Science Education, Department of Energy from the Acting Commissioner, National Center for Education Statistics. Washington, D.C.: U.S. Department of Education.

Gilovich, T. (1991). HOW WE KNOW WHAT ISN'T SO: THE FALLIBILITY OF HUMAN REASON IN EVERYDAY LIFE. New York: The Free Press.

HEARINGS ON THE STATE OF EDUCATION. (1991). Hearings before the Subcommittee on Elementary, Secondary, and Vocational Education of the Committee on Education and Labor, House of Representatives, One Hundred Second Congress. Hearings in Washington, D.C., May 1 and 3, and July 18, 1991. Serial No. 102-28. Washington, D.C.: U.S. Government Printing Office. ISBN 0-16-035543-5.

House, P. (1991). Review of the Sandia National Laboratory report on education. Letter to Richard E. Stephens, Associate Director for University and Science Education, Department of Energy, from the Director of the Division of Policy Research and Analysis, NSF. Washington, D.C.: National Science Foundation.

Huelskamp, R. (1993). Perspectives on education in America. PHI DELTA KAPPAN, 74(9), 718-722.

Kozol, J. (1991). SAVAGE INEQUALITIES: CHILDREN IN AMERICA'S SCHOOLS. New York: Crown Publishing.

Lapointe, A., Mead, N., & Askew, J. (1992). LEARNING MATHEMATICS. Princeton, New Jersey: Educational Testing Service.

Linn, R. L., Graue, M. E., & Sanders, N. M. (1990). Comparing state and district results to national norms: The validity of claims that "everyone is above average". EDUCATIONAL MEASUREMENT: ISSUES AND PRACTICE (Fall), 5-14.

Miller, J. (1991, October 9). Report questioning 'crisis' in education triggers an uproar. EDUCATION WEEK, p. 1, 32.

Mullis, I. V. S., Dossey, J. A., Foertsch, M. A., Jones, L. R., & Gentile, C. A. (1991a). TRENDS IN ACADEMIC PROGRESS. Washington, D.C.: U.S. Government Printing Office. (ED 338 720)

Mullis, I. V. S., Dossey, J. A., Owen, E. H., & Phillips, G. W. (1991b). THE STATE OF MATHEMATICS ACHIEVEMENT. Washington, D.C.: U.S. Department of Education.

Olson, L. (1995, February 8). Students' best writing needs work, study shows. EDUCATION WEEK, 5.

Shanker, A. (1991, Fall). Do private schools outperform public schools? AMERICAN EDUCATOR, 8-15, 40-41.

Stedman, L. C. (1993). The condition of education: Why school reformers are on the right track. PHI DELTA KAPPAN, 75 (3), 215-225.

Stedman, L. C. (1994a). Incomplete explanations: The case of U.S. performance in the international assessments of education. EDUCATIONAL RESEARCHER, 23(7), 24-32.

Stedman, L. C. (1994b). The Sandia Report and U.S. achievement: An assessment. JOURNAL OF EDUCATIONAL RESEARCH (January- February), 133-146.

Stedman, L. C. (1995, November 5). Putting the system to the test. [Review of THE MANUFACTURED CRISIS.] Education Review section of the Washington Post, 16-17.

Stedman, L. C. (1996). The achievement crisis is real. [Review of THE MANUFACTURED CRISIS.] EDUCATION POLICY ANALYSIS ARCHIVES 4(1). http://seamonkey.ed.asu.edu/epaa/

Stedman, L. C., & Kaestle, C. F. (1985). The test score decline is over: Now what? PHI DELTA KAPPAN, 67(3), 204-210.

Stedman, L. C., & Kaestle, C. F. (1991a). The great test score decline: A closer look. In C. F. Kaestle, H. Damon-Moore, L. C. Stedman, K. Tinsley, & W. V. Trollinger (Eds.), LITERACY IN THE UNITED STATES (Chapter 4). New Haven: Yale University Press.

Stedman, L. C., & Kaestle, C. F. (1991b). Literacy and Reading Performance in the United States from 1880 to the present. In C. F. Kaestle, H. Damon-Moore, L. C. Stedman, K. Tinsley, & W. V. Trollinger (Eds.), Literacy in the United States (Chapter 3). New Haven: Yale University Press.

Stedman, L. C., & Smith, M. S. (1983). Recent reform proposals for American education. CONTEMPORARY EDUCATION REVIEW, 2(2), 85-104.

Tanner, D. (1993). A nation 'truly' at risk. PHI DELTA KAPPAN, 75(4), 288-297.

Viadero, D. (1995, September 13). Book that bucks negative view of schools stirs debate. EDUCATION WEEK, p. 8.

Westbury, I. (1992). Comparing American and Japanese achievement: Is the United States really a low achiever? EDUCATIONAL RESEARCHER, 21(5), 18-24.

Williams, P., Lazer, S., Reese, C., & Carr, P. (1995). NAEP 1994 HISTORY: A FIRST LOOK. Washington, D.C.: U.S. Department of Education.

Williams, P., Reese, C., Campbell, J., Mazzeo, J., & Phillips, G. (1995). NAEP 1994 READING: A FIRST LOOK. Washington, D.C.: U.S. Department of Education.

Williams, P., Reese, C., Lazer, S, & Shakrani, S. (1995). NAEP 1994 GEOGRAPHY: A FIRST LOOK. Washington, D.C.: U.S. Department of Education.

Wirtz, W. et al. (1977). ON FURTHER EXAMINATION. New York: College Board.

About the Author

Lawrence C. Stedman

stedman@binghamton.edu

Lawrence C. Stedman is Associate Professor of Education at the State University of New York at Binghamton. His Ph.D. is from the University of Wisconsin at Madison in Educational Policy Studies with a minor in Sociology. He has worked as a school district policy analyst, secondary school teacher, VISTA volunteer, and educational researcher. He has a keen interest in equal opportunity and school reform. His dissertation and early articles centered on effective schools research and the reform reports of the early 1980s. He has helped evaluate ESL, minority achievement, merit pay, and dropout intervention programs.

More recently, his research has focused on the general condition of education and its implications for policy-making. He has written articles on the test score decline, literacy trends, the international assessments, and the Sandia Report. He is currently investigating historical trends in students' and adults' general knowledge. It is the outgrowth of a book he helped author with Carl Kaestle and others on the history of the U.S. reading public (Literacy in the United States: Readers and Reading Since 1880, Yale University Press, 1991). This new research has been funded by a SUNY Faculty Research Grant and Fellowship and by a National Academy of Education Spencer Foundation post-doctoal fellowship.

---

Copyright 1996 by the Education Policy Analysis Archives

EPAA can be accessed either by visiting one of its several archived forms or by subscribing to the LISTSERV known as EPAA at LISTSERV@asu.edu. (To subscribe, send an email letter to LISTSERV@asu.edu whose sole contents are SUB EPAA your-name.) As articles are published by the Archives, they are sent immediately to the EPAA subscribers and simultaneously archived in three forms. Articles are archived on EPAA as individual files under the name of the author and the Volume and article number. For example, the article by Stephen Kemmis in Volume 1, Number 1 of the Archives can be retrieved by sending an e-mail letter to LISTSERV@asu.edu and making the single line in the letter read GET KEMMIS V1N1 F=MAIL. For a table of contents of the entire ARCHIVES, send the following e-mail message to LISTSERV@asu.edu: INDEX EPAA F=MAIL, that is, send an e-mail letter and make its single line read INDEX EPAA F=MAIL.

The World Wide Web address for the Education Policy Analysis Archives is http://seamonkey.ed.asu.edu/

Education Policy Analysis Archives are "gophered" in the directory Campus-Wide Information at the gopher server INFO.ASU.EDU.

To receive a publication guide for submitting articles, see the EPAA World Wide Web site or send an e-mail letter to LISTSERV@asu.edu and include the single line GET EPAA PUBGUIDE F=MAIL. It will be sent to you by return e-mail. General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, Glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. (602-965-2692)

Editorial Board

John Covaleskie jcovales@nmu.edu	Andrew Coulson andrewco@ix.netcom.com
Alan Davis adavis@castle.cudenver.edu	Mark E. Fetler mfetler@ctc.ca.gov
Thomas F. Green tfgreen@mailbox.syr.edu	Alison I. Griffith agriffith@edu.yorku.ca
Arlen Gullickson gullickson@gw.wmich.edu	Ernest R. House ernie.house@colorado.edu
Aimee Howley ess016@marshall.wvnet.edu	Craig B. Howley u56e3@wvnvm.bitnet
William Hunter hunter@acs.ucalgary.ca	Richard M. Jaeger rmjaeger@iris.uncg.edu
Benjamin Levin levin@ccu.umanitoba.ca	Thomas Mauhs-Pugh thomas.mauhs-pugh@dartmouth.edu
Dewayne Matthews dm@wiche.edu	Mary P. McKeown iadmpm@asuvm.inre.asu.edu
Les McLean lmclean@oise.on.ca	Susan Bobbitt Nolen sunolen@u.washington.edu
Anne L. Pemberton apembert@pen.k12.va.us	Hugh G. Petrie prohugh@ubvms.cc.buffalo.edu
Richard C. Richardson richard.richardson@asu.edu	Anthony G. Rud Jr. rud@sage.cc.purdue.edu
Dennis Sayers dmsayers@ucdavis.edu	Jay Scribner jayscrib@tenet.edu
Robert Stonehill rstonehi@inet.ed.gov	Robert T. Stout stout@asu.edu