All may not be right with education in Texas. But neither is it all wrong, as Walter Haney would have everyone believe, judging by his article "The Myth of the Texas Miracle In Education."(Note 1) Haney wastes no space in getting to his main conclusion. In the first paragraph of the introduction he asserts that "In this article, I review evidence to show that the "miracle" of education reform in Texas is really a myth and illusion."(Note 2) However, he generously invites each reader to arrive at his or her own decision as to whether he was fair in arriving at his conclusions. "I leave it to others to judge how fair-minded I have been in recounting this version of the Texas miracle."(Note 3)

There is no attempt here to deal with all of the issues raised by Prof. Haney. In fact, only one issue is dealt with, but it is the one that is central to his thesis, namely whether or to what extent increases in dropout rates in Texas were caused by the Texas Assessment of Academic Skills (TAAS) exit test and the extent to which any increase in dropouts resulted in an unwarranted increase in the calculated pass rate on that test. At a minimum, this is a good example of how two different analysts can draw opposite conclusions from the same data.

Haney asserted at numerous points in his article, and elsewhere, that the TAAS exit test directly resulted in an increase in dropouts, which in turn inflated apparent increases in pass rates on the exit test between the years 1994 to 1998.

Typical of statements attributed to Haney include the following: "I would guess at least half of the apparent increases are a mirage resulting from increasing numbers of students being excluded from test results—either because they dropped out of school or they've been misclassified as special education students.(Note 4) "The Texas miracle in education is a myth," said Walter Haney, a Boston College researcher who studies test statistics. Texas schools, he said, have some of the nation's highest dropout rates, and the system of accountability that Bush touts helps drive tens of thousands of students, mostly minorities, to quit school each year—a loss that in turn boosts test scores, he said.(Note 5)

The Haney article, as published in the Education Policy Analysis Archives is a distillation of his two-year effort as an expert—and presumably paid—witness for the Mexican American Legal Defense and Education Fund (MALDEF) in their suit against the State of Texas.(Note 6) The plaintiffs claimed that the exit test in Texas, first administered to students in the spring of their tenth grade, is unfair and discriminates against minority students. The goal of the suit was to prevent the State of Texas from continuing to make passing the exit test a condition for high school graduation. Much of Haney's effort was directed towards trying to convince Judge Prado that the 20-point increase in the pass rate on the exit test between 1994 and 1998 was due to substantial increases in numbers of students who dropped out before even taking the exit test. Haney tried to demonstrate that if students who would most likely fail the exam dropped out of school before they were scheduled to take the test in the spring of their tenth grade, then the calculated pass rate from the remaining students would be greater as a consequence. In addition, of course, the alleged increase in dropouts, especially if they occurred disproportionately among minority students, would directly demonstrate the damaging impact upon minority students.

Professor Haney went beyond factual arguments and attempted to impute motive to administrators and teachers. For example, "These results clearly support the hypothesis advanced in my December 1998 report, namely that after 1990 schools in Texas have increasingly been retaining students, disproportionately Black and Hispanic students, in grade nine in order to make their grade 10 TAAS scores look better [emphasis added] (Haney, 1998, pp. 17-18).(Note 7)

After having convinced himself, at least, that students are intentionally retained so that the exit test pass rate would be higher, he concluded: "Hence, it is fair to say that the soaring grade 10 TAAS pass rates are not just an illusion, but something of a fraud from an educational point of view."(Note 8)

While it was important to Haney and his clients to attempt to demonstrate that the very existence of the TAAS exit exam in some way, intentionally or not, caused an increase in grade nine retention and subsequent increase in dropouts, it was perhaps even more important to their case to demonstrate that these factors, in turn, were responsible for the dramatic improvement in TAAS exit test pass rates. For if the pass rate increase could be shown to be primarily a result of the increased retentions and dropouts (and also, perhaps, increases in the use of special education exemptions) then the primary justification for the TAAS-based accountability system itself would be discredited. That is, as long as the state could demonstrate that the academic performance of students who remained in school was improving, then it could be argued that this benefit offset if not outweighed the alleged increase in the number of dropouts. But if the improved performance, as measured by the pass rate on the exit test, was an illusion, as Haney asserted, due to the very increases in dropouts and special education exemptions, then the overall impact of the exit test could be argued to be a burden to the state as a whole as well as to the additional students who dropped out and thereby failed to earn their high school diplomas.

As pointed out by Haney, Judge Prado held that the hypothesis that "schools are retaining students in ninth grade in order to inflate tenth-grade TAAS results was not supported with legally sufficient evidence demonstrating the link between retention and TAAS (Prado, 2000, p. 27)"(Note 9) In fact, it will be shown below, utilizing data contained in the Haney article itself, that the possible impact of increased dropouts and ninth grade retentions actually decreased during the period 1994 to 1998, using Haney's own methodology. Judge Prado was correct.

In setting out to quantify the relationship between increases in ninth grade retentions, student dropouts and TAAS pass rates Haney relied upon the ratio over time of eleventh grade enrollment in a given year to sixth grade enrollment five years earlier. It is important that the reader be familiar with Haney's own justification for the use of this procedure, and to see how he misused it in reaching his conclusion. Following a discussion of the relationship of grade 9 to grade 8 enrollments, we find the following:

At the same time, the analyses of progress for grade 6 cohorts presented in Section 5.3 revealed that grade 6 to grade 11 progression ratios for Whites and minorities varied by not more than 5% during the 1990s (for Whites, the ratio was consistently between 85% and 89%; and for minorities between 75% and 80%). The reason for focusing here on progress to grade 11 is because the data on enrollments is from the fall whereas TAAS is taken in the spring. But if students progress to grade 11, they presumably have taken the exit level version of TAAS in spring of the tenth grade.

What this suggests is that the majority of the apparent 20-point gain in grade 10 TAAS pass rates cannot be attributed to exclusion of the types just reviewed. Specifically, if rates of progress from grades 6 to grade 11 have varied by no more than 5% for cohorts of the classes of the 1990s, this suggests even if we take this as an upper bound, the extent to which increased retention and dropping out before fall grade 11, and add 2% for the increased rate of grade 10 special education classification, we still come up with less than half of the apparent 20-point gain in grade 10 TAAS pass rates between 1993 [sic, 1994] and 1998.(Note 10)

To emphasize, if a student is enrolled in the fall of his or her eleventh grade level, presumably the student would have been in the tenth grade the previous spring of the same calendar year, and would therefore have been in the pool of students who would have taken the tenth grade exit test for the first time.(Note 11) Several issues are glossed over here, such as (a) students repeating grade 11, (b) students beginning public school in Texas as eleventh graders (immigrants or students previously enrolled outside the public school system), (c) students who may have taken the exit test in the spring but who dropped out over the following summer, and (d) changes in the proportion of students exempted from taking the exit TAAS. Regarding this latter, Haney presented data that suggested that the number of tenth grade students in special education had increased by approximately two percent during the 1994 to 1998 period.(Note 12) Accepting the simplifying assumptions in Haney's procedure, the results hinge on whether or not the ratio of eleventh graders to sixth graders (five years earlier) increased or decreased during the 1994 to 1998 period. Haney referred to these ratios as grade 6 to grade 11 progression ratios. For clarity, let the progression ratios for 1994 and 1998 be defined as follows:

Progression Ratio(94) = (Grade 11 enrollment in fall 1994/Grade 6 enrollment in fall 1989)

Progression Ratio(98) = (Grade 11 enrollment in fall 1998/Grade 6 enrollment in fall 1993)

If the data show that Progression Ratio(98) is less than Progression Ratio(94), then Haney has made his case. On the assumption that the 1998 ratio was reduced by an increased rate of dropout behavior occurring before the exit tests were taken in the spring of 1998, or an increase in ninth grade retentions, and assuming that all of the additional students who thereby did not take the exit test would have failed the test, then the pass rate would have increased by approximately the same rate as the increase in the rate of dropouts and ninth grade retentions, relative to the pass rate with no change in these phenomena.

Of course, the opposite might also occur, in which case an adjustment to the observed pass rate in the opposite direction should be made. That is, assume the proportion of dropouts decreased, or smaller proportions of students were retained in the ninth grade. This would cause an increase in Progression Ratio(98) as compared to Progression Ratio(94), Still assuming, with Haney, that all of the marginal students would fail the exit test, then the pass rate for the set of students in 1998 that would be comparable to the equivalent set in 1994 would be greater than the observed pass rate for all students tested in 1998. This is because, under such assumptions, greater proportions of low performing students would be tested in 1998 then in 1994. In short, if Haney's methodology would call for a negative adjustment to the pass rate if the progression ratio decreased from 1994 to 1998, then applying the same methodology to the opposite outcome, (i.e., an increase in the progression ratios) should require an upwards adjustment to the pass rate actually observed 1998.

What really happened? It is difficult to know, if one relies only upon Haney's text. In the first of the two paragraphs, there is a reference to progression ratios during the 1990s without mention of whether they tended upwards or downwards, stating only that "they varied by not more than 5% during the 1990s..." In the second paragraph quoted above, he then suggests that this vague 5% variation which occurred sometime "during the 1990s" can be taken as an upper bound of the impact of grade retentions and increased dropouts upon the exit test pass rate increase.

One cannot tell from Prof. Haney's own statements whether the ratio of grade 11 to grade 6 students increased or decreased during the particular period 1994 to 1998, which is the period during which the exit test pass rates increased by 20-points. In proposing to adjust the increase in the passing rates, what happened before or after this interval is irrelevant. Hence, reference to a range of variation "during the 1990s" is not helpful. Nor is reference to variation in these ratios for Whites and minorities taken separately. The 20-point improvement in test pass rates includes all students taking the exit test, in all subjects. If the grade 11/grade 6 enrollment ratios of Whites increased by 5%, but those for minorities decreased by 5%, they would approximately offset one another. If they both decreased by 5%, then the total effect would also be 5%. They are not additive. It is necessary to know what happened to the grade 11 to grade 6 enrollment ratios for all students in order that any change be relevant in adjusting changes to pass rates for all students. It is also necessary that they be based on grade 11 enrollments in 1994 and in 1998.

Fortunately, Haney included in an appendix the data necessary to clarify these ambiguities.(Note 13) The enrollment data, by grade level, are available there for the three major ethnic groups from 1989 to 1998. The three major ethnic groups included 98 percent of total enrollment in 1989. The major group not included was Asian-Americans, for which dropout behavior and TAAS performance is not a major issue. Their omission does not alter the results presented below in Figure 1 or in Table 1.

Figure 1. Progression Ratios, 1990–1998.

The progression ratios for grade 6 to grade 11 are presented in Figure 1. They are shown separately for African Americans and Hispanics combined, for non-Hispanic Whites, and for the totals of all three major ethnic groups. The ratios were calculated by dividing the appropriate eleventh grade enrollments by corresponding sixth grade enrollments five years earlier. The time scale in Figure 1 indicates the year for which the eleventh grade enrollments correspond. As pointed out by Haney (see above) the calendar year for which eleventh grade enrollments are recorded in the fall is also the year in which the same cohort of students would have been first administered the exit test the preceding spring. Therefore, a significant change in the progression ratio for a given year would be expected to have influenced the percentage of students passing the exit test in the same calendar year.(Note 14)

It is interesting to compare the data plotted in Figure 1 with Prof. Haney's comments. As he stated (above) the "progression ratios for Whites and minorities varied by not more than 5% during the 1990s..."(fn) Indeed, the highest ratio for minorities was 0.79, the lowest was 0.75 for a range of 0.04 or 4 percentage points. The high for Whites was 0.89 and the low was 0.85, for a range also of 4 percentage points. But it is important to note how these ratios changed over the specific interval of 1994 to 1998, which corresponds to the interval over which the 20-point increase in exit test pass rates occurred. It is not sufficient to merely note the "range". What is important is whether the ratios increase or decreased over this particular period. Additionally, it is the change in the ratio for all ethnic groups combined that is relevant to an attempt to explain the 20-point improvement in the test pass rate, as this improvement reflected the improved performance for all groups combined.

So what actually happened to the relevant progression ratios? As can be seen in Figure 1, the progression ratio for minorities increased from 0.76 in 1994 to 0.78 in 1998, albeit with a one percent decrease in 1995. For Whites, the progression ratio also increased, from 0.86 in 1994 to 0.89 in 1998. For all three major ethnic groups combined, the progression ratio increased from 0.81 in 1994 to 0.83 in 1998. Therefore, instead of adjusting the improvement in the pass rate from 20-points down to 15-points, as implied by Haney, it should be adjusted upwards by 2-points to 22-points due to the increase in the grade 6 to grade 11 progression ratio.

Granting a negative adjustment of 2 points due to an increase in special education exemptions among tenth-graders, the effect upon the improvement in the exit test pass rate attributable to special education exemptions, ninth-grade retentions, and dropout rates net to zero—no impact whatsoever.

Total enrollment figures for the three major ethnic groups for the relevant years are shown in Table 1. Once again, the results shown in column 5 of Table 1 are exactly opposite to the assertions by Haney. Instead of a negative adjustment of 5 points to the 20-point improvement in the exit test passing rate over the 1994 to 1998 period, a positive 2 point adjustment should be made, following the same logic.

Data source: Haney (2000), Appendix 7, pages 138-139.

Notice also that grade 6 enrollment for the three major ethnic groups increased by 32,835 students between 1989 and 1993, while eleventh grade enrollment corresponding to the same cohorts, five years later, grew by 33,062 students from 1994 to 1998. The fact that grade 11 enrollments increased more than grade 6 enrollments certainly does not support Haney's claim of increasing dropouts and/or rates of ninth grade retentions during this time period.

Therefore, using Haney's own suggested methodology, and data which he himself provided, none of the improvement in the TAAS exit test pass rate has been shown to be a myth or otherwise fraudulently obtained. Instead of demonstrating that "at least half of the apparent increases are a mirage resulting from increasing numbers of students being excluded from test results", as he had claimed, his data and procedures account for none of the exit test pass rate improvement.

Prof. Haney invited readers of his article to form their own judgments as to the fairness of his analysis. We have formed our own judgment as to whether Professor Haney was fair-minded in his use of data and in the conclusions he drew from those data regarding the quality and effectiveness of education reform in Texas over the last decade. Like Professor Haney, we invite readers to arrive at their own judgments on this matter.

Notes

General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu .