In an analysis of the Massachusetts graduation examination, Bolon (2001) examined the aggregate grade 10 mathematics test scores for 47 high schools and the demographic characteristics of the communities in which they were situated. From several data analyses, Bolon determined that since the best single predictor of mean high school score was community per capita income,

"The state is treating scores and ratings as though they were precise educational measures of high significance. A review of thenth-grade mathematics test scores from academic high schools in metropolitan Boston showed that statistically they are not."

Further, when removing the variability due to per capita income,

"Large uncertainties in residuals of school-averaged scores, after subtracting predictions based on community income, tend to make the scores ineffective for rating performance of schools. Large uncertainties in year-to-year score changes tend to make the score changes ineffective for measureing performance trends."

Predicting aggregate test scores

The difficulty with using a community aggregate measure as a predictor is that it is a surrogate for many other indicators, some of which are absurd at face value but interpretable. Variables such as driver's-license passing rate or per capita champagne consumption may predict student achievement as well as community per capita income. We can construct meaningful arguments why they might. For none is the test invalidated using accepted standards (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999).

In other areas of research such aggregation has produced fundamentally misleading conclusions. For example, the literature on intelligence and income is directly parallel to the discussion here. White (1982) demonstrated the difference between using an aggregate measure of SES (school or community) and individual measure in relating SES to intellectual functioning. Since Bolon used school as his unit of analysis, he eliminated proximate measures more appropriate to his analysis. The school-level variables Bolon eliminated are more appropriate than community per capita income on this basis if in fact they were school-based and not district-based. Measures such as free and reduced lunch (FRL) are better indicators for elementary school than for secondary school analyses, however, because of social undesirability of either participating or reporting among secondary students, who tend to have independent means for buying lunches.

The principle of proximity in selecting variables should be carefully considered and invoked. Mixing levels of analysis produces uninterpretable results, as hierarchical linear modeling advocates have pointed out. Bolon erred in this way, we argue.

Test validity: AERA/APA/NCME Standards

Challenges to content validity have been consistently thrown out by courts, including a recent case here in Texas (Mehrens, 2000, citing GI Forum et al vs. TEA et al., CA No. SA-97-1278-EP, U.S. District Court, Western District of Texas, San Antonio, TX). Mehrens invoked the 1985 Standards to review the Texas statewide assessment in a process we follow more briefly here.

The congruence of test content with intended instruction is a central focus of test development. Nothing about test content appropriateness was evident in the analysis of income prediction of performance by Bolon. A more comprehensive and focused analysis might ask if schools in lower income communities do not adhere to the state guidelines, or if their teachers are unprepared to teach the mathematics required, or suitable textbooks are not available, so that students do not have an opportunity to learn. These representations might make a case for the relevance of income in dismissing the mathematics test as a precise educational measure of high significance.

The per capita income disparities in our state are much greater than those shown by Bolon. Our experience with our own Texas Assessment of Academic Skills (TAAS) at all levels and content areas has convinced us that income inequities, while important, are not the most useful explanatory variable in school performance. With much larger databases available to us, such as multiyear summaries of all schools in Texas by grade, we see much greater variation in school performance than is shown in the 47 schools Bolon selected. While the correlation is much weaker than the .9+ Bolon presented, nevertheless it is substantial and meaningful. When looking at scatterplots of performance, however, we are struck by the existence of very high-poverty community schools that manage to score very high on the TAAS. For example, Fig. 1 shows the scatter for about 3000 schools with 3^rd grade classrooms of TAAS reading and percent economically disadvantaged students (school level measurement), what we would call a surrogate for per capita income. What is of interest is the top of the graph, and the many schools that perform in a manner the state defines as excellent. The correlation data are reported in Table 1 (the approximately 800 schools not reporting economic disadvantage had somewhat higher TAAS scores than those are that did report).

Per capita income is an uninterpretable predictor; its relatedness, or not, to school achievement tells us nothing about the stakes being tested, high or not. It fails the theory criterion of standard 1.1.

Instructional effects

We are unaware of any state that has actually conducted an instructional effect study with pilot versions of its tests to examine the sensitivity of their high stakes tests to instruction. The first author was a member of a committee formed by the legislature of the state of Texas to recommend the structure of the current accountability system (College of Education, Texas A&M University, LBJ School of Public Affairs, The University of Texas at Austing, College of Business, University of Houston, 1993). In the course of committee discussion, the suggestion was raised by the first author that only with some form of pre-post within year assessment at the student level would there be even minimal evidence for instructional change. This suggestion was ultimately rejected by politicians as too costly to consider. Instead, year-to-year student (and school) change was later made into amethodologically suspect statistic, the Texas Learning Index. Bolon has made the same error in considering longitudinal change within test. The alternative explanations for yearly change negate any interpretation about large uncertainties. Student composition, student mobility, curricular emphases, teacher stability, administrative upheavals, and historical internal validity threats all may explain the variation in a school. Unless and until those are explored and discounted, Bolon's analysis does not support any particular validity threat to the test. We agree that schools, before being held accountable, must be examined carefully for the alternatives listed above. Year-to-year comparisons are inherently flawed due to internal validity threats; the connection between instruction and student performance is weak. It is only because of unwillingness to investigate the actual productivity of the school that a year-to-year comparison is made.

Content limitations

The Texas released 10th grade mathematics examination (Texas Education Agency, 2000) has 40 items. From a review of the content, it appears that at best only one or two assess topics not covered in grades 8 or below, while one item (19) appears to be a spatial rotation task more appropriate to an intelligence test. The inadequacy of such a test to evaluate 9 or 10 years' mathematics learning is, if not self-evident, at least empirically testable. One can conceive of various research investigations involving interview with teachers and students and performance demonstrations by students on the full range of TEKS objectives to evaluate how well a short form such as the TAAS estimates actual mathematics declarative and procedural knowledge. In the 1993 discussions in Texas cited above, the introductory letter by Charles Miller (1993) made clear that the committee proposed to eliminate the 10th grade test in favor of specific grade 10 subject matter tests such as Algebra and Biology. While there was an obvious concern for creating a set of hurdles, the committee's recommendation was based on testing students over content more proximate to their instruction.

Year-to-year stability

Table 1
Correlations within and across year for grades 3-5 for 1999-2000

Note that within year, cross-grade correlations are higher than between-year, within grade correlations or between-year, cross-grade correlations, generally. The cohort effect, however (grade 3 in 1999 to 4 in 2000, grade 4 in 1999 to grade 5 in 2000) appears supported since these correlations are higher than the cross-year, within-grade correlations for grades 3-5. That at least appears consistent with what might be expected: between-student correlations are lower than within-student correlations, however attenuated they might be for school averages. If these correlations are to be used as part of a school-level assessment, however, they appear woefully inadequate psychometrically.

A different approach can be taken by treating the within-year cross-grade TAAS scores as scale variables. Then coefficient alpha for 1999 is.8539 and for 2000 is .8637. If one is evaluating schools, these are reasonably good values.

Table 2 presents change correlations. While cautions abound about interpreting change scores, if change is the currency to be used, descriptive and correlational characteristics must be considered. Obviously, the change measures that include the same score, such as D33 with D43, are inflated by the self-covariation. The other correlations, that are not self-inflated, are generally positive but modest, almost all in the .1 to .2 range. The D43 and D54 correlation of .165, for example, supports a conclusion that schools' cohorts improve together (or fall behind together).

Table 2
Correlations for yearly change within and between for grade 3-5 and changes for grades 3 to 4 and 4 to 5 for Texas schools 1999-2000

Note: D33, D44, and D55 are the stability coefficients in Table 1.

Table 3 presents correlations between change measures and available school characteristics such as size of school (ENROLL 2000), percent of school on free-and-reduced lunch, percent of school with Limited English Proficient (LEP) students, and percentage of the ethnic groups that in Texas are the focus of civil rights enforcement (African-American and Hispanic). LEP students were exempted from the TAAS in these years, so high performing schools with high LEP percentages can be based on very small samples of white students. The correlations are of interest insofar as they provide different ways to examine school performance at the individual school level rather than using the block method employed by Bolon.

Conclusions

Table 3
Correlations between TAAS 1999-2000 grade changes and selected school characteristics

Bolon, C. (2000). School-based standard testing. Education Policy Analysis Archives 8(23), May, 2000, available at http://epaa.asu.edu/epaa/v8n23.

Bolon, c. (2001). Significance of test-based ratings for metropolitan Boston schools. Education Policy Analysis Archives 9(42). October 16, 2001, available at http://epaa.asu.edu/epaa/v9n42.

Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPortland, J., Mood, A. M., Weinfeld, F. D., & York, R. L. (1966). Equality of educational opportunity. Washington, DC: U.S. Government Pringing Office, 1966.

College of Education, Texas A&M University, LBJ School of Public Affairs, The University of Texas at Austing, College of Business, University of Houston, (1993). A New Accountability System for Texas Public Schools, Vol. 1. Austin: Educational Economic Policy Center, The University of Texas at Austin.

Mehrens, W. A.(2000). Defending a state graduation test: GI Forum et al. vs. Texas Education Agency. Measurement perspectives from an external evaluator. Applied Measurement in Education, 13(4), pp. 387-401.

Miller, C. (1993). Introductory letter. In College of Education, Texas A&M University, LBJ School of Public Affairs, The University of Texas at Austing, College of Business, University of Houston, A New Accountability System for Texas Public Schools, Vol. 1. Austin: Educational Economic Policy Center, The University of Texas at Austin.

Texas Assessment of Academic Skills Exit Level . Austin: Texas Education Agency, February 2000. Available at http://www.tea.state.tx.us/student.assessment/resources/release/taas/release01/xl.pdf

White, Karl R. (1982). The relation between socioeconomic status and academic achievement. Psychological Bulletin, 91(3), 461-81.

About the Authors

Email: v-willson@tamu.edu

Victor L. Willson is Professor of Educational Psychology and Teaching, Learning and Culture, and Director of the Cognition and Instructional Technologies Laboratory, Texas A&M University. His current research interests focus on the intersections of psychometrics, children's reading development, and individual differences in cognitive measurement.

J. Thomas Kellow
Visiting Assistant Professor
College of Education
University of Houston

Email: tkellow@pdq.net

J. Thomas Kellow is currently Visiting Assistant Professor in the College of Education at University of Houston. He received his Ph.D from Texas A&M in Educational Psychology, and has research interests in high-stakes testing, applied statistics, and disability studies with an emphasis on the community integration of persons with mental retardation.

General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu .