Contents

• Section 1: Background
  • A. State Testing in Massachusetts Public Schools
  • B. Schools in the Boston Metropolitan Area
  • C. Statewide MCAS Test Results
  • D. Test Score Studies
  • E. Sources of Data
• Section 2: Data Analysis
  • A. Trends Study of Variability
  • B. Effects Study Involving Social Factors
  • C. Observations
  • D. Summary Analysis
• Section 3: Results
  • A. Opportunities and Questions
  • B. Conclusions
• Appendix 1: Education Reform in Massachusetts
• Appendix 2: Massachusetts Vocational Schools
• Appendix 3: Metropolitan Boston MCAS Mathematics Scores
• Appendix 4: Metropolitan Boston School Characteristics

Section 1: Background

A. State Testing in Massachusetts Public Schools

The most recent form of state testing in Massachusetts public schools is the Massachusetts Comprehensive Assessment System (MCAS), a set of achievement tests sponsored and produced by the Massachusetts Department of Education and administered in the spring of years beginning in 1998 (see Appendix 1 and Bolon, 2000). These tests have four sections: English language arts, mathematics, science and technology, and history and social science. For the years 1998-2000, tests were administered in grades 4, 8 and 10. Beginning in April, 2001, the former grade 4 test sections were divided between grades 4 and 5; new tests have been added in grades 3, 6 and 7.

MCAS tests are loosely timed and include questions in multiple choice, short answer and extended answer formats; they are provided in English and Spanish. All public school students are required to take MCAS tests; there are no "opt-out" provisions. Students taught in parochial schools, other private schools, home schooling and out-of-state schools are not required to take or pass MCAS. For 1998 through 2000, the Department of Education has published test questions used for scoring approximately six months after test administration (see Mass. DoE, 2000g, for example). It has produced new test forms each year. According to current plans, starting with the class of 2003 minimum scores on the English language arts and mathematics sections will be required to graduate from high school (Mass. DoE, 1999f) and to enroll at state colleges, except for MCAS test preparation courses at two-year colleges.

The Massachusetts Department of Education publishes MCAS results as scaled scores in a range of 81 scale points, using the integers 200 through 280 (Mass. DoE, 2000h). The Department assigns labels it calls "performance levels" to four scaled score intervals (Mass. DoE, 1998b) and currently considers 220 the minimum passing scaled score on all test sections (Mass. DoE, 1999f). The Department has not fully disclosed details of assigning scale factors, assuring consistent scores across test forms or assuring that scores quantitatively reflect published academic standards. It has not published distributions of either raw scores or scaled scores. It has released limited information about test design and properties in "technical reports" for the years 1998 (Mass. DoE, 1999c) and 1999 (Mass. DoE, 2000i). After an independent analysis of score averages by "racial" and "ethnic" categories for 1998 (Uriarte and Chavez, 1999), the Department published its own analysis of this type for 1999 (Mass. DoE, 2000c).

Massachusetts tests appear to rank near the high end of state achievement tests in difficulty, although failure rates are lower than those for some tests used in Arizona and Virginia. As in several other states, substantially higher failure rates are found on mathematics than on language tests in high schools. Since the tenth-grade version of the mathematics test section sets the graduation threshold for most students, its scores have been used as subjects for these studies.

B. Schools in the Boston Metropolitan Area

Metropolitan Boston is diverse. Besides the City of Boston it includes many smaller municipalities, all operating their own school systems. These studies consider communities inside Route 128, a highway designed in the late 1940s (now an Interstate), enclosing areas within about 9-12 miles from Boston's government center--that is, Boston and its inner and middle suburbs. They share a public transit system, several public and private utilities, and an economy dominated by service industries. They include poverty areas, concentrations of wealth, middle-income communities, prosperous suburban towns, a few medium-sized cities and one large city. The areas are bounded by the Massachusetts Bay and Atlantic Ocean to the east, Salem and Peabody to the north, Waltham and Newton to the west, and Braintree and Quincy to the south (see Metropolitan Area, 1997).

Schools in the Boston metropolitan area are also diverse. These studies, focusing on testing for graduation, consider only high schools. While a majority of the area's population of high-school age attends public schools, (Note 2) a substantial proportion attends parochial schools that began to be established by the Roman Catholic Church more than 150 years ago. A smaller fraction is taught in other private schools or though home schooling. The Metropolitan Council for Educational Opportunities (METCO), founded in 1963, uses state funding to help send over 3,000 Boston minority students to suburban schools (Orfield, et al., 1997).

Within public school systems there is also substantial diversity. All communities must support regional "vocational," "technical" and "agricultural" high schools. Some such schools began as "manual training" schools in the 1800s. Some communities have closed their local vocational schools; some have merged them with their academic schools. These studies look in detail only at academic schools, because the curriculum of vocational schools is substantially different and is not designed to prepare students for MCAS tests, an issue of controversy (Nicodemus, 2000). For purposes of these studies there are difficulties with a few communities, including Cambridge, Quincy, Revere and Waltham, which provide vocational education in the same facilities as academic programs (Mass. DoE, 2000f). I chose to include such schools in these studies while noting their special characteristics.

Several communities also operate experimental schools, including "pilot schools" in Boston and "charter schools" in several communities (Partee, 1997, and Wood, 1999), as regulated under the Massachusetts Education Reform Act of 1993. All that offer ninth-grade curriculum and above are smaller than the regular academic schools. These schools provide motivational environments and may exercise indirect forms of student selection that differentiate them from other public schools. Primarily because of concerns about small sample sizes, schools with fewer than 100 students per grade are excluded from these studies. So far no experimental school is that large.

The City of Boston presents a unique situation. Of its large academic high schools, three are exam schools: the Boston Latin School (founded in 1635) and the more recent Latin Academy (formerly Girls Latin) and O'Bryant School of Mathematics and Science (formerly Boston Technical). These draw away many Boston students who tend to score well on achievement tests, promoting a longstanding social stratification in Boston schools. Over half the students at Boston Latin come to it from parochial and other private schools (Daley, 1997); some say those students would not otherwise attend Boston schools. However, other public school students who are not admitted leave the district for high school. Starting in 1975, because of federal court orders to desegregate, exam school admission policies included a 35 percent set-aside for African American and Latino students, maintained voluntarily after 1987. As a result of another federal court decision (McLaughlin, 1996), this approach was weakened in 1997. As with academic schools that provide vocational education, the Boston exam schools are included in these studies, but their special characteristics are noted.

C. Statewide MCAS Test Results

Table 1-1 shows that statewide, tenth-grade MCAS test scores have remained nearly constant in English language arts and in science and technology for the years 1998-2000, while scores in mathematics have risen substantially (Mass. DoE, 2000h). (Tenth-grade tests were not given in history and social science.)

Table 1-1 reflects Massachusetts Department of Education practice of recording students absent for a test section as scoring 200 and in Level 1, the lowest level (Mass. DoE, 2000h, Table 11 footnote). An undisclosed fraction of students were excluded from testing because of special conditions and are not counted in this report; others may have been provided with an alternative assessment. As currently planned, students in Level 1 will be ineligible to graduate from high school as of 2003. Based on this record of scores, about half of all Massachusetts public school students are at risk of being denied graduation.

The labels of the four "performance levels" designated by the 1998 Board of Education(Note 3) for reporting MCAS results are:

• Level 4, "Advanced"
• Level 3, "Proficient"
• Level 2, "Needs Improvement"
• Level 1, "Failing"

Although these levels have qualitative descriptions (Mass. DoE, 1998b), there are no quantitative links to levels of achievement specified in academic standards; content of standards has not been prioritized; nor have standards been promulgated through state regulations, as anticipated by law. (Note 4) Although Massachusetts law requires "competency determination" in mathematics, science and technology, history and social science, foreign languages and English, (Note 5) Massachusetts laws and regulations continue to require only US history and physical education as subjects of instruction. Massachusetts tries to set legal standards for learning indirectly (Note 6) through MCAS tests, procedures to set scale factors, and regulations for minimum scaled scores. It lacks corresponding legal commitments for instruction. It has made major changes to "curriculum frameworks" every few years (Mass. DoE, 2000k) and has not provided reasonable spans of time for instruction to catch up before using revised "curriculum frameworks" as a basis for revised MCAS tests. Its teachers, parents and students cannot find out exactly what must be learned in order to meet minimum standards for high school graduation. The 1993 Education Reform Act left several such problems; few have been addressed yet by the Massachusetts legislature or Board of Education.

Students with disabilities (also called special education students) and students with limited English proficiency (LEP students) tend to receive drastically lower MCAS scores than other students, although some students with disabilities are soon to be provided alternate assessments (Mass. DoE, 2000l), and some LEP students have been able to take tests in Spanish (Mass. DoE, 2000d). The Department of Education has not disclosed the fractions of students who are eligible for or have utilized its special accommodations, although it has published statewide summary data using these student categories (Mass. DoE, 2000i, Table 14.5). Most minority students also receive lower scores than other students. The Department of Education has published 1999 statewide and district summary data for students categorized as "African American / Black," "Asian or Pacific Islander," "Hispanic / Latino," "Native American," "White" and "Mixed" (Mass. DoE, 2000c, Tables 5-10). As previously noted, most students in vocational programs receive lower MCAS scores than students in academic programs; this can readily be shown for the state's more than 30 vocational, technical and agricultural high schools (Appendix 2).

Based on the sources of information cited, Table 1-2 shows statewide impacts of these known risk factors on average 1999 tenth-grade mathematics scores and rates of failure.

The Department of Education has not reported scores classified by other potential risk factors on which it collects information. These include: Gender of students, Tests taken in Spanish or as alternate assessments, Free or reduced price lunches, as indicators of poverty, Schools with large class sizes, especially in early grades, Students retained below grade or placed below grade level, Teachers who lack certification in their subjects of instruction.

There is also little published information about combinations of risk factors. However, since the Department of Education lists regional vocational, technical and agricultural schools as separate districts in its reports of MCAS results, it is possible to use their categories of minority students (Appendix 2). For those schools for which categories are reported, results are shown in Table 1-3.

While the results in Table 1-3 are not strictly comparable with Table 1-2, because not all the schools and categories can be found in published data, they indicate that factors can combine to worsen the scores of students with more than one risk factor.

D. Test Score Studies

Recent studies question assumptions that "high-stakes" tests like MCAS can provide valid measures of either student achievement or school performance, showing gains on them that are not matched by gains on other tests for closely related educational content (Haney, 2000, and Klein, et al., 2000). Political environments of "high-stakes" tests create heavy pressure to improve scores, regardless of underlying educational progress. For "low-stakes" tests aimed at measuring long-term trends, like those of the federal NAEP, it has been shown that "family variables explain most of the variance across scores in states" (Grissmer, et al., 2000, Chapter 9). Individual and longitudinal studies demonstrate strong influences of parenting practices, family structure, parent education and degrees of poverty on cognitive development (for example, Smith, et al., 1997). Other longitudinal and cross-sectional studies show cumulative responses of test scores to educational environments (for example, Phillips, et al., 1998, and Ferguson, 1998). However, the data generally available for test score research fail to capture much of the critical information needed to understand development of cognitive abilities and educational achievement in the settings of public schools.

MCAS test scores have already been the subject of several attempts to explain, predict or interpret them (Mass. DoE, 2001, Gaudet, 2001, Tuerck, 2001a, and Tuerck, 2001b). These prior MCAS test score studies fall into three main categories: 1) Trends studies of year-to-year and multi-year changes; 2) Effects studies involving social factors for the population; 3) Effects studies involving operating factors for the schools.

Research on scores from school-based standard tests suggests that many such studies are likely to yield results of low significance. Grissmer, et al., 2000, among others, show that:

• Real year-to-year changes in average student performance, as assessed by conventional tests, are relatively small; they can easily be masked by statistical uncertainties.
• Social factors are strongly associated with test scores.
• Self-reported social information tends to have high error and omission rates.
• Census and other community-based social information often includes confounding factors that require adjustment to reflect the households for a school population.
• Uncategorized school spending is only weakly associated with test scores.

The MCAS test score studies cited use scores and statistical data to estimate the performance of schools or districts according to simple formulas, unsupported by other evidence. They frequently present results in a table that is ranked or can be ranked like the teams in a sports league. The "league table" approach to presenting such results begs the question of whether the ordering of schools or districts and the differences in performance estimates have educational significance, that is, whether such rankings may instead be largely matters of chance or be associations with factors other than school performance. This article presents a trends study and an effects study I conducted to explore the significance that can be associated with such results.

E. Sources of Data

The school characteristics used in these studies are taken from information reported by public schools to the Massachusetts Department of Education for 1999 and published by the Department (Mass. DoE, 2000f). MCAS test scores summarized by schools are from 1998-2000 Department reports (Mass. DoE, 2000h). Other information is published by the Department for school districts, including program budgets and percentages of special education students. Information for census tracts and communities is available from the US Bureau of the Census and other sources. Data analysis for these studies focuses on information associated with individual schools because aggregate information for school districts or general populations can mask school characteristics. Data used in these studies are reproduced in Appendix 3 and Appendix 4; interested readers can confirm them at the sources and can repeat these studies or perform other analysis with them.

The Department of Education and the school districts collect other potentially useful information that is not currently published. Of particular interest are data on class size and teacher preparation. Recent research has shown significant association of educational achievement as measured by "low-stakes" tests with small class size in elementary schools (Nye, et al., 1999, and Krueger, 1999) and with teacher certification and education (Darling-Hammond, 2000), after adjustments for student backgrounds. Studies of the development of cognitive abilities cast doubt on whether other information currently published by government sources about population and economic characteristics in large geographical areas would substantially improve the understanding of test scores.

This study considers 47 academic high schools in 32 metropolitan Boston communities through the average tenth-grade MCAS mathematics test scores recorded for years 1998-2000. Achievement tests in mathematics typically require substantial skill at language interpretation (see, for example, Gipps and Murphy, 1994, Chapter 6, p. 183). Haney, 2000, in a study of another state, found stronger correlations of state mathematics test scores with grades in English than with grades in math. As previously noted, the tenth-grade mathematics test is used in this study of significance because it sets a graduation threshold for most students.

Test boycotts have been organized by students in several schools each year (Steinberg, 2000), involving 10 to 31 percent of students in 19 cases out of the 141 test samples. To be able to compare average scores of schools more accurately, the average scores reported by the Department of Education have been adjusted by removing the scores of 200 that were assigned to students who did not take the test, averaging only scores of students who participated.

Table 2-1 shows changes in schools' average scores (Appendix 3) between 1998 and 1999 and between 1999 and 2000, expressed in units of scale points and of standard deviations.

A "delta" is a change in scale points between two years, minus the average change for the year span, divided by a standard error of the change that is estimated from test reliability and number of participants. The uncertainty for one test score is based on an average standard error of 6.7 scale points, from reliability estimated by the Department of Education for the tenth-grade mathematics test of 1998, using randomized split-half comparisons (Mass. DoE, 1999c). The variance of an average score for a school is estimated by the square of this quantity, plus variance contributed by roundings, divided by the number of test participants.

A delta expresses the change for a particular school beyond the average change, in units of estimated standard errors. When standard errors are accurately estimated, deltas outside +/- 2 are statistically significant at the p< .05 level, but here about half of the schools in both year spans have deltas well outside this range. Especially large deltas were those shown in Table 2-2.

Statistically large deltas occur about ten times as frequently as they would by chance, were test reliability the only factor in standard errors. Since deltas adjust for changes in test difficulty and in average efforts at teaching, learning and test preparation, there appear to be other factors affecting test score changes that differ from school to school and from year to year. However, the Department of Education has not published any studies on other variability factors.

Table 2-3 reflects data for all 47 schools included in this study, showing the average score changes in scale points for the 1998-2000 years, weighted by numbers of test participants.

The average score change for all schools would be highly significant for both year spans at the p<.0001 level and better if test reliability were the only factor in variability. Scatterplots of score changes in Figure 2-1 and Figure 2-2, showing deltas for a span versus average scores in 1998, do not indicate strong relationships but show possible outliers; they are schools noted in Table 2-2. These plots of year-to-year changes provide a picture of the variability in school-averaged test scores, which is much greater than estimates based on test reliability and number of test participants. Evaluating year-to-year point score changes, excluding three cases that have deltas beyond +/- 10 as outliers, yields a practical measurement of variability for school-averaged MCAS test scores.

Figure 2-1: Changes in MCAS Grade 10 Math Test Scores, 1998-1999

.

.

Figure 2-2: Changes in MCAS Grade 10 Math Test Scores, 1999-2000

Standard deviations of school-averaged score change, less average score change for a year span for all schools, are 2.9 scale points for 1998-1999, 3.1 scale points for 1999-2000 and 3.0 scale points for both spans combined. These are about five times larger than the uncertainty estimated on the basis of test reliability. As one might expect, scatterplots of score changes in scale points versus school sizes show greater variability for smaller schools. An analysis found that an equivalent standard error in scale points for school averages of grade 10 MCAS mathematics test scores can be estimated as the constant 33. divided by the square root of the number of test participants. This estimate of standard error combines contributions from test reliability with random variations in student mental alertness, student backgrounds and school performance. For typical metropolitan Boston schools from these studies, school-averaged test score gains, losses or differences of less than about five scale points are not statistically significant at the p<.05 level.

Average mathematics score increases from 1998 to 2000 for all schools in these studies combined substantially exceed the rates of change that other studies have found to reflect genuine and sustainable improvements in learning. Based on test reliability as a measure of variance, the change is more than 20 standard deviations; based on calculated variance, it is more than four standard deviations. As noted, other tenth-grade MCAS tests showed little score change. Statistical magnitudes of mathematics test score changes strongly suggest causes other than or in addition to improvements in learning. Anecdotal accounts report heavy efforts at test preparation in some schools, but the general upsweep in scores indicates that the 2000 mathematics test may also have been significantly easier than corresponding 1998 and 1999 tests. Sparse published information from the Department of Education about test calibration makes this issue difficult to trace.

B. Effects Study Involving Social Factors

This study, like the trends study in Section 2A, considers 47 academic high schools in 32 metropolitan Boston communities, through average tenth-grade MCAS mathematics test scores as recorded for 1999. School-specific social factors considered in this study (Appendix 4), as of 1999, are listed in Table 2-4.

The factors in Table 2-4 were used as independent variables in linear models for a dependent variable of 1999 school-averaged tenth-grade MCAS mathematics test scores. (Note 7) Residuals from the models are considered as estimators of school performance. Variances and error estimates are calculated by conventional multivariate methods (Bevington and Robinson, 1992).

For 1999, MCAS tests were, in the parlance of US public schools, a "medium-stakes" enterprise, associated with some indirect social pressures but no hazardous consequences for students. No current high school students stood to be denied graduation because of test scores, although summaries of test scores by districts and schools were being published. This was the second year of regular testing. It had been preceded by a year of trial testing, following three years of curriculum specification and test development.

Note that grade size reduction, calculated as a percent decrease in grades 11 and 12 school population as compared with grades 9 and 10, is not identical with dropout rate. While dropout statistics are available, as in other states they are compromised by lack of consistent longitudinal data for educational outcomes. Grade size reduction simply indicates that, for whatever reasons, later year high school grades are smaller than earlier year grades. When substantial, it suggests that many students do not graduate in a normal time sequence. (Note 8) Values in about +/- 5 to +/- 8 percent ranges will be typical of fluctuations for schools of these sizes with stable area boundaries and populations and very low transiency, retention and dropout rates, according to Poisson statistics.

The Department of Education publishes only district information about budgets and special education. Without an accurate means to apportion such measurements to high schools (and in four communities to specific high schools), they have been excluded from consideration. There is considerable variation. District percentages of special education students ranged from 11.1 to 25.5 percent of all students and grades in 1999. Annual district spending reported for all regular education programs ranged from $3,986 to $9,251 per student in 1999. Even within this small group of communities, the Education Reform Act of 1993 failed to equalize school spending.

Factor distributions and correlations for this model are shown in Figure 2-3, a matrix of histograms and scatterplots with unweighted, unconstrained lines of best fit.

Figure 2-3: Factor Distributions for MCAS Grade 10 Math Test Scores

Although there are strong associations in Figure 2-3, such as "Percent Hispanic / Latino" with "Percent limited English proficiency," there are no factors so highly correlated as to be entirely redundant. The numerical correlation matrix is presented in Table 2-5, corresponding to the matrix of plots in Figure 2-3, with all values beyond about +/- 0.5 significant at the p<.05 level.

Some of the correlations in Table 2-5 are strong enough that multiple regression coefficients are likely to be unstable. Therefore a model was developed in stages, examining factors for significance.

The full model from the factors in Table 2-4 was first evaluated with weights proportional to numbers of test participants. It yielded two strong factors with low correlation (C and E): "Percent Asian or Pacific Islander" at p<.02, with a positive coefficient, and "Percent limited English proficiency" at p<.002, with a negative coefficient: Factors for school population and percent grade reduction had particularly small coefficients and low significance. They were removed, and a model with the remaining five factors then associated 67 percent of the variance and produced the factor weights shown in Table 2-6.

With model factors in Table 2-6, high factor weight and significance found in other studies for percentages of African American or Latino students disappear. Both factors have small coefficients and low significance. Statistical weight that might have been attached to these factors instead follows cultural and economic factors: "Percent limited English proficiency" and "Percent free or reduced price lunch." As an experiment, the model was rerun with the latter factors removed; only 57 percent of the variance was associated, and factor weights became those shown in Table 2-7.

In Table 2-7, two "racial" or "ethnic" factors (marked *) have become significant at a p<.05 level. The coefficient for "Percent African American" has turned from positive to negative, and the coefficient for "Percent Hispanic / Latino" has become strongly negative. It seems likely that these two factors are acting as proxies for cultural and economic factors with more predictive power.

Residuals from the five-factor model of Table 2-6 are shown in Table 2-8. This include standard error estimates based on results from the trends study of Section 2A.

At first glance, some residuals in Table 2-8 look substantial, several scale points of difference from the average scores predicted by the model. However, residual ratios for most schools are within +/- 2 standard errors, not significant at a p<.05 level. Someone familiar with metropolitan Boston will recognize that schools with high and low residual ratios tend to be in high-income and low-income communities, respectively. It therefore seems likely that adding a factor for incomes can increase the predictive power of the model.

The most recent community income data were from the US Census of 1990, for 1989 per-capita income. Comparable 1999 income statistics were not yet available. The Massachusetts Department of Revenue could produce current community income statistics but has not done so; the state continues to use 1989 federal census data on incomes to apportion aid to public schools. After adding 1989 per-capita community income in $1,000s as a factor (Mass. DoR, 1999), without any attempt to adjust incomes so as to reflect school districts or student households, the model associates 80 percent of the statistical variance, and factor weights became those shown in Table 2-9.

Three factors in Table 2-9 (marked *) have substantial significance, at a p<.005 level or better, and three have very low significance. Factor weight has shifted from "Percent free or reduced price lunch" to "Per-capita community income (1989)," while "Percent limited English proficiency" retains a large coefficient and high significance. Dropping low-significance factors, the resulting three-factor model is shown in Table 2-10.

The three-factor model of Table 2-10 also associates 80 percent of the statistical variance. All of its factors are statistically significant at a p<.001 level.

For each school included in these studies, Table 2-11 presents adjusted average 1999 tenth-grade MCAS mathematics test scores and residuals from the three-factor statistical model of Table 2-10, with the uncertainties in average scores and residuals expressed as standard errors, based on the variance estimate calculated in the trends study of Section 2A.

Stepwise analysis shows that combinations of the factors in the three-factor model of Table 2-10 associate statistical variance in the amounts listed in Table 2-12.

Table 2-12 shows that the major factors are "Percent limited English proficiency" and "Per-capita community income (1989)." Although statistically significant, "Percent Asian or Pacific Islander" is a weak cofactor, associating only 1 percent of the variance by itself.

The three-factor model of Table 2-10 was evaluated for predictions of 1998 and 2000 average test scores. It was not expected to perform as well, since most factor data were for 1999. However, the factor weights and significance proved robust, and the model associated statistical variance as shown in Table 2-13.

The residuals in Table 2-11 suggest systematic contributions to test score averages at some schools that might have been produced by unusual efforts. Probable outliers for the predictive model with correspondingly large year-to-year average score changes had a strongly positive bias: those for Boston Latin, Latin Academy and Swampscott.

Model behavior for metropolitan Boston schools may be biased by special characteristics of City of Boston schools, because the students who score well on school-based standard tests are selected for admission to the three exam schools. (Note 9) Current data also attribute average Boston per-capita income equally to all school districts instead of adjusting by districts or census tracts. The Boston cross-enrollment and busing programs would complicate an income analysis. Behavior of the three-factor model of Table 2-10 was examined for 1999 tenth-grade MCAS mathematics test scores, considering only schools outside the City of Boston. These 34 schools had a 1999 total population per grade of about 10,200 students out of 13,730 for all schools considered in these studies.

When applied only to schools outside the City of Boston, the three-factor model of Table 2-10 showed significance at the p<.05 level for "Percent limited English proficiency" and "Per-capita community income (1989)" but not for "Percent Asian or Pacific Islander." A two-factor model based on the first two of these factors, with schools weighted by numbers of test participants, associates 86 percent of the statistical variance. Factor weights became as shown in Table 2-14.

Several trial factors were added individually to the two-factor model of Table 2-14, but none showed statistical significance at the p<.05 level. The statistical variance associated when each trial factor was added to this two-factor model is shown in Table 2-15.

The last three trial factors in Table 2-15 are districtaverages; the last and third from last are for all schools in all grades. Three districts, Lynn, Newton and Quincy, each operate two academic high schools which will not be distinguished by these factors. Outside the City of Boston, only "community income" and "limited English proficiency" are significant contributors to 1999 tenth-grade MCAS mathematics test scores; their indicators are effective predictors.

A one-factor model, using only "Per-capita community income (1989)," performed almost as well as any combination of factors shown in Table 2-15, associating 84 percent of the variance. The factor weight is in Table 2-16.

"Percent limited English proficiency" is much less effective in a one-factor model, associating only 38 percent of the variance. Community income appears to be the dominant factor associated with these test scores. The 1999 adjusted average tenth-grade MCAS mathematics test scores by school, plus residuals from the two-factor and one-factor models for 1999, shown in Table 2-14 and Table 2-16, with standard error estimates for each, are in Table 2-17.

In an attempt to improve accuracy of the model in Table 2-14, schools with residuals from the two-factor model for 1999 that were greater than two standard deviations were dropped, Belmont with positive residual and Cambridge Rindge & Latin, Marblehead High and Quincy High with negative residuals. The two-factor model for 1999 scores then produced the factor weights shown in Table 2-18.

Chi square for the two-factor model of Table 2-18 was 36.2 with 27 degrees of freedom (p=.11). The one-factor model of Table 2-16 was also evaluated for 1999 with the set of cases used in Table 2-18, producing the factor weights shown in Table 2-19.

Chi square for the one-factor model of Table 2-19 was 43.4 with 28 degrees of freedom (p=.03). However, the cases and models from Table 2-18 and Table 2-19 did not provide significant chi square probabilities for 1998 or 2000 scores; all other attempts to improve estimation by removing outliers also proved unstable.

Residuals from the two-factor model of Table 2-14 for schools outside the City of Boston are strongly autocorrelated. The coefficient was .45 between 1998 and 1999 and .67 between 1999 and 2000. Scatterplots for successive years are shown in Figure 2-4 and Figure 2-5, comparing 1998 and 2000 residuals with 1999 residuals:

Figure 2-4: MCAS Grade 10 Math Test Score Residuals, 1998 versus 1999

.

.

Figure 2-5: MCAS Grade 10 Math Test Score Residuals, 2000 versus 1999

To the extent that a trend can be observed by applying the model of Table 2-14 to different years, it appears that schools scoring higher than predicted tend to increase scores in successive years, and schools scoring lower than predicted tend to decrease scores. Departures from model predictions are not all random. The two spans of years available for analysis suggest a systematic trend that could stratify high-scoring and low-scoring schools.

C. Observations

The trends and effects studies presented in Section 2A and Section 2B show how studies of these types tend to yield results with low statistical significance unless "something unusual" is going on. In the cases of the 2000 tenth-grade mathematics test and of the high scores and large increases at Boston Latin, Latin Academy and Swampscott, the studies do indicate "something unusual," although they cannot tell what it is. They also illustrate that commonly published "league tables" of scores strongly reflect social factors associated with school populations, not factors clearly associated with school performance. The effects study shows a robust, positive correlation of scores with household incomes, plus a smaller, negative correlation with limited English proficiency. Factor weights and statistical significance for other factors considered are small. School-based data are not currently published for several potentially significant factors, such as class sizes in elementary grades, mathematics course enrollments and levels of teacher preparation.

By far the strongest factor in predicting tenth-grade MCAS mathematics test scores is "Per-capita community income (1989)." For the schools outside the City of Boston this factor alone performed nearly as well as all available factors combined, associating 84 percent of the variance compared with 88 percent when all available factors were used. The scatterplot in Figure 2-6 shows 1999 average scores for these schools versus 1989 per-capita community income:

Figure 2-6: 1999 MCAS Grade 10 Math Test Scores versus Community Income

In Figure 2-6, the relation between school-averaged test scores and per-capita community income looks linear over an income range of about 2½ to 1 for this set of schools. There is no obvious threshold or saturation behavior. As it happens, spending on regular education programs in school districts also varies over a range of about 2½ to 1 for those schools (Mass. DoE, 2000f). However, there is only a weak relationship between 1999 spending on regular education programs in those school districts and their 1999 tenth-grade MCAS mathematics test scores, associating only about 3 percent of the variance and not statistically significant at a p<.05 level, as shown in the scatterplot of Figure 2-7.

Figure 2-7: 1999 MCAS Grade 10 Math Test Scores versus School Spending

Although a tendency for scores on school-based standard tests to rise with incomes has been recognized in the US for more than 70 years (Bolon, 2000), there has been relatively little research on this phenomenon in ordinary income ranges, compared with the attention given to associations between test scores and conditions of poverty or race. The strong, apparently linear association of average test scores with community incomes shown here, as contrasted with their weak association with school spending, calls for investigation but is beyond the scope of this report. It seems likely that community incomes are providing something beyond what school programs provide, but if so we cannot tell from these data what it might be. Perhaps this effect should not be surprising, since most students spend more than three-fourths their waking hours outside school.

The factor "Percent limited English proficiency" was the second strongest influence on predicted test scores. (Note 10) Previous effects studies of MCAS scores might not have shown this if they failed to utilize the entire variety of school-associated data available, including "limited English proficiency," "racial" or "ethnic," and "free or reduced price lunch" student categories. A hypothesis inviting study is that students classified with "limited English proficiency" might receive mathematics instruction less relevant to curriculum tested by MCAS than other students. Another hypothesis is that strengthening English language instruction for "limited English proficient" students might improve their MCAS mathematics test scores, provided all current efforts to teach mathematics and other subjects are maintained. While statistical associations suggest these conjectures, only observations and experiments could prove or disprove them.

Factors of "Percent African American" and "Percent Hispanic / Latino" did not make significant contributions to school-averaged tenth-grade MCAS mathematics test scores after other factors were introduced. Although "Percent Asian or Pacific Islander" retains statistical significance in certain models for some years, it is a much weaker factor than "Per-capita community income (1989)" or "limited English proficiency." The latter factors, and not "racial" or "ethnic" percentages, provide by far the strongest statistical associations with school-averaged tenth-grade MCAS mathematics test scores in metropolitan Boston. (Note 11)

Statistical significance of test-based ratings has been the principal focus of the studies. Some studies developing or using such ratings do not provide an analysis of variability, without which significance cannot be determined; or they estimate variability from single test session reliability measurements, which were shown to be optimistic (for example, Gaudet, 2001, and Tuerck, 2001a). The trends study in Section 2A of this report provides evidence derived from tenth-grade MCAS mathematics test scores for larger variability estimates.

D. Summary Analysis

My summary analysis is based on a one-factor model for metropolitan Boston communities that each operate only a single academic high school, weighted by numbers of students tested. The effects study showed that "Per-capita community income (1989)" was the dominant factor in predicting 1999 school-averaged tenth-grade MCAS mathematics test scores. All other factors made only small contributions to predictions with much lower significance. From published data, community income could not be estimated reliably for each of the multiple academic high schools in Boston, Lynn, Newton and Quincy. Reduction in scope leaves 28 high schools in the same number of communities, with a total of about 8,200 students per grade recorded for 1999. Estimates of uncertainties in school-averaged test scores are based on findings of the trends study, which showed year-to-year variability of school-averaged scores several times greater than the variability implied by conventional test reliability measurements. Plots of results include uncertainty intervals (sometimes called "error bars") equivalent to +/- 1.4 estimated standard errors. When item intervals do not overlap, when standard errors have been accurately estimated, and when items are uncorrelated, then differences between items are significant at the p<.05 level. The one-factor model for 1999 with this set of cases associated 82 percent of the variance and produced the factor weights shown in Table 2-20.

The 1999 adjusted average tenth-grade MCAS mathematics test scores by school, plus residuals from the foregoing one-factor model for 1999, with standard error estimates for each, are shown in Table 2-21.

The plot in Figure 2-8 shows school-averaged adjusted scores on the 1999 tenth-grade MCAS mathematics test and the corresponding uncertainty intervals (or "error bars"). In this and the next two plots, the 28 schools considered in this summary analysis have been rank-ordered from the lowest to the highest average scores on the 1999 tenth-grade MCAS mathematics test.

Figure 2-8: Average 1999 MCAS Grade 10 Math Test Scores by School

The plot in Figure 2-9 shows residuals for school-averaged 1999 tenth-grade MCAS mathematics test scores, the differences left after subtracting away predictions calculated from the factor "Per-capita community income (1989)."

Figure 2-9: Residuals of 1999 MCAS Grade 10 Math Test Scores by School

The picture in the plot of average scores of Figure 2-8, with significant separations between high-scoring and low-scoring schools, is shown by the residuals plot of Figure 2-9 to be associated largely with differences in community income. Chi square for the residuals distribution is 63.2 with 27 degrees of freedom. After subtracting predictions based on community income, residuals of average scores settle to just a little more than statistical noise. Only five or six schools can be reliably distinguished other than by community income.

The last plot, Figure 2-10, shows the changes in school-averaged 1999 tenth-grade MCAS mathematics test scores from the same schools' average scores in 1998.

Figure 2-10: Changes in MCAS Grade 10 Math Test Scores by School, 1998-1999

There was low statistical significance in year-to-year changes of the school-averaged adjusted scores on tenth-grade MCAS mathematics tests between 1998 and 1999. The distribution of changes about an average change of +1.2 scale points, as shown in Figure 2-10, fits a normal distribution with weighted chi square of 22.7 for 26 degrees of freedom (p = .65). Despite the possibility of little significance in score changes, such changes are criteria by which the Massachusetts Department of Education has begun to rate school Performance (Note 12) as "Failed to Meet Expectations," "Approached Expectations," "Met Expectations" or "Exceeded Expectations." There are severe penalties, including state closure or seizure, for schools with low scores and ratings of "Failed to Meet Expectations."

The statistical significance of differences in school-averaged scores, aside from their reflection of community income, may not be not enough to compare schools reliably and may not be enough to evaluate short-term changes in teaching and learning. Whatever aspects of school performance these scores might measure can be lost in the fluctuations for a particular school over any one year or few years. Only averages and trends in scores over several years would be likely to yield statistically useful information.

Some observers question whether it is realistic to expect standardized tests to yield significant information comparing school performance, even over a period of years (for example, Rowe, 1999a, and Rowe, et al., 1999b). They argue that variations in student achievement are commonly greater within schools than between schools. Others contend that adaptive and defensive behavior encouraged by "high-stakes" political environments grossly distorts outcomes from all types of educational assessment and robs them of meaning (for example, Sacks, 1999, and Hayman, 1998). It is important to point out that significance attributed to results in these studies is purely statistical. Neither these studies nor any others known to the author have shown that MCAS test scores have practical significance, in the sense of predicting success in adult activities to any greater degree than could be done with knowledge of student backgrounds.

A. Opportunities and Questions

In future work, it may be helpful to examine categories of metropolitan Boston schools that were excluded from or specially identified in these studies:

• Vocational, technical and agricultural schools
• Academic schools that also provide vocational education
• Pilot schools, charter schools, specialty schools and small schools

Other questions might be answered by extending data coverage:

• Do other test scores yield similar results?
• Do small class sizes have significant effects?
• Does teacher preparation have significant effects?
• Do math course enrollments have significant effects? (Note 13)
• Can other factors be found that increase significance?
• Are the same effects found in other Massachusetts schools?
• Are similar effects observed in elementary and middle school grades?
• Are the same or different patterns observed in other US metropolitan areas?
• Can income factors be estimated for communities with multiple high schools?
• If individual data were available, would multi-level analysis show different results?

The strong, linear relation found between school-averaged tenth-grade MCAS mathematics test scores in a relatively calm year and community income in a previous year, within ordinary ranges of incomes, leads to several questions. Does community income primarily determine educational achievement, regardless of school performance? Do MCAS and similar tests measure skills and knowledge that are acquired in schools, or do they measure skills and knowledge that are largely acquired outside schools? Do communities with substantially different incomes have substantially different expectations for student performance on MCAS and similar tests? Is current community income also strongly correlated with test scores? After a turbulent period of local efforts to raise scores, will the correlation between income and scores remain as strong? Exploring and understanding some of these issues will require a different approach.

B. Conclusions

Community income has been found strongly correlated with tenth-grade MCAS mathematics test scores and associated more than 80 percent of the variance in school-averaged 1999 scores for a sample of Boston-area communities. The influence of community income was robust against several sets of model variables and cases.

Community income swamped the influence of the other social and school factors examined. Once community income was included in models, other factors--including percentages of students in disadvantaged populations, percentages receiving special education, percentages eligible for free or reduced price lunch, percentages with limited English proficiency, school sizes, school spending levels, and property values--all failed to associate substantial additional variance.

Large uncertainties in residuals of school-averaged scores, after subtracting predictions based on community income, tend to make the scores ineffective for rating performance of schools. Large uncertainties in year-to-year score changes tend to make the score changes ineffective for measuring performance trends. In their present state, considered as a means to rate the performance of public schools, tenth-grade MCAS mathematics tests mainly appear to provide a complex and expensive way to estimate community income.

Massachusetts has experienced many education experiments and reforms over the past few centuries. (Note 14) During the 1990s Massachusetts education reform was driven by the McDuffy school finance lawsuit (McDuffy, 1993), originally filed in 1978. In its McDuffy decision, the Massachusetts Supreme Judicial Court said that Massachusetts funding disparities harmed the quality of education for some students, denying them education to which they were constitutionally entitled. This June, 1993, decision was widely anticipated. The Massachusetts Education Reform Act of 1993 (Education Reform, 1993) was signed less than a week after the decision was released.

A group called the Massachusetts Business Alliance for Education, (Note 15) organized in 1988 and led by the late John C. (Jack) Rennie, then CEO of the former Pacer Infotec, Inc., of Burlington, MA, (now the AverStar division of Titan Corp., San Diego, CA), and S. Paul Reville, then director of the Worcester Public Education Fund, wrote the reform bill sponsored by the Education Committee of the legislature. In 1991 the Business Alliance produced a document entitled Every Child a Winner. (Note 16) A story from the May 2, 1993, Northwest edition of the Boston Globe quoted former Rep. Mark Roosevelt, then House Education Committee Chair, as saying that the House education reform bill then pending "is essentially [the Business Alliance document]." In a publication of MassINC, Rennie is quoted as saying, "We bought change" (Walser, 1997). Most of this work was carried out in secret. As late as December, 1992, then Lt. Gov. Cellucci was calling on the Education Committee chairs, Sen. Thomas Birmingham of Chelsea and Rep. Roosevelt of Beacon Hill, to disclose their bill (Howe, 1992). Almost all the controversy generated by this legislation focused on its funding formulas. Until 1993, the public had hardly any knowledge of its sweeping changes in school policy and regulation. The following newspaper report was printed December 23, 1992 (Overdue, 1992):

"The bill also calls for higher student achievement and curriculum standards."

This was the most thorough description in mainstream news media from 1988 through 1992. The bill was released in an emergency legislative session of January 4-5, 1993, but it failed to pass.

Soon after the bill became public, education and public interest groups began to react. As reported in the Boston Globe on January 26, 1993, a coalition headed by Stephen Bing of the Massachusetts Advocacy Center predicted major problems with the legislation, including these (Ribadeneira, 1993):

• The reform bill will institutionalize unfair teaching practices such as using tests to track students into different ability levels.

• By substituting different certificates in place of the high school diploma, the bill will contribute to the dropout problem rather than ameliorate it.

• The legislation provides no mechanism for meaningful participation by parents or students in the development of a remedial education plan nor any opportunity to contest an inadequate plan.

Such objections were ignored. Neither the mainstream news media nor the Great and General Court gave these or other educationally oriented issues further attention in 1993.

Rep. Thomas Finneran of Mattapan, then chair of the House Ways and Means Committee, secretly inserted anti-abortion provisions in the bill, provoking a storm of protest (Howe, 1993a). Other House controversy centered on a salary cap for teachers, which was removed. Proposals for "school choice," charter schools and gambling revenues became the focus of activity in the Senate. The bill quickly became a hodge-podge of added provisions with no coordination. Many observers became skeptical about overall benefit. Geoffrey Beckwith of the Massachusetts Municipal Association was quoted as saying, "It certainly doesn't appear at this time that this bill will bring about any fundamental reform" (Howe, 1993b). In February, 1993, the Supreme Judicial Court heard testimony in the McDuffy case. In March, former Rep. Roosevelt began a (losing) campaign for Governor (Lehigh, 1993, Howe, 1993c). In April, the Edison Project, a business corporation, announced interest in privatizing Massachusetts schools (Nealon, 1993). By May, an impasse over "school choice" had developed, the then Senate President William Bulger of Boston demanding it and the then House Speaker Charles Flaherty of Cambridge rejecting it. At the time, the Business Alliance opposed the "school choice" and charter school amendments (Taylor, 1993). However, another business group calling itself "CEOs for Fundamental Change in Education" had appeared, dominated by banking and large business interests and supported by the Pioneer Institute. It was actively promoting charter schools and "school choice" through the Massachusetts Senate (Vennochi, 1993). After compromising with limits and delays on "school choice" and charter schools, the House passed the bill through second reading June 2 and the Senate passed it June 3. The Supreme Judicial Court released its McDuffy case decision June 15. Former Gov. William Weld signed the Education Reform Act on June 18, 1993.

In seven years under the Education Reform Act, state aid to Massachusetts public schools has grown from $1.3 billion to $3.0 billion per year, almost all the increase going to communities with low household incomes. For example, Holyoke, a low-income community, now receives over 90 percent of its school funding from the state, while Brookline, a high-income community, receives only about 10 percent (Mass. DoE, 2000e). In 1992, Holyoke spent less than 75 percent as much per student as Brookline, but now it spends about 95 percent of what Brookline does (School reform, 1992). Still, the Act has tended to provide more of a windfall for Holyoke's taxpayers than for its public school students.

Besides setting state commitments to equalize school funding, the Education Reform Act made many changes to Massachusetts education policy and regulations, including the following, as described by the Business Alliance (Taylor, 1993):

• New goals, standards and indicators of performance for schools, students and teachers

• Financial rewards to teachers and schools that excel

• Decentralized authority, limiting school committees to policy-making and oversight, making CEOs of superintendents, and giving hiring and firing power to principals

• Preschool for all 3- and 4-year-olds

• Expanded professional development for teachers

More than seven years later, some of these changes are only starting to be implemented.

Before 1996, the Massachusetts Board of Education regarded test scores as only one component of school accountability. In a 1993 policy advisory cited by Wheelock, 1999, the Board warned that an accountability system based primarily on test scores would be likely to produce harmful long-term consequences, including: exclusion of weaker students from the assessed pool of students; lowered morale among teachers and students; the loss of experienced educators from schools enrolling many disadvantaged students; distortion of instruction and curriculum to reflect test content and format; cheating and corruption of test scores.

Nevertheless the Board began development of testing programs to satisfy the provisions of the Education Reform Act, which eventually became MCAS. It appointed committees of educators and parents to help insure that tests were meaningful, fair and free from overt forms of bias (French, 1998). From 1993 through 1996, Massachusetts invested more than $2 million to support education reform study groups seeking ways to set high expectations for students (Antonucci, 1997a).

By March of 1996 the Department of Education had completed a Common Core of Learning (Mass. DoE, 1994), released six of seven planned curriculum frameworks based on it, and begun the development of MCAS based on the frameworks. In addition, it had announced plans (Mass. DoE, 1996a) to: award grants to school districts for assessment activities such as portfolio development; hold statewide conferences on local assessment strategies; publish examples of student work that meet the statewide standards so that districts have a model of what to strive for; develop a bank of assessment exercises linked to the curriculum frameworks for use by classroom teachers. All of these satisfy or support provisions of the Education Reform Act.

Development of MCAS took a sharp turn away from public participation (Mass. DoE, 1996b) after the appointment of John Silber as chair of the Massachusetts Board of Education in November, 1995 (Pawlack-Seaman, 1996). In August, 1996, Silber, former president of Boston University and an unsuccessful candidate for Governor, working with then Gov. Weld, engineered replacement of the 17-member Board of Education, including four African-Americans and Latinos, with a 9-member board, including several with ties to school privatization and charter schools, only one African-American and no Latinos (Jackson, 1996, Wong, 1996). At his first meeting with the Board, Silber said the Education Reform Act's underlying principle that all students are capable of learning at high levels was "rubbish" (Future, 1996). Responding to a demand for his resignation in 1997 he commented, "Some of the things that pass for learning disabilities used to be called stupidity" (Pawlack-Seaman, 1997). Soon after the Board replacement, the committees of educators and parents that had been formed to oversee curriculum frameworks, test development and other education reforms were disbanded (Antonucci, 1997b).

In December, 1996, Silber proposed a two-track system (Avenoso, 1996) with a general diploma awarded for passing the GED, a test introduced during World War II by Everett F. Lindquist, developer of the Iowa test series, and now administered by the American Council on Education. An honors diploma would be awarded for high scores on the Massachusetts test series. Silber was forced to abandon the plan in January, 1997, when his personally chosen Board of Education refused to support it (Leung, 1997). However, a legacy of Silber's proposal remains, the view that MCAS should be aimed at the exceptional student. In August, 1999, the Business Alliance revived the two-track concept (Still, 1999) with a proposal to award general diplomas to students who satisfy "essential requirements in English and math." The Business Alliance did not specify how this would be administered, and the Department of Education and Board of Education still oppose the concept. What they have done instead is to make a "competency determination" required by the Education Reform Act for a high-school diploma depend on achieving relatively low MCAS test scores, (Note 17) answering about 40 percent of the questions. (Note 18) A "certificate of mastery," as specified by the Act, is to be awarded for much higher scores, (Note 19) answering about 80 percent of the questions to achieve an "advanced" rating on one or more tests.

After the loss of two Education Commissioners in rapid succession (Battenfield and Pressley, 1999), Silber resigned during a struggle over a new Commissioner in March, 1999. The outcome of the controversy (Estrin, 1999) was replacement of Silber by James Peyser (see Peyser, 1996, and Peyser, 1998), head of the reactionary Pioneer Institute, tied to school voucher and privatization movements, and retention of the compliant acting Commissioner David Driscoll. Since the Silber era, MCAS development has been closely monitored by Board of Education member Abigail Thernstrom, a fellow of the Manhattan Institute, (Note 20) and hired consultant Sandra Stotsky (see Stotsky, 1999), a writer for the Fordham Foundation and now an Associate Commissioner of Education. Both of these right-wing foundations have supported forms of school privatization. Four rounds of MCAS tests have now been administered, in the spring of 1998, 1999, 2000 and 2001. The Board of Education has made the questions used in scoring available to the public, although they have not disclosed their standards for evaluating essay questions or all the details of their approach to computing scores.(Note 21) Students in religious-run and other private schools and students being taught at home are not required to take or pass MCAS tests. Bills have been filed but have not been enacted to include private schools in testing and to exclude charter schools. A system of "school accountability" has been defined by the Department of Education (Mass. DoE, 1999b). It is based entirely on MCAS scores, a violation of Education Reform Act requirements. (Note 22)

MCAS has been heavily promoted by a business-oriented group organized as Mass Insight Education and Research Institute, Inc., in Boston, founded in 1997 by registered Massachusetts lobbyist William H. Guenther, who is its president. (Note 23) Guenther is also involved with three other public relations organizations, Mass Insight Corp., in Cambridge, Opinion Dynamics Corp., in Cambridge, and New England Economic Project, in Walpole. Mass Insight Education and Research Institute is a non-profit corporation that coordinates several policy groups and has close relationships with business and education executives. Leaders of its "Campaign for Higher Standards" have included Gloria Larson, former Mass. Secretary of Economic Affairs, the late John C. Rennie, former Chairman of the Massachusetts Business Alliance for Education and Vice-Chairman of AverStar, Inc., and Cathy Minehan, President of the Federal Reserve Bank of Boston. Leaders of its "Coalition for Higher Standards" include James Caradonio, Superintendent of Worcester Public Schools, and Thomas Payzant, Superintendent of Boston Public Schools. Its board of directors has included Maura Banta, Manager for External Programs at IBM Corporation, John Rennie, Abigail Thernstrom, Senior Fellow at the Manhattan Institute and member of the Massachusetts Board of Education, and Bruce Tobey, Mayor of Gloucester. Financial supporters of Mass Insight Education and Research Institute include BankBoston (now FleetBoston Financial), State Street Corp., Bell Atlantic (now Verizon), Boston Edison (now an NSTAR division), Liberty Mutual Group, PricewaterhouseCoopers, Goodwin, Procter & Hoar, AverStar, Inc. (now a division of Titan Corp.), Gorton's Seafoods, Hewlett-Packard, IBM and Intel.

Mass Insight publications promoting MCAS have been distributed to public schools through the Massachusetts Board of Education, (Note 24) and Mass Insight has received public funds for its services. Mass Insight has been cited in minutes of the Board of Education as a source of policy initiatives, (Note 25) including a proposal to use a score of 220 on tenth-grade language arts and mathematics tests as the initial "competency determination" for high-school graduation, which was adopted by the Board in November, 1999. Mass Insight presents a simple but misleading picture of MCAS, saying that it measures "skills that students will need after graduation―at college or on the job" (Why, 1999). No such significance has ever been demonstrated for MCAS or other state accountability tests.

Massachusetts schools are often castigated by newspapers and politicians as mediocre, (Note 26) but actually they are superior. In the October, 1999, Boston Magazine, Jon Marcus wrote:

"According to assessment tests and other measures, Massachusetts schools are among the nation's best. Students here rank fourth nationally in reading, sixth in math, and eighth in science on the National Assessment of Educational Progress, administered by the US Department of Education. They scored higher this spring in reading than 69 percent of their peers across the country on the Iowa Test of Basic Skills; a third of the state's third graders were at the advanced level, compared to 19 percent nationwide. A Boston College correlation of NAEP results with international tests found that in eighth-grade science Massachusetts students performed as well as, or better than, their counterparts in 40 out of 41 other countries, including Germany and Japan; only kids in Singapore were rated higher. More students study algebra and upper-level math and science than the national average, and Massachusetts also has the fifth-lowest high school dropout rate, the nation's highest percentage of graduates who enroll in college, and the third-highest proportion of students who take the SAT. Massachusetts students' SAT results have risen steadily since 1994; last year, they were the highest in a decade." (From Marcus, 1999.)

Such a contrast between political bombast and educational reality has become common. Part of the long record of declining SAT scores in the 1960s and 1970s, for example, had a straightforward cause, the rapidly expanding number of students taking the tests, including many low-income students and students with lower grades who would not have taken them in prior years (Koretz, 1992, Berliner and Biddle, 1995, Berliner and Biddle, 1996). When education researchers looked at comparable groups of students, SAT scores were gradually rising during much of this period; unadjusted averages began to rise as the growth in the number of test takers slowed. With the gratuitous abuse regularly heaped on public schools during this time, few members of the public would have guessed that some of the real trends in scores were positive. Even now, many politicians and most news media find the actual results inconvenient; they prefer simple, strident bashing of public schools, uncomplicated by facts.

Many observers and columnists have commented on the complex language, mental tricks and obscure bits of knowledge found in MCAS questions (see, for example, Vaishnav, 2000, and Kohn, 2000). How elitist are the MCAS tests? One way to look at this is to ask the fraction of questions that must be answered to pass them and the fraction of students who cannot do this. Table A1-1 compares the current graduation level tests in Massachusetts (Note 27) (10th grade MCAS) with those in New York (Note 28) (the revised Regents series) and Texas (Note 29) (the TAAS series).

Massachusetts, with by far the lowest passing score, has by far the highest rate of failure. Yet year after year, nationwide measures of academic performance rate Massachusetts students well ahead of those in New York and Texas. (Note 31) Passing an MCAS test says little about the education imparted through many years of schooling. On an MCAS tenth-grade mathematics test, the difference between passing and failing can be getting 24 questions right rather than 23 (see Mass. DoE, 1999d, and Mass. DoE, 1999c).

A recent study performed by Catherine Horn and others at Boston College (Horn, et al., 2000) showed that a barely passing score on the tenth-grade MCAS math test was approximately equivalent to the 50th percentile score for the PSAT math test. Students taking the PSAT are aiming for college. Many are taking the test as part of applying for National Merit and other scholarships; they tend to be good students. Therefore it should not be surprising when half or more of the general student population may "fail" the current tenth-grade MCAS math test.

A large share of MCAS test questions is aimed at students with exceptional skills and knowledge rather than at typical students. If Massachusetts designed tests to measure competence rather than mastery, it would be setting much higher passing percentages. If Massachusetts genuinely cared about assessing student skills and knowledge, it would satisfy Education Reform Act requirements (Note 32) calling for a "variety of assessment instruments," including "consideration of work samples, projects and portfolios," facilitating "authentic and direct gauges of student performance," and it would provide for circumstances of special education students, students entering the public schools from households that speak a first language other than standard English, (Note 33) and students whose immediate aims focus on employment rather than higher education.

The Massachusetts Board of Education has ample access to information of this sort and has received many recommendations to improve its practices and make its system of assessments more realistic and fair. It has had more than $25 million to spend on developing MCAS (Szechenyi, 1998). It is also well aware that "high-stakes" testing systems in other states have sharply narrowed the school curriculum (Note 34) and increased the population of school dropouts, (Note 35) who are likely to be eligible only for the "McJobs" of the future. So far, however, members of the Massachusetts Board of Education remain rigid, programmatic and hostile to facts that do not support their policies. Their attitude does not originate from lack of information or resources.

MCAS, like the other "achievement tests" used in state accountability systems, has never been shown to predict success in adult life to any greater degree than could be done with a knowledge of student backgrounds. Instead of trying to show practical significance the Board assumes it, in proposing to use this test as the sole state criterion to deny high-school diplomas and state college eligibility to low-scoring students, making it difficult for them to find responsible jobs and other forms of advancement. Students from households that already have the least suffer the most from such a system, tending to widen an economic gap between haves and have-nots in our society, already among the greatest of the industrial nations.

Massachusetts municipal school districts support either jointly or individually more than 30 "vocational," "vocational-technical," "technical," "agricultural-technical" and "agricultural" high schools, (Note 36) most of which the Department of Education recognizes as separate school districts. Table A2-1 includes the 29 vocational schools that are now operated as separate school districts plus the "technical" high school operated by the City of Boston. (Note 37)

Vocational schools typically provide instruction for grades 9 through 12 and devote about half their instructional time to traditional academics and about half to vocational training. Some communities, including Cambridge, Quincy, Revere and Waltham, provide vocational education in the same facilities as academic programs. In contrast to academic high school programs, which mainly concern themselves with preparing students for college, vocational programs train students for specific occupations, and their faculty tend to rate themselves according to graduates' success in finding satisfactory employment in those occupations.

Vocational schools, represented through the Massachusetts Association of Vocational School Administrators, have presented the Massachusetts legislature with a bill to decouple those schools from MCAS and substitute a special examination system based on the occupational categories for which they provide training, sponsored by Sen. David Magnani of Framingham (Magnani, 2000). They are well aware, as the foregoing table shows, that their students score far below state averages on MCAS tests; but they claim that MCAS tests are directed toward a curriculum that they do not teach and cannot teach without weakening their key programs. Table A2-2 identifies some of the major sources of information for Massachusetts vocational schools available on the Internet.

Appendix 3: Metropolitan Boston MCAS Mathematics Scores

Schools marked with asterisks (*) in Table A3-1 are those providing vocational education in the same facility as academic programs.

Data in Table A3-1 are tenth-grade MCAS mathematics test scores for 1998-2000, averaged by schools, and numbers of test participants per school, obtained from the Massachusetts Department of Education (Mass. DoE, 2000h) and adjusted for percentages of students enrolled in schools but not taking the test. Adjustment formulas are as follows: