This article has been retrieved   times since April 5, 2004

The Achievement Gap: Should We Rely on SAT Scores to Tell Us Anything About It? Dale Whittington Shaker Heights (OH) City School District Citation: Whittington, D., (2004, April 5). The achievement gap: Should we rely on SAT scores to tell us anything about it?. Education Policy Analysis Archives, 12(12). Retrieved [Date] from http://epaa.asu.edu/epaa/v12n12/. Abstract Increasing numbers of students taking the SAT have declined to identify their race/ethnicity.  I examined the impact of non-respondents on the validity of reported racial/ethnic differences and year-to-year changes in test performance.  Using an analysis reported by Wainer (1988) and SAT data from 1996 to 2003, I confirmed Wainer’s findings that non-respondents prevent accurate estimations of group differences based on SAT data.  I then explored the impact of College Board press release information on news reports about the achievement gap.  I found frequent reports of racial/ethnic differences in SAT scores and year-to-year changes in scores but negligible consideration of non-respondents.  Press releases and media reports should include information about non-respondents and their impact on accuracy of reported differences based on race/ethnicity. Introduction The term “achievement gap” has taken on particular and important meanings in the past decade. “The achievement gap” has become a shorthand way to refer to differences in academic achievement between European Americans and members of minority groups who historically have been disenfranchised. For some, the gap refers exclusively to differences between African Americans and European Americans. For others, it refers to a broader group of students: those who aren’t facile in English, the poor, or members of other disadvantaged ethnic groups.  Regardless of who is included in one’s definition, the literature abounds with descriptions of gaps in student performance on test scores, which are probably the most commonly used indicators of student achievement (Jencks & Phillips, 1998; Herrnstein & Murray, 1994; Lee, 2000; Kober, 2001). Additional evidence of a gap has been found in data related to other indicators of achievement, such as grades, dropout rates, college attendance or earnings. (Portes, 1996; Ferguson, 2001; Kane, 1998; Vars & Bowen, 1998; Johnson & Neal, 1998; Roderick & Cambrun, 1999) National organizations of schools dedicated to closing the gap, such as the Minority Student Achievement Network, have been formed to address this phenomenon. The benchmark against which minority achievement is measured is White/European-American performance, and closing the gap usually means increasing minority achievement relative to that of White/European Americans. Reasons for the gap have been explored; the resulting explanations have been legion. Some have focused on the historical legacy of racism, prejudice and segregation (Spring, 2000; House, 1999). Others have examined factors affecting children’s readiness for school and their ability to learn in school, such as the role of poverty and other differences associated with SES (Ferguson, 2001; Barton, 2003), cultural differences in language or in adaptation to school (Mercado, 2001, Portes, 1996; Portes, 1999, Ogbu, 1999; Ogbu, 2003), family and parenting (Okagaki & Frensch, 1998; McAdoo, 1978). Another arena of inquiry has focused on how well education serves its varied constituents.  Some have looked at inequities in resources and opportunity to learn (Kozol, 1991; Barton, 2003; Mickelson, 2001); others have looked at how schools and educators respond or relate to student diversity (Spring, 2000; Pollock, 2001; Ferguson, 1998; Delpit, 1996; Cohen & Steele, 2002), or how they encourage (or fail to encourage) academic excellence (Ogbu, 2003; Barton, 2003). Finally, many have discussed the qualities of tests or teachers’ assessments of students that can contribute to extraneous differences, i.e. error and bias. (Airasian, 2001, Gould, 1981; Valencia & Suzuki, 2001)   One common research focus has been change in the gap over time, and change typically is examined using test data. Studies have traced the ups and downs of minority student test performance, usually compared to that of European Americans. These studies employ various types of test scores, most often using NAEP and college admissions scores such as the SAT. (Kober, 2001; Lee, 2000; Miller, 2003; Powell & Steelman, 1996) For several decades, NAEP report cards have been issued that describe student achievement in eight subject areas; these report cards not only provide a current snapshot, but also depict longitudinal change. The accountability movement has triggered issuance of school, district and state report cards, and with the passage of the No Child Left Behind act, these report cards must include disaggregated data and report differences in test performance from one year to the next. Finally, each summer, the College Board and ACT issue reports on student performance on college admissions tests, notably the SAT and ACT assessment. These reports link student achievement to a variety of variables including race/ethnicity. NAEP reports are based on the results of samples that were selected to be representative of the national population of students. Data from state achievement tests are based on the performance of all students at a particular grade level. However, college admissions tests are administered on a self-selected set of students who have interest in attending college. This self-selection means that the data are not representative of students in the nation or a particular state in any year. Furthermore, the percentage of high school students who intend to apply to college fluctuates from year to year. One consequence is that year-to-year differences in student performance can be based on true changes in student knowledge or ability, on changes in who takes the test, or both.  These sampling issues present serious problems with using college admissions test scores to make inferences about the academic qualifications of students in the nation, to make comparisons among states, or to track year-to-year differences in test performance. ACT and the College Board are careful to warn against using SAT or ACT test results to make comparisons among states (College Entrance Examination Board, 2002; ACT, 2003). ACT’s web site includes information briefs that explain how changes in a school’s test performance over time can be influenced by a variety of factors, including who decides to take the test (ACT, 2004). The College Board’s guidelines on test use include a warning against using aggregate data to make judgments, not only about states, but also about schools or districts. However, they are less explicit about comparisons among groups of students disaggregated by, among other things, race/ethnicity (College Entrance Examination Board, 2002). Racial/ethnic comparisons of SAT or ACT data are particularly suspect because each year a substantial portion of students do not report their race/ethnicity on the survey questionnaires that they complete when registering for the tests they are about to take. This is not a recent problem. Fifteen years ago, Howard Wainer (1988) examined the impact of this missing information on the accuracy of racial/ethnic comparisons using data from SAT’s administered from 1981 to 1985. Among his findings are the following: The percentage of students not reporting their race/ethnicity (to be called non-respondents here) was substantial each year (12 to 14%)--large enough to be called the “second largest minority group” taking the test. Non-respondents, as a group, were not similar to or representative of the students who did report their race/ethnicity. While there were variations in the performance of non-respondents from year to year and in their performance relative to the nation, their underperformance was somewhat consistent (22 to 26 points below the national SAT-V means and 21 to 28 points below the national SAT-M mean). The error caused by the missing racial/ethnic information for the non-respondents overwhelmed any differences detected each year among disaggregated groups as well as any changes in test performance gaps over time. He concluded, “…the nonresponse to ethnic identifiers is sufficient to introduce noise of a greater magnitude than the changes being interpreted as real.” (Wainer, p. 778). Wainer’s data suggest that the non-respondent group, while not representative of those who did report their race/ethnicity, were nevertheless comparatively stable in terms of their numbers and test performance. Since the mid-1990’s, however, this group has changed considerably.  Table 1 reports the numbers of students (in thousands) in each group taking the test each year covered by this study. It also reports the combined number of non-white respondents. Examination of the table reveals that, with the exception of American Indians, the numbers of students in each non-white group increased since 1996. By contrast, the numbers of white students increased, then decreased. Finally, both the proportion and number of non-respondents more than tripled.   Table 1. Students (in thousands) from Each Racial/Ethnic Group Taking the SAT by Year Year Am. Indian Asian Am. African Am. Mexicn Am. Puerto Rican Other Latin/Hisp. Other Total Non-White White No Resp. 1996 9 84 107 37 13 32 28 310 681 94 1997 11 89 110 40 13 33 31 327 694 106 1998 10 94 115 41 14 36 36 345 704 123 1999 8 96 119 43 14 38 38 357 718 146 2000 8 97 120 45 14 39 39 360 712 188 2001 8 102 121 47 14 40 39 370 704 202 2002 8 103 123 48 14 42 39 377 699 253 2003 7 101 126 50 15 43 39 381 670 355 Figure 1 reports change in the percentage of examinees who were non-respondents on the question of racial/ethnic identity. In the mid-nineties, the percentages resembled percentages in the data reported by Wainer (1988), but this group has steadily increased so that by 2003, they made up one-fourth of all those taking the test. In the process, they became, to use Wainer’s parlance, the largest minority group taking the SAT.  It should be noted that throughout this time period, non-respondents were the only “racial/ethnic” group that has been majority male. Figure 1. Percentage of SAT 1 Examinees Not Reporting Their Race/Ethnicity In sum, then, the recent escalation of interest in the achievement gap, continued use of tests to describe that gap, and the dramatic growth of the non-respondent group raise two questions. First, how have students who declined to indicate their race/ethnicity from 1996 to 2003 resembled those reported by Wainer (1988) and how has this affected the validity of his conclusion regarding the amount of noise that overwhelms the use of SAT scores as an index of the achievement gap and changes in that gap? Second, are there practices in releasing and disseminating information about college admissions test scores and racial/ethnic performance on those tests that seem to foster inappropriate use of these scores when describing the achievement gap to the public in general and the education establishment?  To answer the first question, I used data released on the College Board’s web site to track the test performance of non-respondents and to replicate the study conducted by Wainer (1988). To answer the second question, I examined the College Board’s press release information from the summer of 2003, each state’s data on non-respondents, press releases about the SAT score results for 2003 issued by state departments of education throughout the United States, and articles from selected local and national newspapers about SAT results. I also revisited articles about the achievement gap that have included SAT information. In all cases, I looked for two kinds of information: 1) data disaggregated by race/ethnicity and discussion of that data and 2) information about non-respondents and their impact on our ability to make inferences about racial/ethic groups or differences between groups. Question 1 How have students who declined to indicate their race/ethnicity from 1996 to 2003 resembled those reported by Wainer (1988), and how has this affected the validity of his conclusion regarding the amount of noise that overwhelms the use of SAT scores as an index of the achievement gap and changes in that gap? Method. Using College-Bound Seniors data reported annually by the College Board on its web site, I determined how many students took the test each year and how they performed on the test. This permitted a comparison with those examinees employed in Wainer’s analysis. Then I employed Wainer’s procedure for estimating the representation of white students in the non-respondent group and determining the degree to which these non-respondents were affecting the validity of examining test performance disaggregated by race/ethnicity. For this procedure, Wainer used the numbers of students in each racial/ethnic group and the mean SAT-V and SAT-M scores for each group to develop estimates of the percentage of non-respondents who were white. He employed this approach to test the assumption that the mean scores of the non-respondents from each group were the same as those of students who reported their race/ethnicity. Were this to be the case, then the reluctance of some to report their race/ethnicity would have little or no effect on the validity of differences between groups that were found for those based who did report their race/ethnicity. For each year, he made two estimates, one based on the math scores and the other based on the verbal scores. These estimates were compared for consistency. The degree of difference between them indicated how much the assumption that non-respondents were like those who responded was violated. It should be noted that this estimating procedure yields only rough estimates. Wainer estimated the proportion of whites in the non-respondent group using the following formula:   P(MSATwhite) + (1-P) (MSATnonwhite) = MSATnonresponse, Where P = the proportion of white non-respondents, (1-P) = the proportion of non-white non-respondents, MSATwhite = SAT mean of self-identified white examinees MSATnonwhite = SAT mean of examinees identifying themselves as members of another group, and MSATnonresponse = SAT mean of non-respondent examinees. Like Wainer, I used this formula to estimate the percentage of white non-respondents twice, once using SAT-Verbal scores and once using SAT-Math scores. I repeated this analysis for each year from 1996 to 2003. Findings.   Table 2 reports the SAT I mean scores for the verbal portion of the test for each year from 1996 to 2003. In general, the amount of change for any self-identified group over that time period was negligible. The largest change for self-identifying students was an increase in 12 scale-score points in the mean for Asian American students. In contrast, the change for non-respondents was dramatic, an increase of 24 scale-score points. This change in non-respondents’ mean scores also contrasts with the means reported by Wainer. The difference between the highest and lowest non-respondent means in that study was 10 points, and there was no discernable pattern of change. Table 2. SAT Verbal Mean Scores for Each Year YEAR Am. Indian Asian Am. African Am. Mexican Am. Puerto Rican Other Latin/Hisp. Other White No Resp. Total Non-White 1996 483 496 434 455 452 465 511 526 486 466 1997 475 496 434 451 454 466 512 526 489 466 1998 480 498 434 453 452 461 511 526 490 467 1999 484 498 434 453 455 463 511 527 492 467 2000 482 499 434 453 456 461 508 528 495 467 2001 481 501 433 451 457 460 503 529 497 466 2002 479 501 430 446 455 458 502 527 501 464 2003 480 508 431 448 456 457 501 529 510 466 Another way to look at non-respondents’ SAT performance is to examine how they compared to the national mean. Figure 2 shows that the SAT I verbal performance of non-respondents steadily increased so that by 2003, their mean score slightly exceeded that of the national mean. In other words, not only has this group grown, but it now includes more high-performing examinees. Figure 2. Comparison of SAT Verbal Scores: National Mean vs. Mean of Those Not Reporting Their Race/Ethnicity Examination of the SAT I results for the mathematics portion of the test yields similar results. As can be seen in Table 3, the largest increase in mean scores for any identified racial/ethnic group was that for Asian-American scores, an increase of 17 scale-score points. By contrast, the mean for non-respondents increased by 31 scale-score points. The comparison to the change in the national mean depicted in Figure3 shows an increase in scores, with the non-respondents outperforming the national group of examinees in 2003. In other words, the non-respondents became an increasingly able group with respect to math from 1996 to 2003, a pattern not evident in the math scores reported for 1980 to 1985 by Wainer (1988). Table 3. SAT Math Mean Scores for Each Year YEAR Am Indian Asian Am African Am Mexicn Am Puerto Rican Other Latin/Hisp. Other White No Resp. Total Non-White 1996 477 558 422 459 445 466 512 523 494 479 1997 475 560 423 458 447 468 514 526 502 480 1998 483 562 426 460 447 466 514 528 503 483 1999 481 560 422 456 448 464 513 528 505 480 2000 481 565 426 460 451 467 515 530 509 484 2001 479 566 426 458 451 465 512 531 510 484 2002 483 569 427 457 451 464 514 533 516 485 2003 482 575 426 457 453 464 513 534 525 485 Figure 3. Comparison of SAT Math Scores: National Mean vs. Mean of Those Not Reporting Their Race/Ethnicity These changes in the non-respondents’ mean test performance on the verbal and math portions of the SAT I , in addition to the dramatic increase in their numbers, suggest that there have been substantial and steady changes in the non-respondent group since the mid-1990’s. These changes not only made the group more variable over time, compared to their counterparts in the early 1980’s, but also reinforce the need to re-estimate the composition of this group with respect to race/ethnicity and compare it to the composition of those who do report their race/ethnicity. Using the same procedure employed by Wainer (1988), I estimated the percentage of whites in the non-respondent group twice for each year covered by this study, once based on verbal scores and once based on mathematics scores. Then I used the difference between the verbal and mathematics-based estimates to calculate the difference in the number of estimated white examinees in the non-respondent group for each year. Table 4 reports the results. Table 4. Estimates of Percent of White Non-Respondents based on SAT-V and SAT-M and Difference in the Estimated Number of White Non-Respondents, based on the two estimates Year Percent based on SAT-V Percent based on SAT-M Difference in Estimated Number of White Non-Respondents 1996 33.6 34.3 643 1998 39.5 47.4 9,244 1997 38.6 44.7 6,481 1999 41.8 52.2 15,062 2000 46.3 54.7 15,713 2001 49.0 55.3 12,646 2002 58.7 64.7 15,105 2003 70.0 81.7 41,507 The range of non-respondents who are estimated to be White based on the verbal and math means is much more varied for the verbal portion of the test for this time period (33.6% to 70.0%) than for the time period reported by Wainer (43% to 51%). For the mathematics portion of the test, the estimates based on recent means tend to be larger and more varied (34.4% to 81.7%) compared to those reported by Wainer (1988) (29% to 46%). The differences in the estimated number of white students ranged from very small (632) to more than double the largest estimate reported by Wainer (1988) (41,507 vs. 18,000).  Wainer develops a hypothetical “extreme case” to illustrate the consequence of these differences on the “real” means of African Americans, compared to Whites. By holding numbers of examinees and means constant, he estimated that the 1980 mean verbal score for African Americans would have to change about 16 points in order for the percentage of non-white respondents based on the SAT-V score mean to equal the estimated percentage of non-white non-respondents based on the SAT-M score mean. Based on 1984 data, the requisite change would have to be 35 points. Following Wainer’s lead, I found that the smallest difference, based on 1996 data, would be 1.8 scale-score points, slightly larger than the 1-point change reported for 2002 to 2003. For all other years, the differences are much larger, ranging from 17 points in 1998 to 122 points in 2003. Hence, Wainer’s conclusion that the error associated with non-respondents dwarfed the size of any year-to-year gains reported is confirmed.  It should be noted, furthermore, that the implications of this finding cannot be applied in a consistent fashion to each of the states. The map in Figure 4, developed using SYSTAT’s boundary map of the continental United States (SYSTAT, 2000), shows the range of percentages of SAT examinees declining to report their race/ethnicity by state. The percentages range from a low of 12.4 percent (North Dakota) to a high of 30.4 percent (Connecticut), and the percentage of non-respondents is moderately correlated (r =.61) with the participation rate reported for each state. Suffice it to say that this analysis would need to be carried out for each state in order to determine the impact non-respondents have on the racial/ethnic findings in that state. Figure 4. U.S. Map Showing Percentage of Examinees Declining to Specify their Race/Ethnicity on the Student Descriptive Questionnaire Note 1 Note: Data source: non-respondent data reported by the College Board (2003).  Question 2. Are there practices in releasing and disseminating information about college admissions test scores and racial/ethnic performance on those tests that seem to foster inappropriate use of these scores when describing the achievement gap to the public in general and the education establishment? The initial source of information about SAT results for 2003 appears in a College Board press release and associated tables, charts and reports that were issued on August 26, 2003 (College Board, 2003). The press release includes text reporting overall changes in performance and participation, differences by gender, changes in performance by members of various ethnic groups, upcoming changes in the SAT Verbal test and a “snapshot” of test takers.  Race/ethnicity is included in statements regarding the diversity of the test-taking population: This year saw the largest increase in the number of SAT takers in more than 15 years. Thirty-eight percent of SAT takers are first-generation college-bound students. The proportion of minority students taking the SAT is at an all-time high of 36 percent, up 1 percentage point from last year and 6 points from 10 years ago. “Higher SAT scores, a record number of test-takers, and more diversity add up to a brighter picture for American education. While we certainly need to make more progress, the fact remains that we are clearly headed in the right direction,” said College Board President Gaston Caperton. Thirty-six percent of SAT takers in the class of 2003 were minorities. The number of Mexican American SAT takers increased by 56 percent between 1993 and 2003. SAT takers in the Other Hispanic category increased by 50 percent during the same period. Race/ethnicity also figured into statements about test performance: The overall verbal scores were aided by a strong showing from Asian American SAT takers, whose mean verbal scores were, for the first time, higher than the national mean. Additionally, Mexican American and African American SAT takers improved their average scores by two points and one point, respectively, from a year ago. In fact, virtually all ethnic and racial groups showed stronger performance on their verbal scores compared to a year ago. Accompanying the text of the press release is a set of 18 graphs and tables depicting, among other things, the diversity of the test-takers (in one table and one graph), the changes in college bound students over time (four graphs/tables), plus racial and ethnic differences in high school preparation, grades and test performance (eight graphs/tables). Nowhere is there a statement about non-respondents, nor are they included in any table or graphic in the press release. This information does appear in Table 4-1 on page 10 of the College Bound Seniors report that can be accessed in a box entitled “Archives” next to the press release documentation on the web site.  In other words, while it is possible to find out that many examinees are declining to report their race/ethnicity, it is not evident in the materials featured as part of the press release. Furthermore, the language of the press release and its featured tables and graphs makes assertions about minority and non-minority SAT test takers that may not be true. The number of non-respondents simply overwhelms any trends that can be discerned from the information about the respondents. SAT results are treated as major news by most states’ departments of education, as well as the press. On the same day as the College Board’s press release, or soon thereafter, 29 departments of education in the various states and the District of Columbia issued press releases about the SAT results for 2003. (Six issued no press releases in 2003, and 16 issued press releases but none about the SAT. Either they reported nothing on testing (n=3) or reported about other test results such as NAEP, state tests, and /or ACT results (n=13).) Of the 29 press releases about the SAT, 18 included information about race/ethnicity. Sixteen of them included text and/or tables about racial/ethnic test performance. Eight of them included comments or tables pertaining to the diversity of test-takers.  The number or percentage of students declining to indicate their race/ethnicity appears in five of these press releases and accompanying materials. In three cases, tables include non-respondent numbers. The Texas press release reports an increase in the percent of non-respondents in Texas and the nation.  The most detailed set of information appears in the Florida press release and accompanying report entitled SAT Trends: Florida and the Nation.   The press release itself merely comments on the change in scores compared to 2002: “Florida’s average verbal score rose two points, due largely to higher scores among Hispanic, African-American and Asian males.” The report’s summary includes several bulleted items about the racial/ethnic composition of the test takers. Example: “Nationwide, the percentage of minority test takers has also been increasing, but at a slower rate than in Florida. In 1988, minorities represented 23% of the test takers nationwide, about the same as in Florida; by 2003 the percentage had increased to 36%, with Asians, whose scores are typically well above average, representing 9.6%, compared to 4.4% for Florida.” (p. i) SAT Trends’ introductory summary includes a set of bulleted items entitled “ SAT Scores by Racial-Ethnic Groups.” Not all of these bulleted items discuss test performance; they also cover diversity of the test-taking group, first language, and income. One item provides the most detailed discussion from any press release about non-respondents and how they have changed over time: An increasing percentage of test takers are declining to indicate their race-ethnicity. In 2002, 19% of test takers in both Florida and the U.S. did not do so. The number of non-respondents in 2003 rose to 24% for Florida and 25% for the U.S….In past years, those who did not provide this information had lower average scores than those who did. In 2003 the trend was reversed…This break in the trend makes any changes in scores by race problematic…. “ (p. iv). Despite this warning, however, five of 21 pages of tables included in this report focus on racial/ethnic differences in one way or another. A content analysis of a representative sample of newspapers or televised news reports is beyond the scope of this research. However, I did gather articles about the 2003 release of SAT scores from newspapers from various cities in the United States; many of them have a national presence in the sense that they are distributed nationally, or other journalists frequently cite them as sources of news information. Akron Beacon Journal Atlanta-Journal Constitution Boston Globe Chicago Sun-Times Christian Science Monitor Dallas Morning News Denver Post Indianapolis Star Las Vegas Review-Journal Los Angeles Times Miami Herald New York Times Philadelphia Inquirer San Francisco Chronicle Seattle Times St. Petersburg Times Wall Street Journal Washington Post All 18 of these newspapers published at least one article on the SAT. Twelve of them discussed racial/ethnic performance in one or more of the following ways: Differences among groups at the national level Differences among groups at the state or local level Changes in group members’ performance compared to last year or past years The alleged presence of bias in the test (cited FairTest) Possible causes for differences, such as differences in school funding, SES, or inequalities in education based on race, ethnicity or income Ten of them discussed the participation of students of various racial/ethnic groups in the College Board testing program. Some articles mentioned the level of participation; others discussed increases in participation. A quotation from the president of the College Board, Gaston Caperton, was often included: “Higher SAT scores, a record number of test takers, and more diversity add to a brighter picture for American education.” News sources designed to serve a national audience, either a lay audience or one of educators, were mixed in their reporting on SAT scores for the class of 2003. Major national news magazines—Time, Newsweek, and U.S. News and World Report—did not report SAT results. However, USA Today included SAT results in its August 27 issue and of the 15 paragraphs in the article, five discussed the achievement gap. Education Week, in an article dated September 3, 2003, reported on the increase of SAT and ACT examinees and the questionable level of college preparation coursework these students have had. Part of the article focuses on the level of high school courses these students have taken; the last half of the article focuses on racial/ethnic differences in test performance and why such differences exist. Black Issues in Higher Education published an article on September 11 that, like the Education Week article, discusses differences in college preparation for students from various racial/ethnic groups. The article includes two tables, one based on ACT scores, the other based on SAT scores, that report means for disaggregated groups. Of note, the ACT table includes two additional categories: "Prefer not to respond" and "No Resp.". The College Board table does not. Last, but not least, the United States Department of Education, on August 28, 2003, issued a statement about SAT scores as evidence of the achievement gap and disparities in the education system. In his October 8, 2003, prepared remarks for the High School Leadership Summit, Secretary of Education Rod Paige stated that the disparity continues between the education of disadvantaged or low-income students and that of other students. The speech goes on to use SAT performance of African American and Hispanic students as the sole illustration of this point. The only news source that reported on the presence of students who declined to indicate their race/ethnicity was The Miami Herald.  Its August 29, 2003, article on the SAT scores of Broward County School District high school seniors states: “Results were broken down by race and ethnicity, but it was difficult to determine how Broward was doing in narrowing the achievement gap between black and Hispanic and white and Asian students. The reason: Almost 1 in 4 students chose not to list their race or ethnicity. The number of students who didn’t answer increased by almost 100 percent.”   The source of this insight is unclear. The article refers to Katherine Blasik, Broward’s director of research and evaluation. It also reports state results, indicating, perhaps, familiarity with Florida’s report on SAT trends. Nevertheless, the readers of the Miami Herald were cautioned not to look at the SAT as a source of information about the achievement gap. Discussion Taken together, the College Board’s press release, those of various states, and news reported from large urban and national news sources suggest that the information the public reads about the SAT is grounded in what the College Board reports in its press release about SAT results. If one were to depend on the information provided in the press release without referring to the College Bound Seniors tables, several incorrect inferences could be made: That the magnitude of “achievement gaps” among racial/ethnic groups can be determined from SAT data That changes in the SAT performance of students from various racial/ethnic groups can be determined That the diversity of those taking the SAT can be determined and tracked over time That racial/ethnic differences in courses taken, income, first-generation college enrollment, high school GPA can all be determined from College Board information Wainer’s results make it clear that even when the proportions of non-respondents was smaller, there were enough of them to create noise that overwhelms any evidence of change over time in test scores reported for students from various racial/ethnic groups or any evidence of group differences based on SAT results. The results of this study not only confirm Wainer’s results but also show that the noise generated by recent groups of non-respondents has increased from overwhelming to deafening.  While Wainer does not discuss inferences about racial/ethnic differences of other kinds, the magnitude of the non-respondent rate and the change in this group’s test performance suggest that the other inferences associated with racial/ethnic differences reported in the College Board’s press release should be treated as suspect as well. The graphs in Figure 4 illustrate why this should be the case. The first two breakdowns report take-takers based on data reported in the College Board’s press release and College Bound Seniors for the class of 2003; the third and fourth breakdowns make use of the estimates of white non-respondents from Table 4. The lower estimate, based on SAT-V 2003 score means, yields a percentage of white test-takers in Breakdown 3 that exceeds by one percent that reported in the College Board’s press release with non-respondents removed. Coincidentally, the “increased diversity” from 2002 to 2003 was also one percent. Note that the remaining percentage of non-respondents (those who are estimated to be non-white) is larger than any of the minority groups, except African Americans.  The estimate based on math score means yields a percentage of white test-takers in Breakdown 4 that more substantially exceeds that reported in the press release. Indeed it all but negates the increase in minority representation among test-takers reported in the press release (from 30% minority in 1993 to 36% in 2003).  The proportion of estimated minority non-respondents is larger than any of the groups of minority respondents except African Americans and Asian Americans. Basis for Breakdown of Racial/Ethic Groups Pie Chart Associated with Each Basis Breakdown 1. Based on breakdown reported in the College Board press release Breakdown 2 Based on data as reported in College Bound Seniors with actual non-respondents included Breakdown 3. Estimated breakdown based on SAT-V analysis Breakdown 4. Estimated breakdown based on SAT-M analysis Figure 5. Breakdowns of the Race/Ethnicity of SAT-1 Test Takers from the Class of 2003 Based on Different Estimates Note 2 These results suggest that non-respondents prevent valid use of data from College Bound Seniors to describe majority and minority test-takers with respect to: The size of the achievement gap Changes in the magnitude of the achievement gap The diversity of those taking the SAT Differences in GPA among racial/ethnic groups First-generation college attendance High school preparation Recommendations The combined results from this study suggest several further steps. First, we need to find out more about these non-respondents. Who are they? Why are they declining to indicate their race/ethnicity? Several hypotheses have surfaced.  One idea, derived from the growth in resistance to affirmative action, is that the non-respondents are white males who see no benefit in reporting their race/ethnicity. It is documented that non-respondents are majority male and both sets of estimates appearing in Table 4 suggest a dramatic increase in white non-respondents.  A second suggestion is that students from various groups are resisting reporting their race/ethnicity as a protest against the use of a category they regard as arbitrary and grounded in mistaken notions about race.  A third suggestion is based on findings of Claude Steele and others pertaining to stereotype threat. (Aronson, 2002; Steele & Aronson, 1998) The notion that reporting one’s race during a test affects test performance may be filtering its way into the middle-class, professional African-American community. Hence, increasing numbers of high-achieving minority students may not be revealing their race on test applications.  In all probability, there are multiple reasons, as might be inferred from a brief analysis of the 2003 data from my district. It was possible to identify the race/ethnicity of all students, match that to their SAT scores and then compare this racial/ethnic and test score data to that from the College Board’s College Bound Seniors 2003 report for Shaker Heights High School  The data from the College Bound Seniors report for the high school reveal a 23 percent non-response rate in 2003, which is similar to our state and the national data in terms of percentage. Our non-respondents, unlike their national and state counterparts, are majority female. Compared to those reporting their race/ethnicity, a disproportionate number of our non-respondents are White/European-American. Eighty-two percent of them are White; 18 percent are Black/African-American. By contrast, 36 percent of our high school’s SAT test takers in the class of 2003 are Black/African-American; 59 percent of them are White/European-American. The non-respondent verbal mean was higher than the overall mean, and both groups of non-respondents had higher SAT-V scores than their counterparts. The math scores, on the other hand, present a contradiction. While the White non-respondents had higher SAT-M scores, their Black/African-American counterparts had lower SAT-M scores.  Second, while descriptions of racial/ethnic differences and score changes over time have always been questionable, the sheer size of the non-respondent group makes use of SAT data for such purposes irresponsible. At the very least, test sponsors and state departments of education need to be clear in their press releases about the presence of non-respondents and their impact on one’s ability to make inferences about students taking college admissions tests in the same way they warn about other misuses of test results.  The College Board and ACT provide very clear cautions against using their data to make inferences about state rankings, for example, and these cautions seems to have taken hold. Education departments’ press releases and newspaper articles often make adjustments that seem to heed this caution. For example, they reported data only for other states with similar participation rates, or they warn readers that participation rate and mean scores are strongly related to each other. Furthermore, test companies’ press releases themselves need to provide clear information about non-respondents—the number of them, their known characteristics compared to respondents (i.e. test performance, gender).  Finally, those who have used these data in news articles, policy papers, or published research need to reconsider their use. The difference in the opportunities and achievement of children from various racial/ethnic groups is perhaps the most important issue facing education today. We need to examine and describe these differences, but we need to do it with data that we can count on to provide an accurate picture. Alternative data sources such as state data and NAEP results can do the job much more effectively. References Aronson, J. (2002) Stereotype threat: Contenting and coping with unnerving expectations. In J. Aronson (Ed.), Improving academic achievement (pp. 279-301). San Diego CA: Academic Press. ACT. (2004) How significant are changes in my school’s average ACT composite scores over time?, ACT Research information brief 98-3. Retrieved January 9, 2004 from http://www.act.org/research/briefs/98-3.html. ACT. (2004) Monitoring changes in high school ACT composite scores over time, ACT Research information brief 2000-2. Retrieved January 9, 2004 from http://www.act.org/research/briefs/2000-2.html. ACT. (2003) ACT national and state scores. Retrieved January 9, 2004 from http://www.act.org/news/data/03/index.html Airasian, P. (2001) Classroom assessment, 4th ed. Boston: McGraw Hill. Barton, P.E. (2003) Parsing the achievement gap (ETS Policy Information Center Report). Princeton, NJ: Educational Testing Service Cohen, G.L. & Steele, C.M. (2002), A barrier of mistrust: How negative stereotypes affect cross-race mentoring. In J. Aronson (Ed.), Improving academic achievement (pp. 303-327). San Diego CA: Academic Press. College Board. (2003) SAT verbal and math scores up significantly as a record-breaking number of students take the test.  Retrieved January 9, 2004 from http://www.collegeboard.com/press/article/0,3183,26858,00.html College Entrance Examination Board. (2002) Guidelines on the used of college board test scores and related data. New York: College Entrance Examination Board. Ferguson, R.F. (2001) A diagnostic analysis of black-white GPA disparities in Shaker Heights, Ohio. In D. Ravitch, (Ed.), Brookings Papers of Education Policy, 2001 (pp. 417-414). Washington, DC: Brookings Institution Press. Ferguson, R.F. (1998) Teachers’ perceptions and expectations and the black-white test score gap. In C. Jencks & M. Phillips, (Eds.), The black-white test score gap (pp. 318-374). Washington, DC: Brookings Institution Press Gould, S.J. (1981) The mismeasure of man. New York: W.W. Norton & Company. Herrnstein, R. J. & Murray, C. (1994) The bell curve: Intelligence and class structure in American life. New York: Free Press. House, E.R. (1999) Race and policy. Education Policy Analysis Archives, 7 (16). Retrieved August 11, 2000, from http://epaa.asu.edu/epaa/v7n16.html. Jencks, C & Phillips, M. (1998) The black-white test score gap: An introduction. In C. Jencks & M. Phillips, (Eds.), The black-white test score gap (pp. 1-51). Washington, DC: Brookings Institution Press Johnson, W.R. & Neal, D. (1998) Basic skills and the black-white earnings gap. In C. Jencks & M. Phillips, (Eds.), The black-white test score gap (pp. 480-497). Washington, DC: Brookings Institution Press Kane, T.J. (1998) Racial and ethnic preferences in college admissions. In C. Jencks & M. Phillips, (Eds.), The black-white test score gap (pp. 431-456). Washington, DC: Brookings Institution Press. Kober, N. (2001) It takes more than testing. A report of the Center on Education Policy. Washington, DC: Center on Education Policy. Lee, J. (2002). Racial and ethnic achievement gap trends: Reversing the progress toward equity? Educational Researcher, 31 (1), 3-12. McAdoo, H.P. (1978) Factors related to stability in upwardly mobile black families. Journal of Marriage and the Family. 40 (November) 761-776. Mercado, C. I. (2001) The learner: “Race”, “ethnicity,” and linguistic differences. In V. Richardson, (Ed.), Handbook of research on teaching (pp. 668-694). Washington, DC: American Educational Research Association. Mickelson, R.A. (2001) Subverting Swann: First- and second-generation segregation in the Charlotte-Mecklenburg schools. American Educational Research Journal, 38 (1) 215-252. Miller, G.E. (2003) Analyzing the minority gap in achievement scores: Issues for states and federal government. Educational Measurement: Issues and Practice, 22 (3), 30-36. Ogbu, J.U. (1999) Beyond language: Ebonics, proper English, and identify in a Black-American speech community. American Educational Research Journal, 36 (2), 147-184. Ogbu, J.U. (2003) Black American students in an affluence suburb, A study of academic disengagement. Mahwah, NJ: Lawrence Erlbaum Associates. Okagaki, L. & Frensch, P.A. (1998) Parenting and children’s school achievement: A multiethnic perspective. American Educational Research Journal, 35 (1), 124-144. Pollock, M. (2001) How the question we ask most about race in education of the very question we most suppress. Educational Researcher, 30 (9), 2-12. Portes, P. (1996) Ethnicity and culture in educational psychology. In Berliner & Calfee, eds. Handbook of educational psychology (pp. 331-357). New York: Simon & Schuster Macmillan. Portes, P. (1999) Social ansd psychological factors in the academic achievement of children of immigrants: A cultural history puzzle. American Educational Research Journal, 36 (3), 489-507. Powell, B. & Steelman, L.C. (1996) Bewitched, bothered, and bewildering: The use and misuse of state SAT and ACT scores. Harvard Educational Review, 66 (1), 27-59. Roderick, M. & Cambrun, E. (1999) Risk and recovery from course failure in the early years of high school. American Educational Research Journal. 36 (2), 303-343. Spring, J. (2000). American education, 9th ed. Boston: McGraw Hill. Steele, C.M. & Aronson, J. Stereotype threat and the test performance of academically successful African Americans.  In C. Jencks & M. Phillips, (Eds.), The black-white test score gap (pp. 410-427). Washington, DC: Brookings Institution Press. SYSTAT (2000). SYSTAT 10 graphics. Chicago: 2000. Valencia, R.R. & Suzuki, LA. (2001) Intelligence testing and minority students. Thousand Oaks, CA: Sage Publications. Vars, F. E. & Bowen, W.G. (1998) Scholastic Aptitude Test scores, race, and academic performance in selective colleges and universities.  In C. Jencks & M. Phillips, (Eds.) The black-white test score gap (pp. 457-479). Washington, DC: Brookings Institution Press Wainer, H. (1988) How accurately can we assess changes in minority performance on the SAT? American Psychologist, 43 (10), 774-778. Notes Percentages of non-respondents from Alaska, Hawaii and the District of Columbia were 26.6%, 20.2% and 28.5% respectively. Based on aggregated data reported by the U.S. Census Bureau for the 2000 census, whites made up 63% of 18-year olds in 2000. The second largest group was Hispanic (16.1%), followed by African Americans (14.1%). This age group was majority male (51.3%) in 2000. About the Author Dale Whittington, Ph.D. Director of Research & Evaluation Shaker Heights City School District 15600 Parkland Dr. Shaker Heights, OH 44120 216-295-4363 216-295-4340 (fax) Email: whittington_d@shaker.org Dale Whittington is Director of Research and Evaluation in the Shaker Heights (Ohio) City School District. In addition to managing the district's testing program, she conducts a variety of studies on the schools, students, and programs. She was previously Associate Professor in the Department of Education and Allied Studies at John Carroll University, where she taught courses on research methods, testing and educational psychology.

The World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu Editor: Gene V Glass, Arizona State University Production Assistant: Chris Murrell, Arizona State University General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu. EPAA Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Linda Darling-Hammond Stanford University Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kallós Umeå University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Los Angeles Michele Moses Arizona State University Gary Orfield Harvard University Anthony G. Rud Jr. Purdue University Jay Paredes Scribner University of Missouri Michael Scriven University of Auckland Lorrie A. Shepard University of Colorado, Boulder Robert E. Stake University of Illinois—UC Kevin Welner University of Colorado, Boulder Terrence G. Wiley Arizona State University John Willinsky University of British Columbia EPAA Spanish and Portuguese Language Editorial Board Associate Editors for Spanish & Portuguese Gustavo E. Fischman Arizona State University fischman@asu.edu Pablo Gentili Laboratório de Políticas Públicas Universidade do Estado do Rio de Janeiro pablo@lpp-uerj.net Founding Associate Editor for Spanish Language (1998-2003) Roberto Rodríguez Gómez Universidad Nacional Autónoma de México Adrián Acosta (México) Universidad de Guadalajara adrianacosta@compuserve.com J. Félix Angulo Rasco (Spain) Universidad de Cádiz felix.angulo@uca.es Teresa Bracho (México) Centro de Investigación y Docencia Económica-CIDE bracho dis1.cide.mx Alejandro Canales (México) Universidad Nacional Autónoma de México canalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State University casanova@asu.edu José Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of Chicago Eepstein@luc.edu Josué González (U.S.A.) Arizona State University josue@asu.edu Rollin Kent (México) Universidad Autónoma de Puebla rkent@puebla.megared.net.mx María Beatriz Luce(Brazil) Universidad Federal de Rio Grande do Sul-UFRGS lucemb@orion.ufrgs.br Javier Mendoza Rojas (México) Universidad Nacional Autónoma de México javiermr@servidor.unam.mx Marcela Mollis (Argentina) Universidad de Buenos Aires mmollis@filo.uba.ar Humberto Muñoz García (México) Universidad Nacional Autónoma de México humberto@servidor.unam.mx Angel Ignacio Pérez Gómez (Spain) Universidad de Málaga aiperez@uma.es Daniel Schugurensky(Argentina-Canadá) OISE/UT, Canada dschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil) American Institutes for Resesarch–Brazil (AIRBrasil) simon@sman.com.br Jurjo Torres Santomé (Spain) Universidad de A Coruña jurjo@udc.es Carlos Alberto Torres (U.S.A.) University of California, Los Angeles torres@gseisucla.edu