Introduction

As a step toward answering this question, I examine two common criticisms of the methodology that U.S. News uses to rank colleges and graduate schools. These are: (1) constant changes to the formula make it impossible to interpret yearly shifts in a school's rank in terms of change in its relative academic quality (Levin, 1999; Pellegrini, 1999), and (2) the score used to assign schools to ranks is overly precise, creating a vertical column where a group might more properly exist (Machung, 1998; Smetanka, 1998). The first section of this article gives a brief introduction to the U.S. News rankings as well as the questions addressed by this study. The next section outlines the methodology used to answer these questions and the results of the analyses. The final section presents conclusions and recommendations.

Before proceeding, a caveat is in order. While many have questioned the overall concept of academic quality rankings as well as the validity of the different indicators and weights used, I suspend judgment on these issues to focus on the extent to which methodological problems may impact the interpretation of the U.S. News rankings.

Background on the U.S. News Rankings

The most recent rankings still use this basic approach. At the undergraduate level, schools are categorized by mission and region (e.g., national universities, national liberal arts colleges, regional universities, and regional liberal arts colleges). Up to sixteen pieces of information are collected on schools in each category, including academic reputation; freshmen retention and graduation rates; average test scores for entering students; per-student spending; and alumni-giving rate. These indicators are standardized, weighted, and summed to produce an overall score on which to rank schools in each category against their peers.

At the graduate level, schools are categorized by type—business, education, engineering, law, and medicine/primary-care. Depending on the type of school, data on up to fourteen indicators—including test scores, research expenditures, graduate employment rates, and reputation—are collected. Similar to the undergraduate rankings, the indicators are standardized, weighted, and summed to produce an overall score on which to rank schools in each category against their peers. Detailed information on the indicators and methodology that U.S. News uses to rank colleges and graduate schools is found in Appendix A. (Note 2)

Criticisms of the U.S. News Rankings

Could there not, though, at least be a move toward greater honesty with, and service to, your readers by moving away from the false precision? Could you not do away with rank ordering and overall scores, thus admitting that the method is not nearly that precise and that the difference between #1 and #2 - indeed, between #1 and #10 - may be statistically insignificant? Could you not, instead of tinkering to "perfect" the weightings and formulas, question the basic premise? Could you not admit that quality may not be truly quantifiable, and that some of the data you use are not even truly available (e.g., many high schools do not report whether their graduates are in the top 10% of their class)? Parents are confused and looking for guidance on the best choice for their particular child and the best investment of their hard-earned money. Your demonstrated record gives me hope that you can begin to lead the way away from football-ranking mentality and toward helping to inform, rather than mislead, your readers. (Note 3)

Others have voiced these methodological concerns. In particular, critics have noted that yearly formula changes make it almost impossible to interpret shifts in a school's rank in terms of change in its relative academic quality: a college that is ranked 4^th one year and 7th the next may have had no change in its performance relative to other schools, yet still have moved because of changes in the ranking methodology (Levin, 1999; Machung, 1998; Pellegrini, 1999). U.S. News' response to this issue has been that they prefer to make incremental changes every year to produce the "best possible rankings" than to use the same indicators every year to facilitate precise year-to-year comparisons.

Critics have also pointed out that the use of overall scores to rank schools magnifies small—and often insignificant—differences among schools, and that small changes by the school or the magazine can move a college half a dozen places up or down the ranking list (Crenshaw, 1999). U.S. News acknowledged this issue in 1998 when it began rounding overall scores to the nearest whole number in recognition, the editors noted, of the fact that small differences after the decimal point may reflect non-significant differences between schools (Thompson and Morse, 1998). Subsequently, the number of schools tied for overall score (and thus rank) increased dramatically.

While much criticism and debate has focused on the methodology used to produce the rankings, the majority of research has focused on the extent to which the rankings are used by students and parents (e.g., Art and Science Group, 1995; McDonough, Antonio, Walpole, and Perez, 1998) or their effect on institutions (e.g., Monks and Ehrenberg, 1999). The research presented here addresses the two methodological concerns outlined above. In particular, this study answers the following questions:

1. What is the extent of change in U.S. News' ranking formulas across years and what are the implications for interpreting shifts in a school's rank over time?
2. How precise is the overall score that U.S. News uses to rank schools and what are the implications for assigning schools to discrete ranks?

Methods and Results

Tracking Changes in Ranking Formulas across Years

Changes in weights, methodology, and the addition or removal of indicators were generally easy to track, although it was not possible to fully track changes in weights at the undergraduate level as this information was not included until the 1998 edition of the guidebook. Changes in indicator definition were harder to identify as the wording for a definition could differ from one year to the next, while the underlying meaning might not. The following rule was used to identify an indicator definition change:

1. The new wording must contain additional detail such as a date, money amount, percent, or other precise information not previously stated or implied.
2. If the new wording does not include such detail, it should be recognized as changed by U.S. News in the guidebook text.

Table 1 summarizes changes in the indicators used for each ranking from 1995 to 2000. The number of changes for each ranking, by type and overall, is shown in columns two through eight. The national university and national liberal arts college changes are shown in one column as they use the same formula. The final column in Table 1 reflects the total number of changes across all seven rankings (i.e., business, education, engineering, law, medical, national university/liberal arts, and primary care), again broken down by type.

Most changes were weight or definition/methodology changes, comprising 85 percent of all changes occurring over the six editions. Very few indicators were added to or removed from the ranking formulas, suggesting that U.S. News generally retained the same set of indicators for each ranking, but consistently refined and redefined these indicators over the years. (Of course, this redefining process can also change an indicator substantially).

The rate of change varied widely across rankings. While most rankings averaged between 6 and 8 formula changes over the six editions, the law rankings experienced 14 and the medical rankings only 4 changes over the same period. Several reasons account for the larger number of changes in the law ranking's indicators, including U.S. News' responses to the complaints of law schools (who tend to complain more than other schools) and the release of new types of quality-related information by the American Bar Association.

While a ranking (e.g., the law rankings) may have experienced a large number of changes relative to other rankings, these changes may be concentrated in a small group of indicators that are constantly being refined. Different rankings of schools also use different numbers of indicators to compute their overall score, and thus two rankings that experience the same types and number of changes may differ in the number of indicators left unchanged overall. Figure 1 shows the proportion of unchanged indicators for each ranking between 1995 and 2000 inclusive.

Figure 1. Proportion of Indicators Remaining Unchanged in Each US News Ranking, 1995-2000.

The undergraduate rankings (both national university and national liberal arts college) have the largest proportion (.73 approximately) of unchanged indicators. In contrast, only about one third of the law school indicators remained unchanged. For most rankings, about half to two thirds of the indicators remained unchanged over the six editions. This suggests that while it may not be always possible to interpret changes in a school's overall rank across years, it is possible to track performance on individual indicators that have remained unchanged across the years. Most of the unchanged indicators are related to selectivity (e.g., test scores and the proportion of applicants accepted into the program) and institutional resources (e.g., student-faculty ratios).

In Table 2, an X indicates when it is possible to make cross-year comparisons for a ranking. The criteria used to make this determination include the four types of indicator changes discussed above as well as more general formula changes. The latter occurred twice over the six editions examined here: In 1998 when overall scores were rounded to the nearest whole number, and in 1999 when a school's performance on each indicator was standardized before obtaining the overall rank score. While it was not possible to make cross-year comparisons for most rankings over the six years, the last column in Table 2 suggests that the ranking formulas may be stabilizing. Between 1999 and 2000, there were no changes in the formulas used to rank schools of education, engineering, law, and medicine, suggesting that change in a school's rank between 1999 and 2000 could be interpreted in terms of change in its relative academic quality.

It is important to remember that even when a formula appears to remain stable across years, there can still be difficulties with cross-year interpretation of ranks. This is due to problems with the accuracy of the information obtained and critics have pointed out several errors that have arisen due to mistakes (both accidental and deliberate) in reporting by institutions, and due to the differing ways in which schools compute figures for certain indicators (Machung, 1998, Smetanka, 1998, Stecklow, 1995, Wright, 1990-91). U.S. News has tried to reduce the error introduced by these practices by cross-checking data sent in by schools with data collected by debt-rating agencies, investors and national organizations such as the National Collegiate Athletic Association, and tightening up their survey questions, but issues still remain.

The final stage of the comparability analysis examined the extent to which the amount of change in a ranking formula is related to the amount of movement in schools' ranks for that ranking across years. Table 3 shows the correlation (r) between the 1995 and 2000 ranks for the top-fifty schools in each ranking in 1995.

There is no definite relationship between the amount of change in the indicators for a ranking and the correlation between the 1995 and 2000 ranks for the top-fifty ranked schools in 1995. For example, while law schools experienced the most change in their indicators over the six editions of U.S. News, there was not much difference (r = .92) in the rank ordering of the top-fifty law schools in 1995 and their ordering in 2000. While varying amounts of change was experienced in the indicators used for the other rankings, they still show a high degree of similarity (with r's between .88 and .95) in the rank ordering of their top 50 schools in 1995 and 2000. The main exceptions to this are the education (r = .72) and primary-care (r = .08) rankings. The low correlation between the primary-care rankings in 1995 and 2000 can be explained by changes in the population of schools that U.S. News included in these rankings during this time period. In contrast, the low (relative to the other rankings) correlation between the 1995 and 2000 ranks of the top-fifty schools of education in 1995 is linked to the fact that 16 of the top 50 schools in 1995 had experienced large changes in rank–of ten or more–by the 2000 edition. Table 4 shows the 16 schools of education. The first six schools all experienced a decline in rank, ranging from a drop of 10 places for the University of Southern California and the University of Iowa to a drop of 22 places for Syracuse University. The remaining schools all improved their rank since 1995. Improvement ranged from an increase of 10 places for the Rutgers University to a jump of 30 places for Arizona State University.

^aThis table does not include schools that were not ranked in 1995 but appeared in the top 50 in the 2000 edition.

Cross-year data for the top-fifty schools in 1995 in other rankings were also examined to assess the extent to which similar movements in rank occurred (only data for the top 25 schools of medicine/primary-care and the top 40 national liberal arts colleges were available). Only nine business schools, one engineering school, eight law schools, no medical or primary-care schools, three national liberal arts colleges and two national universities differed by ten or more places in their 1995 and 2000 ranks.

It is not clear why there was more movement among schools of education compared to other types of schools. If changes in indicators (i.e., weight, definition, or other changes) are not responsible, movement could be due to changes in schools' performance on the indicators or errors or inconsistencies in the information reported by schools. Unfortunately, it is difficult to identify the real reasons for these movement patterns among schools of education over time, as well as why these differ from other rankings, as U.S. News did not print much information on schools' performance on the individual indicators until 1999.

Estimating Error or Uncertainty around the Overall Score

There is no universally agreed-upon set of information for creating academic quality rankings. Thus, various ranking efforts use indicators that differ in whole or in part from those used by others even when attempting to rank the same schools. It is not difficult to imagine that slight changes in the set of indicators used–such as the addition or removal of a single indicator–may move a school up or down a ranking, depending on how it performs on the indicator relative to other schools. To gauge the effect of slight changes in the set of indicators on the stability of the overall score and subsequent ranking for a school, a technique called jackknifing (Efron and Tibshirani, 1993) was applied to the data for the top-50 schools in each of the 2000 business, education, law, national liberal arts college, and national university rankings. (Note 4)

First, a baseline regression model was created for each of the rankings, with schools' overall scores as the dependent or outcome variable and the indicators used for each ranking as the independent or predictor variables. The overall fit of the model to the data was assessed in terms of the adjusted R Squared. Values of .9 and above were considered a good fit, meaning that the overall score predicted by the model for a school was highly correlated with the score produced by U.S. News' ranking formula, and that the regression model was an effective substitute for the weights-and-sum formula used by U.S. News. All models met this criterion, with adjusted Rs Squared varying between .99 for the national liberal arts college and national university models, .98 for the business school and law school models, and .95 for the education school model. (Note 5)

An approximation to a standard error for each school's overall score was obtained using the following formula (Efron and Tibshirani, 1993): (Note 6)

The removal of one indicator at a time for the jackknife regression models did not seem to affect substantially the overall adjusted R Squared in most instances. For example, for each of the 9 models estimated using the law school data, the adjusted R Squared never varied by more than .01 from the adjusted R Squared for the overall model (i.e., .98), suggesting that the indicators are contributing fairly similar information to the estimation of the overall score. As a result, the jackknife standard errors are quite small, varying, in the case of law schools, from a low of .74 for the University of Michigan, Ann Arbor to a high of 3.06 for Harvard University. A similar range of standard error values was obtained for all rankings except for schools of education. The regression model for schools of education was not as robust to changes in indicators and the adjusted R Squared dropped considerably (by .13) when one indicator in particular—Research Expenditure—was removed. The resultant jackknife standard errors for schools of education are therefore quite large, varying from a low of 1.78 for Stanford University to a high of 11.98 for the University of Southern California.

Differences in the standard errors for individual schools are due to differences in how the removal of different indicators from the equation affects the prediction of their overall score. For schools that have large standard errors, the removal of certain indicators makes it much harder to predict the overall score they received from U.S. News. For school with smaller standard errors, the removal of indicators does not appreciably reduce the precision of estimation of their overall score. This suggests that schools are differentially affected by the presence or absence of certain indicators in terms of their overall score and subsequent rank.

This error estimate was then used in a t-test to assess the extent to which one school's overall score was significantly different from that of another. The t-test formula employed was: (Note 7)

The results of these comparisons are summarized in Tables 5 through 9 which are in the form of Excell spreadsheets. In each table, schools are ordered by their overall ranking score across the heading and down the rows. Read across the row for a school in order to compare its performance with the schools listed in the heading of the chart. The symbols indicate whether the overall score of the school in the row is significantly lower than that of the comparison school in the heading (arrow pointing down), significantly higher than that of the comparison school (arrow pointing up), or if there is no statistically significant difference between the two schools (circle). The blank diagonal represents where a school is compared against itself.

If there were no error around the overall scores for schools, Tables 5 through 9 would only consist of arrows pointing up and down, except for instances where two schools have the same overall score and are tied for rank. This is not the case. For example, in the business school rankings comparison table (Table 5) Harvard is listed first in the row and heading as it has the highest overall score among business schools. However, reading across the row, it appears that Harvard's overall score of 100 is not significantly different from that of nine other schools that are ranked beneath it. These include Stanford, which is tied for first rank with Harvard with an overall score of 100, and University of California, Berkeley, ranked tenth with a score of 90. Only schools ranked below tenth have scores that are significantly lower than Harvard's.

Tables 5 - 9
Statistical Significance of Comparisons of Overall Scores
in Five Areas (Data in the Form of Excell Spreadsheets)

Table 5: Business Table 6: Education Table 7: Law Table 8: Liberal Arts Table 9: National Universities

In general, when the overall score for a school is compared to that of every other school in its ranking (top-fifty schools only), three groups emerge: schools that score significantly higher, schools that score significantly lower, and schools with scores that are not significantly different. This pattern is consistent across all the comparison tables. For example, among the business schools in Table 5, three distinct groupings emerge. The first group comprises 10 schools at the top of the rankings, extending from first-ranked Harvard to tenth-ranked University of California, Berkeley. These schools have scores that are not significantly different from each other but that are significantly higher than all other schools' scores. The second grouping extends from eleventh-ranked Dartmouth, University of California Los Angeles, and the University of Virginia to nineteenth-ranked Carnegie Mellon. These schools have scores that are not significantly different from each other but that are significantly lower than the top-ranked schools in the first group and significantly higher than the lower-ranked schools in the third grouping. The third group is the largest. It comprises 31 schools, extending from twentieth-ranked Indiana University to forty-eighth-ranked University of Georgia, University of Illinois-Urbana Champagne, and the University of Notre Dame. These schools all have scores that are not significantly different from each other but that are significantly lower than the scores of schools in the first two groups.

This three-groupings pattern is evident for all rankings except schools of education. There are only two groupings evident in Table 6. The first group comprises the top-three-ranked schools of education—Harvard University, Stanford University, and Teacher's College/Columbia University. These schools have scores that are not significantly different from each other but that are significantly higher than the scores for almost all other schools in the top fifty. The second group of schools extends from fourth-ranked University of California-Berkeley to the four schools tied for fiftieth rank. These schools all have scores that are not significantly different from each other but that are significantly lower than the scores of most schools in the top group. This two-grouping effect occurs because schools of education are more sensitive to changes in the indicators used than other types of schools. This results in larger standards errors around their overall score and fewer significant differences between the scores of neighboring schools.

Conclusions and Recommendations

The results of these analyses show that, given the number and annual nature of changes to each ranking formula, it is generally not possible to interpret year-to-year shifts in a school's rank in terms of change in its relative academic quality. Depending on the ranking, it is possible to make cross-year comparisons of a school's relative performance on between a third to three-quarters of the individual indicators used. While not experiencing much change to their ranking formula over time, schools of education have experienced markedly more movement in their ranks than other schools. It is not evident why this has occurred or what it says about the U.S. News rankings as a measure of the relative quality of these schools. The overall rate of change in the ranking formulas appears to be slowing and it was possible to make cross-year comparisons of schools' ranks for almost all rankings between 1999 and 2000.

The results of the error analyses call into question the use of overall scores to assign schools to individual ranks. The analyses show that when interpreting scores for school with the aid of their standard errors, precision blurs and schools start to group in bands rather than discrete ranks. The results confirm the critics' sense of unease at the precision of a single score, particularly in the case of the education rankings.

At least five recommendations can be made for improving the interpretability and usefulness of the U.S. News rankings.

1. First, U.S. News needs to stabilize their ranking methodology. This is particularly important since the rankings are annual in nature and imply some kind of comparability. A related issue to consider is whether the rankings need to be annual in nature. While there is an obvious commercial value to annual rankings, particularly one that keeps changing the winners, it is doubtful whether there is an educational or consumer value.

2. Second, U.S. News needs to recognize the uncertainty around schools' overall scores. The results of this analysis suggest that it would be more accurate to group schools in bands than to assign them discrete ranks. This approach would avoid the misleading effect that small changes in a school's rank from year to year produces in terms of the public perception's of its academic quality.

3. Third, the schools of education rankings need to be reassessed since they do not seem to "hold together." Better comparisons might emerge if they were divided into two more conceptually coherent groups (e.g., those that are primarily research oriented and those that are primarily teacher-training oriented.) U.S. News already does this for schools of medicine—i.e., there is an overall ranking of medical schools as well as a ranking of schools that focus on the training of primary-care physicians.

4. Fourth, in order to be accountable to consumers, U.S. News needs to make available all data used to create the rankings. Currently, US News only publishes information for the top-ranked schools and less or no information on lower-ranked schools. While space constraints may make it difficult to publish this information in the magazine, no such restrictions apply on the US News website.

5. A final general recommendation is that U.S. News should adopt a model similar to that used by Consumer Reports for reporting its quality ratings. Consumer Reports rates products, but does not allow the product manufacturers to use these ratings in their advertising. Similarly, U.S. News should not allow schools to use their ratings in their promotional materials or other advertising. This approach might relieve some of the tension and debate that currently surrounds the rankings and make their annual arrival on newsstands a less stressful event for the higher education community.

Notes

General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu .