Predicting Variations in Math Performancein Four Countries Using TIMSS

Abstract

Although international comparisons of average student performance are a staple of U.S. educational debate, little attention has been paid to cross-national differences in the variability of performance. It is often assumed that the performance of U.S. students is unusually variable or that the distribution of U.S. scores is left-skewed – that is, that it has an unusually long ‘tail' of low-scoring students – but data from international studies are rarely brought to bear on these questions. This study used data from the Third International Mathematics and Science Study (TIMSS) to compare the variability of performance in the U.S., Australia, France, Germany, Hong Kong, Korea, and Japan; investigate how this performance variation is distributed within and between classrooms; and explore how well background variables predict performance at both levels. TIMSS shows that the U.S. is not anomalous in terms of the amount, distribution, or prediction of performance variation. Nonetheless, some striking differences appear between countries that are potentially important for both research and policy. In the U.S., Germany, Hong Kong, and Australia, between 42 and 47 percent of score variance was between classrooms. At the other extreme, Japan and Korea both had less than 10 percent of score variance between classrooms. Two-level models (student and classroom) were used to explore the prediction of performance by social background variables in four of these countries (the U.S., Hong Kong, France, and Korea). The final models included only a few variables; TIMSS lacked some important background variables, such as income, and other variables were dropped either because of problems revealed by exploratory data analysis or because of a lack of significance in the models. In all four countries, these sparse models predicted most of the between-classroom score variance (from 59 to 94 percent) but very little of the within-classroom variance. Korea was the only country in which the models predicted more than 5 percent of the within-classroom variance in scores. In the U.S. and Hong Kong, the models predicted about one-third of the total score variance, and almost all of this prediction was attributable to between-classroom differences in background variables. In Korea, only 19 percent of total score variance was predicted by the model, and most of this most of this was attributable to within-classroom variables. Thus, in some instances, countries differ more in terms of the structure and prediction of performance variance than in the simple amount of variance. TIMSS does not provide a clear explanation of these differences, but this paper suggests hypotheses that warrant further investigation.