This article has been retrieved
times since August 21, 2000
This article has been retrieved
| ||
Volume 8 Number 43 |
August 21, 2000 |
ISSN 1068-2341 |
|
Editor: Gene V Glass, College of Education Arizona State University
Copyright 2000, the
EDUCATION POLICY ANALYSIS ARCHIVES. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education. |
Consistency of Findings Across International Surveys
|
|
Abstract The investigation reported in here was prompted by discrepancies between the performance of Irish students on two international tests of science achievement: the Second International Assessment of Educational Progress (IAEP2) administered in 1991 and the Third International Mathematics and Science Study (TIMSS) administered in 1995. While average science achievement for Irish 13-year-olds was reported to be at the low end of the distribution representing the 20 participating countries in IAEP2, it was around the middle of the distribution representing the 40 or so countries that participated in TIMSS at grades 7 and 8. An examination of the effect sizes associated with mean differences in performance on IAEP2 and TIMSS indicated that the largest differences are associated with the performance of students in France, Ireland and Switzerland. Five hypotheses are proposed to account for the differences. |
IntroductionInternational comparative studies of student achievement have become part of the educational landscape over the past four decades. In these studies, a number of countries (usually represented by research organizations) agree on an instrument to assess achievement in a curriculum area, the instrument is administered to a representative sample of students at a particular age or grade level in each country, and comparative analyses of the data obtained are carried out. The most frequently assessed areas have been reading, mathematics, and science at ages 9 or 10 and 13 or 14. The number of participating countries has grown from 12 in a pilot project conducted between 1959 and 1961 to over 40 for a survey of mathematics and science achievements in 1995 (see Goldstein, 1996; Husén & Postlethwaite, 1996; Kellaghan, 1996).The potential of international studies to contribute to policy formation was made clear from the earliest studies (Husén, 1967; Lambin, 1995). Over the years, a range of purposes to which information derived from such studies might be put has been suggested. These include the pursuit of equity goals, setting priorities, assessing the effectiveness and efficiency of the educational enterprise and the appropriateness of curricula, evaluating instructional methods and the organization of the school systems, and providing a mechanism for accountability (Kellaghan & Grisay, 1995; Plomp, 1992). While we have relatively little information on the extent to which the findings of international studies have in fact been utilized, there is no doubt that they attract considerable media and public attention. A variety of factors can affect the extent to which data obtained in an international study accurately reflects what students have learned in the participating countries, something that is necessary if valid comparisons between countries are to be made (see Brown, 1996, 1998; Goldstein, 1996; Kellaghan, 1996; Kellaghan & Grisay, 1995; Murphy, 1996; Nuttall, 1994). One relates to the adequacy of a uniformly administered assessment procedure to measure the outcomes of a variety of curricula. Since curricula differ from country to country, an assessment instrument will not reflect the curricula of all countries participating in an international study to the same degree. The second factor relates to the extent that the populations and samples of pupils for whom data are obtained can be regarded as equivalent. Defined target populations may not be comparable across countries since exclusion practices may differ (e.g., relating to students with handicapping conditions/learning problems or when the language of the assessment instrument differs from the language of the school). Differences in participation rates of selected samples (due to lack of co-operation from schools, student absenteeism) will make matters worse. Many commentators have considered how these problems impact on comparisons based on a single study. Additional problems arise when the findings of two different surveys are being compared. In the case of IAEP2 and TIMSS, instruments used to measure achievement differed in form and content sampled, age-based versus grade-based populations definitions were used, and different methods of data manipulation were utilized. The investigation reported here was prompted by discrepancies between the performance of Irish students on tests of science in the Second International Assessment of Educational Progress in Mathematics and Science (IAEP2) (Lapointe, Askew & Mead, 1992) in 1991 and, four years later in the Third International Mathematics and Science Study (TIMSS) (Beaton, Mullis, Martin, Gonzalez, Kelly & Smith, 1996a; Beaton, Martin, Mullis, Gonzalez, Smith, & Kelly, 1996b). Initially, the intention was to focus on the Irish problem but, as the investigation proceeded, it became clear that discrepancies in performance between the two surveys were not confined to Irish students. In this article, we first present brief descriptions of IAEP2 and TIMSS. We then select 12 countries that participated in both surveys for further analyses: Canada, England, France, Hungary, Ireland, Korea, Portugal, Scotland, Slovenia, Spain, Switzerland, and the United States. Our approach to assessing the consistency of countries' performances is based on an examination of the performance of each country relative to the performance of other countries in both surveys. If results are stable, differences in performance between countries should not vary very much from one survey to the next. To the extent that they do, findings may be regarded as unstable. Change in effect sizes between pairs of means on the two assessments were calculated to obtain an estimate of the magnitude of differences between performance on the two occasions. |
IAEP2 and TIMMSIn IAEP2, representative samples of 9 and 13-year-olds in 20 countries were assessed in mathematics and science in 1991 (Lapointe, Askew & Mead, 1992). In TIMSS, the mathematics and science achievements of students in grades 3, 4, 7, 8, and in the final grade of secondary education were assessed in 1995 (Beaton et al., 1996). Data are reported in our article for 13- year-olds in IAEP2 and for grades 7 and 8 students in TIMSS. However, the main focus is on grade 7 performance, since in all countries that had participated in both assessments, except Scotland, more 13 year-olds were in grade 7 than in grade 8 (Beaton et al, 1996a, p. A12).The IAEP2 tests for 13-year olds were contained in two separate booklets, each of which had to be completed by students in four 15-minute segments (one hour testing time in all). The mathematics booklet contained 76 items and covered four content areas: Measurement, Geometry, Data Analysis/Statistics/Probability, and Algebra/Functions. The science test consisted of 72 items and covered four content areas: Life Sciences, Physical Sciences, Earth/Space Sciences, and the Nature of Science. Students completed either a mathematics or science test and were administered all items on the test. Unlike IAEP2, the TIMSS test booklets contained both mathematics and science items. At grades 7 and 8, the mathematics test comprised 151 items and the science test 135 items. The TIMSS mathematics items covered six content areas: Fractions/Number Sense, Geometry, Algebra, Data Representations/Analysis/Probability, Measurement, and Proportionality. The science content areas were: Earth Science, Life Science, Physics, Chemistry, and Environmental Issues/Nature of Science. Items were rotated across eight test booklets and student performance was matrix-sampled using a modified Balanced-Incomplete-Block (BIB) spiraling design (Martin & Kelly, 1997). One and a half hours were allocated for the completion of each booklet. In both studies, performance on both tests was reported in the form of an average percentage correct score. In the case of TIMSS, an average scale score for each country was also reported. While scale scores were calculated for the IAEP2 study, they were not included in the published reports. The Consistency of IAEP2 and TIMSS Science ResultsIn 1991, the average science performance of Irish 13-year-olds is significantly below the average performance of students in all but two of the 'common' countries (Portugal and the US) and also significantly below the international mean (Lapointe, Askew, & Mead, 1992). However, in 1995, the average performance of Irish students on the TIMSS test at grades 7 & 8 compares much more favorably with the 'common' countries and with the overall TIMSS means (Beaton et al, 1996b). This change of fortune is clearly evident in Table 1, in which countries are listed from highest achieving to lowest achieving, and are categorized according to whether their means were statistically significantly above, below, or did not differ from, the Irish mean. |
Table 1
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| M | SE | M | SE | M | SE | |||
| Overallc | 66.9 | 49.8 | (0.1) | 55.5 | (0.1) | |||
| Kor | 77.5 | (0.5) | Kor | 61.4 | (0.4) | Kor | 65.5 | (0.3) |
| Swi | 73.7 | (0.9) | Slo | 57.2 | (0.5) | Slo | 61.7 | (0.5) |
| Hun | 73.4 | (0.5) | Hun | 55.5 | (0.6) | Eng | 61.3 | (0.6) |
| Slo | 70.3 | (0.5) | Eng | 55.6 | (0.6) | Hun | 60.7 | (0.6) |
| Can | 68.8 | (0.4) | US | 54.0 | (1.1) | Can | 58.7 | (0.5) |
| Eng | 68.7 | (1.2) | Can | 54.0 | (0.5) | Ire | 58.4 | (0.9) |
| Fra | 68.6 | (0.6) | Ire | 52.0 | (0.7) | US | 58.3 | (1.0) |
| Sco | 67.9 | (0.6) | Swi | 50.1 | (0.4) | Swi | 56.3 | (0.5) |
| Spa | 67.5 | (0.6) | Spa | 49.3 | (0.4) | Spa | 55.6 | (0.4) |
| US | 67.0 | (1.0) | Sco | 48.2 | (0.8) | Sco | 55.3 | (1.0) |
| Ire | 63.3 | (0.6) | Fra | 46.1 | (0.6) | Fra | 53.7 | (0.6) |
| Por | 62.6 | (0.8) | Por | 41.3 | (0.5) | Por | 49.9 | (0.6) |
| M | SE | M | SE | M | SE | |||
| Overall | 58.3 | 49.3 | (0.1) | 55.1 | (0.1) | |||
| Kor | 73.4 | (0.6) | Kor | 67.0 | (0.6) | Kor | 71.7 | (0.5) |
| Swi | 70.8 | (1.3) | Hun | 53.8 | (0.8) | Swi | 62.0 | (0.6) |
| Hun | 68.4 | (0.8) | Swi | 53.1 | (0.5) | Hun | 61.5 | (0.7) |
| Fra | 64.2 | (0.8) | Ire | 53.3 | (1.0) | Fra | 61.3 | (0.8) |
| Can | 62.0 | (0.6) | Slo | 52.5 | (0.7) | Slo | 61.2 | (0.7) |
| Eng | 60.6 | (2.2) | Can | 51.6 | (0.5) | Ire | 58.7 | (1.2) |
| Sco | 60.6 | (0.9) | Fra | 51.0 | (0.8) | Can | 58.7 | (0.5) |
| Ire | 60.5 | (0.9) | US | 47.7 | (1.2) | Eng | 53.1 | (0.7) |
| Slo | 57.1 | (0.8) | Eng | 47.2 | (0.9) | US | 53.0 | (1.1) |
| Spa | 55.4 | (0.8) | Sco | 44.3 | (0.9) | Sco | 51.6 | (1.3) |
| US | 55.3 | (1.0) | Spa | 42.4 | (0.6) | Spa | 51.0 | (0.5) |
| Por | 48.3 | (0.8) | Por | 36.6 | (0.6) | Por | 42.9 | (0.7) |
Compared to
their performance on the IAEP2 science assessment, four
countries maintain their superiority over Ireland on the
TIMSS assessment at grade 7 (Korea, Slovenia, Hungary,
England). Two, having performed at a superior level on
IAEP2, achieve at levels comparable to Ireland in TIMSS
(Canada, Switzerland), while three that were superior on
IAEP2 record a significantly poorer performance on TIMSS
(France, Scotland, Spain). Comparisons between IAEP2
performance and performance at grade 8 on TIMSS reveal a
somewhat similar pattern in which only two countries (Korea
and Slovenia) maintain their superior position.
It is
apparent that the relative performances of countries other
than Ireland also change between IAEP2 and TIMSS (e.g.,
France and Switzerland). It could be argued that the same
phenomenon occurs in mathematics (compare, for example,
English and Scottish performances in the two surveys).
However, changes in position are less frequent in
mathematics, a finding that is reflected in the magnitude of
the correlations between scores in the two assessments
(Table 2).
Table 2
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| TIMSS Grade 7 Mean Scale Score |
TIMSS Grade 7 Mean Percent Correct |
|
| IAEP2 Mean Scale Score |
||
| IAEP2 Mean Percent Correct |
||
| IAEP2 Mean Scale Score |
||
| IAEP2 Mean Percent Correct |
||
In considering the consistency of scores from
one assessment to another, data on statistical significance
from the published reports could have been used (as they
were in Table 1). However, since our interest is in the
extent to which the size of differences between pairs of
country means changed across the assessments, we chose to
use an effect-size index.