This article has been retrieved
times since February 11, 1999
Education Policy Analysis Archives
Volume 7 Number 4
February 11, 1999
ISSN 1068-2341
A peer-reviewed scholarly electronic journal
Editor: Gene V Glass, College of Education
Arizona State UniversityCopyright 1999, the EDUCATION POLICY ANALYSIS ARCHIVES.
Permission is hereby granted to copy any article
if EPAA is credited and copies are not sold.Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education.
Abstract Less Truth Than Error?
An independent study of the Massachusetts Teacher TestsWalt Haney, Clarke Fowler, Anne Wheelock,
Damian Bebell and Nicole MalecAd Hoc Committee to Test the Teacher Test
Center for the Study of Testing, Evaluation and Educational Policy
Boston College
The Massachusetts Teacher Tests (MTT), introduced last year, have never been subject to external review as required by the measurement profession's standards and many legal precedents. Neither the Massachusetts Department of Education (DOE) nor the tests' manufacturer have made public information about the exams' reliability (consistency) or validity (meaningfulness). Using data from state and academic reports from the April and July test dates, an ad hoc committee of nationally-known researchers has now been able to make a preliminary assessment of the exams. The committee focused on the Communications and Literacy exam that was required of all prospective teachers regardless of grade level or subject area. The purpose of the analysis was to determine the accuracy of the tests in assessing the reading and writing skills of the test-takers.
Scores on the Massachusetts Teacher Tests of reading and writing are highly unreliable. The tests' margin of error is close to double to triple the range found on well-developed tests. A person retaking the MTT several times could have huge fluctuations in their scores even if their skill level did not change significantly. In fact, the 9 to 17 point margin of error calculated for the tests represents more than 10 percent of the grading scale (assumed to be 0 to 100). The large margin of error means there is both a high false-pass rate and a high false-failure rate. For example, a person who received a score of 72 on the writing test could have scored an 89 or a 55 simply because of the unreliability of the test. Since adults' reading and writing skills do not change a great deal over several months, this range of scores on the same test should not be possible. While this test is being touted as an accurate assessment of a person's fitness to be a teacher, one would expect the scores to accurately reflect a test-taker's verbal ability level. In addition to the large margin of error, the MTT contain questionable content that make them poor tools for measuring test-takers' reading and writing skills. The content and lack of correlation between the reading and writing scores reduces the meaningfulness, or validity, of the tests. The validity is affected not just by the content, but by a host of factors, such as the conditions under which tests were administered and how they were scored. Interviews with a small sample of test-takers confirmed published reports concerning problems with the content and administration.
If the Commonwealth wants high standards for its teaching force, it should use assessments that meet high professional standards. The current MTT fail this criterion. Results from the April and July administrations of the MTT reveal that these new tests are so unreliable and of such poor validity that they are passing candidates who should fail and failing ones who should pass. Therefore, the ad hoc committee recommends:
- The Massachusetts Board of Education should immediately suspend the administration of the Massachusetts Teacher Tests.
- The Commonwealth should convene an independent panel to audit the tests' development, administration, and use.
- An investigation should be launched to uncover why the state contracted with this test developer even after learning of the company's poor past performance in developing tests of this type.
About the Authors
Walt Haney is a professor in the School of Education and Senior Research Associate in the Center for the Study of Testing, Evaluation and Educational Policy at Boston College. He is former editor of the journal Educational Measurement: Issues and Practice, advisor to the committee that developed the 1985 Standards for Educational and Psychological Testing and author of numerous articles concerning educational testing and evaluation. He has also served as an expert witness in numerous court cases concerning testing.Clarke Fowler is a professor in the Education Department at Salem State College. He has taught both preschool and kindergarten and currently teaches courses in early childhood education.
Anne Wheelock, an independent education policy writer and researcher, works for several national foundations and is the author of Safe To Be Smart: Building a Culture for Standards-Based Reform in the Middle Grades (1998).
Damian J. Bebell is a doctoral student at Boston College where he is employed at the Center for the Study of Testing, Evaluation and Educational Research. His research interests include educational philosophy, alternative forms of assessment, and homeschooling.
Electronic mail addresses and phone numbers for the authors are: haney@bc.edu (617-552-4199), clarke.fowler@salem.mass.edu (617-524-4704), wheelock@shore.net (802-254-2796), and bebell@bc.edu.
Center for the Study of Testing, Evaluation and Educational Policy Campion Hall, Boston College, Chestnut Hill, MA 02467
Copyright 1999 by the Education Policy Analysis Archives
The World Wide Web address for the Education Policy Analysis Archives is http://epaa.asu.edu
General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-0211. (602-965-9644). The Book Review Editor is Walter E. Shepherd: shepherd@asu.edu . The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu .
EPAA Editorial Board
Michael W. Apple
University of WisconsinGreg Camilli
Rutgers UniversityJohn Covaleskie
Northern Michigan UniversityAndrew Coulson
a_coulson@msn.comAlan Davis
University of Colorado, DenverSherman Dorn
University of South FloridaMark E. Fetler
California Commission on Teacher CredentialingRichard Garlikov
hmwkhelp@scott.netThomas F. Green
Syracuse UniversityAlison I. Griffith
York UniversityArlen Gullickson
Western Michigan UniversityErnest R. House
University of ColoradoAimee Howley
Ohio UniversityCraig B. Howley
Appalachia Educational LaboratoryWilliam Hunter
University of CalgaryRichard M. Jaeger
University of North Carolina--GreensboroDaniel Kallós
Umeå UniversityBenjamin Levin
University of ManitobaThomas Mauhs-Pugh
Green Mountain CollegeDewayne Matthews
Western Interstate Commission for Higher EducationWilliam McInerney
Purdue UniversityMary McKeown-Moak
MGT of America (Austin, TX)Les McLean
University of TorontoSusan Bobbitt Nolen
University of WashingtonAnne L. Pemberton
apembert@pen.k12.va.usHugh G. Petrie
SUNY BuffaloRichard C. Richardson
Arizona State UniversityAnthony G. Rud Jr.
Purdue UniversityDennis Sayers
Ann Leavenworth Center
for Accelerated LearningJay D. Scribner
University of Texas at AustinMichael Scriven
scriven@aol.comRobert E. Stake
University of Illinois--UCRobert Stonehill
U.S. Department of EducationRobert T. Stout
Arizona State UniversityDavid D. Williams
Brigham Young University
EPAA Spanish Language Editorial Board
Associate Editor for Spanish Language
roberto@servidor.unam.mx
Roberto Rodríguez Gómez
Universidad Nacional Autónoma de MéxicoAdrián Acosta (México)
Universidad de Guadalajara
adrianacosta@compuserve.comJ. Félix Angulo Rasco (Spain)
Universidad de Cádiz
felix.angulo@uca.esTeresa Bracho (México)
Centro de Investigación y Docencia Económica-CIDE
bracho dis1.cide.mxAlejandro Canales (México)
Universidad Nacional Autónoma de México
canalesa@servidor.unam.mxUrsula Casanova (U.S.A.)
Arizona State University
casanova@asu.eduJosé Contreras Domingo
Universitat de Barcelona
Jose.Contreras@doe.d5.ub.esErwin Epstein (U.S.A.)
Loyola University of Chicago
Eepstein@luc.eduJosué González (U.S.A.)
Arizona State University
josue@asu.eduRollin Kent (México)
Departamento de Investigación Educativa-DIE/CINVESTAV
rkent@gemtel.com.mx kentr@data.net.mxMaría Beatriz Luce (Brazil)
Universidad Federal de Rio Grande do Sul-UFRGS
lucemb@orion.ufrgs.brJavier Mendoza Rojas (México)
Universidad Nacional Autónoma de México
javiermr@servidor.unam.mxMarcela Mollis (Argentina)
Universidad de Buenos Aires
mmollis@filo.uba.arHumberto Muñoz García (México)
Universidad Nacional Autónoma de México
humberto@servidor.unam.mxAngel Ignacio Pérez Gómez (Spain)
Universidad de Málaga
aiperez@uma.esDaniel Schugurensky (Argentina-Canadá)
OISE/UT, Canada
dschugurensky@oise.utoronto.caSimon Schwartzman (Brazil)
Fundação Instituto Brasileiro e Geografia e Estatística
simon@openlink.com.brJurjo Torres Santomé (Spain)
Universidad de A Coruña
jurjo@udc.esCarlos Alberto Torres (U.S.A.)
University of California, Los Angeles
torres@gseisucla.edu
archives | abstracts | editors | board | submit | comment | subscribe | search