Interviews with MTT-Takers: Vignettes and Summary

After graduation from xx College--one of the best schools in the area, ALL of my daughter's friends failed at least one section of the MA test. Is something wrong with this picture? After a phone chain among many parents, we all agree there is a problem with the MA test, as these students did all pass required testing for other states. [Underline in original; name of College deleted to protect confidentiality.]

My scores on the Praxis series earned me a license to teach Language Arts and Social studies to grades six through nine in North Carolina. Unfortunately, this was not enough to earn a reprieve from the Massachusetts test. This is just one of the aspects of the test with which I take issue.

One major problem with the Massachusetts Teacher Tests is that there is no preparation offered. When I called to request information on the test, I received a packet of test objectives for each of the tests I was taking. This information was practically useless, as I still had no clue as to the format of the test. [Letter to the Ad Hoc Committee, dated August 20, 1998.]

        Interview sample

        Although this was a small, self-selected sample, those who agreed to an interview represented a wide range of experiences. Of the fifteen, seven (47%) passed all three parts of the test (reading, writing, and subject area) on their first try, approximating the passing rate for the state overall. Two additional candidates (13 %) passed both literacy sections, but failed their subject area tests. Five (33%) passed one portion of the literacy section only, with two of them also passing their subject area. Only one out of 15 candidates interviewed failed all three portions of the test.
        The 15 candidates had college degrees from nine private and four public colleges and universities, with two unknown. Although most had received a first degree in 1998, several were teachers who had moved to Massachusetts after teaching in other states, and one had 20 years experience as a teacher.
        At the time of the interviews, eight of the fifteen were certified to teach in Massachusetts; eleven were certified in at least one other state--including Connecticut, New York, Rhode Island, New Hampshire, New Jersey, California, Georgia, Arkansas, Missouri, North Carolina, Maryland, and Tennessee. To receive such certification, candidates typically had passed a required test. Nine of them had taken the National Teachers Examination (NTE), recently renamed and broadened to become the Praxis. Others had taken specific state tests, the Graduate Record Examinations, or the Millers Analogies Test (MAT).
        Four took the MTT in April 1998, during the test's first administration. Nine took the MTT for the first time during the second round of testing, in July 1998. Two took it in October 1998. All candidates interviewed submitted scores for the morning two-part literacy portion of the test. Candidates also submitted scores for elementary education (6), English (2), physical education (2), and general science (1), physics (1), music (1), middle school (1), and special needs (1).
        Although a common rationale for teacher certification tests such as the MTT is that they will protect schools, parents and school children from incompetent teachers, the MTT tests did not prevent most teacher candidates in the sample from securing work in some kind of teaching capacity. Regardless of whether they passed or failed the MTT, 12 of the 15 candidates interviewed currently work in public, private, parochial, and charter schools, both in and out of state.
        Of the seven candidates who passed the MTT, two are working in full-time teaching positions in Massachusetts public schools, while two are working in-state as long-term public school substitutes. Two more work as full-time teachers in public schools out of state. One candidate is not working by choice.

        Of the eight candidates who "failed" the MTT, six are working in schools. One works full-time in a Massachusetts charter school; a second works as a teacher in a Massachusetts private school; and a third works in the Commonwealth as a full-time public school teacher's aide. Of the remaining four, three work as full- time teachers out of state, one each in a public, private, and parochial school. One candidate is working in the travel industry, and one is not working by choice.

        Vignettes

        Before summarizing the general findings from our 15 interviews, it is useful to provide vignettes of two candidates, to highlight the diversity of candidates and their experiences. Pseudonyms are used in these vignettes to maintain our agreement of confidentiality with interviewees. (Note 9)

        "Peter McHugh"

        Peter McHugh recently graduated summa cum laude with a 3.9 average and a major in physics from one of New England's top-ranked private colleges. His scores on the Graduate Record Exam, as documented on score reports he sent us, were 650 on the GRE- Verbal, 750 on the GRE-Quantitative and 700 on the GRE Analytical. These scores correspond to the 91st, 89th and 86th percentile among all those who took the GRE between Oct. 1, 1994, and September 30, 1997 (ETS, 1998). In contrast, he scored only 82 on the MTT Reading test. Since the DOE has issued no technical documentation on the MTT, we cannot be certain what percentile ranking would correspond with a MTT Reading test score of 82; but on the basis of MTT scores we have collected, we estimate that Peter would fall somewhere in the range of 80th to 85th percentile on the MTT Reading test--obviously quite at odds with his performance on the GRE, especially since the population taking the GRE is more selective than that taking the MTT.
        Even before graduation from college, Peter was offered a job to teach at one of the country's outstanding high schools in a state that recognizes a Massachusetts teaching certificate as certifying eligibility to teach.
        Although a Rhode Island resident, Peter did practice teaching in Massachusetts schools and signed up to take the Massachusetts Teacher Tests in April 1998 in the belief that the results would not "count" toward certification. Less than two weeks before the test date, when he learned he would need to pass to receive a state teaching certificate, he threw all his energy into studying for the exam. Reviewing the test guide, he found one sample question for science, but nothing for physics, and, although he was told he would receive a list of test objectives, he never received one. In the absence of any guidance to what he might encounter on the exam, Peter took the state's curriculum frameworks and studied "about 15 hours a day" for two weeks, developing his own study. Like some other candidates interviewed, Peter found testing conditions for the listening portion of the test "absolutely atrocious" and the clarity of directions "not good at all." He explained:

On the communication and literacy sections, sometimes you would have to write in the booklet and sometimes the answer sheet. You'd go back and forth. It wasn't consistent. It was a disaster. The proctors didn't understand. It ended up being a student who figured it out and explained it to the rest of us for one section. This took about five minutes and you started the test pretty frazzled.

There was content on semiconductors. There were graphs and charts I was supposed to analyze, and I knew nothing about semiconductors. There were two 600-1000 word essay questions at the end. One was appropriate; the second was on motors and generators. Most high school textbooks have no more than a section on this. Until the morning of the test, I had never studied motors. In most high school physics courses, you don't get to this. The questions didn't accurately reflect what is covered in a high school physics course. The content went much beyond that. The last 25% of the test had content I had never seen before. I had to skip 7 or 8 questions because they were on concepts I had never seen in 14 college courses. I filled in the bubbles because there was no penalty for guessing.

        "Allegra Karnofsky"

        Allegra Karnofsky, a Connecticut resident, graduated in June 1998 from a nationally-known music college based at a private four-year university out of state. Before taking the Massachusetts Teacher Test in July, Allegra had passed the Reading, Writing and Math and Music Concepts and Content portions of the Praxis, with scores required for certification in Connecticut and New Jersey.
        Allegra registered for the July test knowing little more than what she had read in newspapers about the April MTT results. She reported, "What was nerve-wracking was having heard that so many people failed in April." Although the NES registration booklet noted that study guides were available, Allegra's multiple calls to the testing company and Department of Education did not produce such a guide.
        Because Allegra had taken the Praxis test for music teachers, she expected the MTT would test something about her teaching ability and was surprised at how little subject matter knowledge it covered. She said:

There was a lot missing. There was some music theory, but I felt I was taking a test of music history. I am a K-12 certified teacher in Connecticut, and nowhere do you teach music history unless it's at college. The essay questions were very broad and open-ended, and both were music history -- for example, compare a 20th century composer and an 18th century composer. But that's not music education, it's music history. Most content was on classical, and there were one or two questions on American jazz. As a music education teacher, I need to know about a lot more than classical or jazz. There was no multicultural or world music, no Latin music.

I think all teachers should know their subject. In the Praxis test I took, you take one test, then schedule the computer-based testing when you wanted. I don't see what's wrong with the Praxis test. If Massachusetts went for the computer-based test, it would make so much more sense instead of taking a paper-pencil test.

        Summary of findings from interviews

        The two vignettes above recount how two individuals experienced the MTT. These are only two of the 15 teacher candidates we interviewed. We realize fully that a sample of 15 self-selected individuals provides an extremely limited base from which to try to generalize to the experiences of all candidates who have taken the MTT. But we think it useful nevertheless to summarize some of the themes that emerged from interviews as possible causes for the low reliability and poor validity of the MTT. (A more detailed account of what was learned from the interviews appears in appendix 3.)
        Many of our interviewees were dissatisfied with the information available about the MTT. Among items mentioned were the lack of a study guide, confusion over whether the April results would "count" towards certification, lack of information about conditions of retesting, and lack of detailed feedback on strengths and weaknesses of initial test performance.
        A second theme among our small sample of interviewees was the conditions under which the MTT were administered. Most candidates interviewed found the general testing environment to be reasonable or on par with that of other tests taken. Nonetheless, close to half of them expressed concerns about the clarity of directions during MTT administration and about the conditions under which they performed the dictation exercise portion of the test.
        Virtually all candidates interviewed mentioned the length of the MTT tests overall as excessive and believed that their 8-hour duration adversely affected their performance. Many compared the length of the MTT unfavorably with other tests they had taken and noted that the amount of writing required led to fatigue.
        Teacher candidates interviewed also raised questions about the match between their "real-world" literacy skills and the test content in both the literacy and subject matter portions of the MTT. Regarding the former, candidates questioned the value of the dictation exercise and of specific questions, such as "Define a verb." As for the subject matter tests, interviewees expressed doubts about whether the tests matched Massachusetts curriculum frameworks and whether the test content was a reasonable reflection of the demands of real-world teaching. Some candidates interviewed also reported surprise that the MTT did not cover content they expected based on their experience taking other teacher tests, such as the Praxis or other states' teacher certification tests. Despite the range of concerns interviewees expressed, all agreed it was reasonable to ask teacher candidates to pass a test prior to certification. They were aware, as one put it, that "most of the professions have a test" and viewed testing one of the rites of passage into a profession. But in general, interviewees reported that the MTT compared unfavorably with other teacher certification tests they had taken.