Appendix 3
Summary of Results of Interviews with Examinees

Our interviews with fifteen candidates who had sent us copies of their MTT score reports yielded background on the information available to test takers, administration of the test, including testing conditions, test content, test length, and test-takers attitudes about testing requirements for professional certification. In part 4 of this report, we summarized the nature of candidate views on these topics. Here we provide more detail on what was said by how many of our sample of 15.

Information available to test takers

According to teacher candidates interviewed, neither the testing company, National Evaluation Systems (NES), nor the Massachusetts Department of Education, provided adequate information about the test prior to administration. Regardless of whether they took the MTT in April, July or October, not one of the 15 teacher candidates interviewed reported receiving any useful information (or information they considered useful) from the Department of Education. Nine (60%) saw no study guide at any time. Two July test- takers called the Department of Education repeatedly and still failed to receive any information. Others reported receiving information that was misleading or inadequate, either regarding the consequences of the test or about test content or format.

Candidates reported that when they called the Massachusetts Department of Education, they were told no study guide was available. Of the candidates interviewed, all who contacted the DOE found that contact unsatisfactory. They described this contact and the information provided on the test as "not particularly helpful," "not readable," "vague" and "very limited."

Information for first-time test-takers

April test-takers registering for the test believed that test results would not "count" toward certification in Massachusetts. They reported that their colleges and universities also believed that because this was the first administration of the test, their scores would be used only to determine passing scores for future test-takers. This understanding was based on more than hearsay. The NES "1998-1999 Registration Bulletin" that many first-time candidates received reported:

No qualifying score will be established until after the first two administrations of the tests on April 4 and July 11, 1998. Candidates who must take the tests and are eligible to participate in those first two administrations will satisfy the testing requirement by completing the tests. A qualifying score for each test will be determined by fall 1998 and used beginning with the October 3, 1998 administrations. From then on, candidates for provisional or provisional with advanced standing teacher certification who must take the tests will have to achieve a qualifying score to meet the certification requirement (p. 2).

One candidate prepared her own study guide based on Massachusetts curriculum frameworks, noted, "The six of us who graduated were not told much. Less than two weeks before the test date we were told we needed to receive a certain score to get a certificate."

July test-takers reported they received most of their information about the test from the media. Because of widespread media reports, some reported they were able to anticipate certain test components, specifically the dictation portion used to test "literacy and communication skills." In September 1998, the State Department of Education released a guide entitled "Massachusetts Teacher Tests: Test Information Booklet." The booklet includes test objectives, sample questions, and criteria for test scoring. This resource was unavailable to April and July test takers.

In the view of some candidates interviewed, the confusion about the test's consequences contributed to depressing test results. Whether they passed or failed, spring test- takers in particular attributed low test scores overall to the message that the test would not "count." One April test-taker noted, "They had told us the scores weren't going to matter. I thought it was a joke." Another reported:

Many of us were in the middle of student teaching. We were really busy, and most people didn't study so much as I did. This could explain why so many people failed the first time around and then passed the next time.

A third candidate who passed all three portions of the test nonetheless remarked:

I would guess that a number of people who took the very first test took it in panic mode. Because of all the hype, a lot of people went into the test in a panicked frame of mind. Under these circumstances, test scores are not an accurate representation of what they can do.

Information about Re-testing

Candidates interviewed also reported receiving inadequate information, if any, regarding a re-test for those who failed the test on the first round. For example, test- takers knew that free re-tests were offered to all candidates who failed one or more sections of the April test. However, one July test-taker reported that she did not know if such provisions would be made for those who failed the July test. Another candidate who failed the April test but did not intend to teach in Massachusetts described her uncertainty as to whether a re-test was still required because she had graduated from a private college in Massachusetts. She reported, "I was on the phone all the time. No one was giving me a definite answer."

Information about test results

Several candidates observed that they received individual test results in a format that precluded them from learning from their mistakes. One who passed the test on the first round in April asserted, "It is not surprising at all that there are repeated re-tests. There is no information at all about what you did wrong, no information about what you did right." In part, these test-takers' wish for more precise information about their performance reflected some doubt about the objectivity of the questions and their scoring. As one noted:

I'd like to see what I got wrong. A lot of questions were subjective and poorly worded. I could have supported any of my answers, even if they were marked incorrect.

Given the lack of either a study guide or individual feedback, some candidates were left with the belief that their results were a matter of chance. For example, one experienced teacher who passed still found the morning literacy portion of the test "very subjective." As she explained:

For example, [on a question requiring the] summarizing of an article. I thought I did a good job, but my "bar" said I did an "adequate to inadequate" job on this. You could have three people grade this and come up with three different scores.

A July test-taker added:

[There were] a couple of open-ended questions on how you would teach something. But different people could teach it in different ways. There was more than one right answer. There was no guidance, no rubrics. I would have liked to have seen a study guide.

Some candidates who passed the MTT on the second round attributed their scores as much to luck as to preparation. These candidates saw little difference between the knowledge they possessed at the time of their first test, compared with that knowledge at the time of their second test. Pressed to explain the differences, they alluded to differences in testing conditions, including more time to take specific sections of the test, familiarity with test format and content, or luck. For example, questioned as to what they thought had contributed to raising test scores from one administration to the next, one candidate who had failed the first round on the elementary portion of the test by two points replied, "It's a matter of luck." Another responded, "I don't know. Chance, totally chance!"

Testing Conditions

Candidates interviewed had taken the Massachusetts Teacher Test at seven different locations around Massachusetts. These testing sites included Bunker Hill Community College, Auburn High School, Burncote High School in Worcester, Malden High School, West Springfield High School, Randolph High School, and Wakefield Memorial High Schools. We asked the candidates to describe the testing conditions they experienced as they related to physical comfort, breaks allowed, adequacy of time to complete the test, acoustics, clarity of directions, and variations available for candidates with disabilities.

Most candidates generally found the general testing environment to be reasonable or on par with other tests taken. Problems at specific locations were noted by four candidates, including two July test-takers who mentioned heat and lack of air-conditioning as a problem, one who described proctors who talked "very loudly" among themselves for 20 minutes during the testing, and one who reported the test started late. However, almost half (47%) reported serious problems with acoustics that hindered performance on the written mechanics portion of the test. In addition, almost half also described problems related to the clarity of directions and test organization. Others raised concerns about the length of the test overall.

Acoustics
The morning portion of the MTT purports to test written mechanics by asking candidates to listen to a passage on a tape recorder, then write down the passage word for word. The purpose of this exercise is to "demonstrate the ability to spell, capitalize, and punctuate according to standards of edited American English" (Massachusetts Teacher Tests: Test Information Booklet, 1998, p. 9). Despite the apparently straightforward nature of the task, candidates reported a number of problems in its administration. Although eight candidates tested at four of the sites described the testing acoustics as "fine" or "very quiet," other candidates described conditions that hindered test performance. Five candidates (33.0%) tested at three different locations reported hearing tape recordings from other testing rooms. One test-taker voiced a common complaint:
The acoustics were absolutely atrocious. The classrooms were adjacent to each other. I was near the back. I couldn't hear the tape recorder in my classroom, but I could hear the tape in the classroom behind me, and they were not in sync.

Clarity of test directions and format
In addition to variations in and problems associated with the quality of listening conditions, some candidates described problems with the clarity of test directions and test format. Specifically, while six candidates characterized the proctors' understanding and test's clarity of directions as "fine," "okay," or "not bad," nine others described proctors as "poorly prepared" or "kind of lost," and directions as "vague."
Several April test-takers noted that the lack of consistency in the test's format was a problem. As one explained, "You went from multiple choice to essay to fill- in-the-blank to correct a sentence that was incorrect." A July test-taker described confusing directions in relation to test items on the elementary education sub-test. For example, she explained that when she came to one test item, she was "not sure if they wanted a lesson plan or an essay." By the third administration, test organization and clarity of directions still raised concerns. As one October test-taker reported:
I had a problem with [the clarity of directions]. They were extremely disorganized.... Because there were so many subtests...you had the blue section of Test A ...it was confusing, with too many subtests, colors, page numbers. It was terribly confusing. I don't see why they couldn't have had one test booklet. Also there was a misprint. They had misprinted the direction of where to answer something. They said it was on one page, but it was on another.
Two test-takers also described confusion related to the "tricky" selection of answers for multiple choice items. For example, one candidate estimated that up to 70% of the special education questions offered choices like "1 and 3 only," "2 and 1 only," "3 only," and "4 only."
Length of test
Whether they had passed or failed the first time they were tested, virtually all candidates mentioned the length of the test overall as excessive and believed it had an impact on their performance. One noted, "It was too long being in one room." Another reported, "Time was the big issue. Eight hours was just too long."
In particular, several candidates mentioned how the test's emphasis on writing essay questions or copying down dictated material contributed to increasing fatigue or incoherence over the course of the day. One candidate, an experienced teacher certified to teach high school English in New York, New Hampshire, and Connecticut and who described himself as a "good test-taker," compared the MTT to other state tests and reported:
In other tests, a third was definitely related to education, a third was about literature, and a third had to do with analytical or problem solving skills. The New York test had only one essay. In Massachusetts, there was one in the morning, and two in the afternoon, plus the morning dictation. There was a lot of writing. I was used to it, but my handwriting's awful. By the time you get to the end of the day, you're not legible.
Another candidate, an experienced elementary teacher with teaching certification in two other states added:
There was a lot of writing in terms of essays...summary of a story, essays on different topics, two more essays on elementary ed.... It was such a long process. Having the writing at the end was hard. In the end, I was exhausted. By the time I got to the end of the content test in elementary ed, I was just filling in the bubbles. My knowledge of the elementary subject area is much higher than I scored.
Second-time test takers faced some relief from the demands of an eight-hour testing day. Those taking only one or even two portions of the MTT as a re-test consistently reported that "it was easier" or "it was shorter," leaving them less tired. Candidates interviewed who took a re-test reported benefiting from the opportunity to re-rest under less pressure and with more time allowed. For example, one candidate who scored 60 on the writing portion of the test in April, passed with a score "in the high 80s" in July. Asked how she would account for her improved score, she reported:
I knew I would be out in two hours because I was taking only the writing. I think the only thing I can think of is that I wasn't about to pass out. Maybe my handwriting was better because I only wrote for two hours.
Another April-test taker who passed the writing portion of the test on the second try reported no difference either in her approach to the test or in the format or difficulty of the questions. However, she noted, "It was a lot shorter. I only had to take one part. Mine was the 8:00 to 10:00 slot, so I was done for the day, and I was not exhausted." Some repeat test-takers also noted that on the re-test, they had the entire morning to complete a section that was allotted only two hours the first time around.
In commenting on the impact of the overall length of the MTT, candidates compared the MTT unfavorably to the length of other post-graduate tests. For example, one candidate who had also taken Graduate Record Examinations in her subject area noted that she had taken the GRE in four and half hours. Others noted that because it was possible to take the National Teachers Examination in sections, it was not so tiring. As one elementary teacher explained:
I took the pre-professional skills tests for the NTE on three separate days. You had different portions, but usually you'd come in for testing at 8:00 and be done by 10:30. NTE was offered every weekend, so I could break it up into different days.

Literacy and Communication Skills Content

Teacher candidates had a number of questions about the match between their "real world" literacy skills and test content. Whether they passed or failed the test, candidates reported that they found some of the content perplexing. Without any study guide or information from others who had taken the test before them, the expectations of April test-takers in particular were shaped primarily by other post-graduate tests. Given this, first-time test- takers were especially surprised at the dictation portion of the test, and several who had taken other teacher tests noted they had never encountered anything like this on any other examination. Generally, candidates were confused about what skills and knowledge the dictation portion of the test was attempting to assess. One successful October test-taker commented:

The dictation section was mindless, a complete waste of time. They could have just given us the section and had us punctuate it. You don't have to write it down word for word. A monkey can do that.... It was very simple-minded.

Others also questioned other portions related to reading and writing. As one successful candidate reported:

The reading and writing content didn't seem appropriate. [It was] not testing my knowledge of reading or writing. Like the question, "Define a verb." The test didn't seem to be testing what it said it was testing.

Subject Area Content

Candidates also detailed concerns about the content in the subject area tests. Their comments detailed concerns that much of the content did not match content found in the Massachusetts frameworks or the demands of real-world teaching responsibilities.

Content not included in grade-level curriculum frameworks

Three of the candidates, all of whom passed the test on the first round, believed the content tested extended beyond the knowledge they needed for teaching. One candidate mentioned that content included went well beyond what she would expect to teach her middle grades students. She asserted, "There should be separate tests for middle and high school science. Never will I teach advanced physics in middle school."

Another who took the English subject area test questioned the extent to which some content tested even fit within the boundaries of the discipline. As he commented:

I distinctly remember a test question on "new journalism." I see this as a separate field. The question had to do with who was responsible for the rise of new journalism. I have a BA and an MA, and sometimes journalism students came into our classes, but we didn't take theirs. I've studied criticism, but not journalism.

Lack of content about professional knowledge

Some candidates interviewed reported surprise that the MTT did not cover content they expected based on their experience taking other teacher tests, whether the National Teachers Examination (Praxis) or state tests. Most often, candidates mentioned the lack of content related to professional knowledge and skills and commented on the lack of attention to teaching itself. For example, one recent graduate and physical education teacher noted:

The content wasn't fair. There was not a lot of application to how you taught. The test did not have, "In this situation, what would you do?" kinds of questions.

An experienced elementary teacher certified in two fields in another state likewise wondered how well the MTT could determine teachers' competence in classrooms given the mismatch she perceived between the demands of teaching and content on the MTT. She asserted:

The MTT did not test my ability to teach. As a teacher, I'm constantly reteaching myself. There are so many things you do as teacher that are performance- based, not knowledge-based. You need to know teachers are performing, handling their class in a professional manner.

Another teacher with twenty years experience and certification in two other states, also questioned what she perceived as an overall stress on mechanical skills and content knowledge compared to the limited emphasis on knowledge of classroom practice. She reported:

There was a big lack of anything to do with classroom management. It was all content knowledge. There were no more than a couple of questions on teaching methods, classroom management, anything to do with your ability to teach.

In light of limited content geared toward assessing professional knowledge, candidates raised concerns that the emphasis on content to the exclusion of other aspects of teaching reflected a misunderstanding of what was required to be a successful teacher. As one candidate with an exemplary academic record asserted:

Nothing on the whole test addressed the issue of how well you teach. Even if I'd gotten every question right, it wouldn't have proved I could teach physics to 16-year-olds.

From the perspective of another candidate, even when test items ostensibly melded content knowledge with teaching knowledge, the questions did not provide adequate information to answer them well. For example, one experienced teacher described one question on the middle school portion of the test as unrelated to real-world teaching conditions:

You were supposed to create a unit plan with a team, but in the test, you're on your own. Then you're told you're not being graded on the creativity or usability of your plan. But there are no references, resources, or curriculum frameworks to work with. You're told your response will be graded on your knowledge of your subject area, not on the lesson plan.

Candidates viewed the weighting of basic literacy and writing skills over teaching knowledge, regardless of the subject area, as a major flaw in the test. As one explained:

It's possible with the Massachusetts test for some very good teachers to be knocked out because of problems with spelling. A lot of what we do [in classrooms] is done ahead of time. It's not a handicapping situation not to know everything there is to know. It's also possible that a lot of bad teachers are passing. People could pass the content test without knowing how to teach.

Views of teacher testing

Despite the range of concerns raised about the MTT, the candidates interviewed did not object to standardized testing per se, and all agreed it was reasonable to ask teacher candidates to pass a test prior to certification. However, they added that although they were not opposed to the testing of new teachers, they believed that the MTT should be replaced with a different test. Whether they passed or failed, candidates interviewed were aware that "most of the professions have a test" and viewed testing one of the rites of passage into a profession. However, candidates doubted that the Massachusetts test in particular could adequately assess teacher competence. They compared the MTT unfavorably to other professional tests they had taken and described the latter as having a more balanced focus on all aspects of teaching, including content knowledge, classroom skills, and problem solving. As one noted, "Yes, I'm in favor of a fair test like those offered in Connecticut or New York. But this one is not fair at all."

Candidates noted in particular that, compared with other tests, items pertaining to the teaching process itself were missing from the Massachusetts test. As one explained:

In New York, you have to pass a basic liberal arts and sciences test.... But in New York you have to pass a written assessment of teaching skills. This includes subject knowledge but also more teaching. There are also two essays associated with classroom situations.

A third elaborated:

It's not unreasonable to have new teachers take a standardized test. It's only unreasonable to take an unproven test that has no validation data. This one is much too subjective. The New York test was more fair in assessing teachers' ability to teach.... If they want to know if you can write under pressure, that's what this test shows.

Eight candidates explicitly mentioned the National Teacher Examination as a more accurate test of teaching skills. One experienced teacher who taught elementary school in two other states before moving to Massachusetts asserted, "The Massachusetts Teacher Test needs to look into the NTE. It's recognized, accepted, reliable." Another described the NTE as testing "a wider range of content-- communications, professional knowledge, and general knowledge." Another concluded, "Massachusetts could do better, could have a much better quality test."

Appendix 3 Summary of Results of Interviews with Examinees