Contributed Commentary on
Volume 4 Number 17: Taylor & Nolen Reframing Assessment Concepts
|
Rick Garlikov demo42@uabdpo.dpo.uab.edu |
Catherine S. Taylor ctaylor@u.washington.edu | Susan Bobbit Nolen sunolen@u.washington.edu |
The following is an e-mail exchange that Susan B. Nolen and Catherine S. Taylor had with Rick Garlikov in response to follow-up questions Garlikov had about their article "What Does the Psychometrician's Classroom Look Like?: Reframing Assessment Concepts in the Context of Learning" (EPAA Volume 4 Number 17)Garlikov: I have read "through" (more than cursorily, but less than thoroughly) your EPAA paper, and I have some questions.
1) If teachers and pre-service teachers are evaluating their own assessment instruments, how can they see the flaws that they didn't see when they made up the instruments? Isn't it almost impossible to evaluate your own evaluation efforts very accurately? E.g., a friend made up a test one time where her students had to rank the developmental order of some five or more stages of development. But she took off for each one that was not in the correct stage -- i.e., first, second, third, etc. The problem was that if a kid missed the first one and then had the right order of the rest, but each one slid back one notch, they missed them all. But a kid who had no clue as to the order might get two or more in the right slot by just guessing. My point was that my friend didn't see the problem with this test question. She said each part was worth only five points, but actually, each part was worth much more, because any wrong answer COULD throw off the other ones. If SHE were evaluating her test, she would have said it was a good test. Yes, no? How does your course deal with this aspect of judging one's own evaluations of students? Nolen: This is why the draft-feedback-revision loop is so critical. Our job as instructors is to find the flaws and point them out. Often this entails telling the intern how students might interpret a misleading or unclear question, playing the role of their students. In the course of the term, most students are not able to actually try out items and assessments, though some do. More try them out the following quarter when they are responsible for teaching a unit of this size.
The fact that we require students to construct model responses and scoring rules for their items, as well as write rationales for assessments, scoring, and objectives (and their relationship), provides another way for them to see the problems with their initial work. This comes through pretty clearly in their self-evaluations. Garlikov: 2) It SEEMS to me, at least today, that if you cover all the things in your course that you discuss in the paper, that kids will be hard-pressed to get an intuitive understanding of testing or evaluating students -- though they will probably get an intuitive understanding of the problems of testing or evaluating students. It seems to me there is too much specific, technical detail involved for understanding to become likely. It may be that I just tried to read too much in too short a time, but I felt the quantity and nature of the material you discussed made the concept of evaluating more complex than it had to be -- for a student. Your article seems good for someone who already understands general forms and limits of evaluations --gives nice details in a systematic and thorough way-- but I worry about how much a pre-service teacher could absorb of it all. Or do you teach about this in a way very different from the way you constructed this particular article?
Nolen: In the course we try to teach both by presenting some information through readings and lecture-ettes, and by having students construct a cohesive unit plan. Although the presentation part is necessary, they don't really begin to *learn* it until they try to put the theories and methods into action. This learing (we think) continues long after the course is over as they try to assess fairly and informatively in their own classrooms. There is a lot of information in the course, but I think it is made learnable in several important ways:
First, the fact that all of the activities are embedded in their unit plans, rather than merely appearing as unrelated activities. The unit description and goals lead to the learning targets (objectives), which along with their knowledge of the subject-matter disciplines leads to appropriate methods of instruction and assessment. Because the most important mode of instruction for us is also our mode of assessment, we model for them how assessment can be an instructional tool. Second, we consciously draw on what they have studied in their other teacher prep courses, and expect them to use their knowledge from those courses to justify their assessment decisions. Thus the assessment content is seen, in part, as a natural extension of what they have been learning in the program. Finally, remember that students have been assessed a lot by the time we see them, and most have also seen and/or assisted with assessment in their field experiences. Most are quite concerned about being able to fairly and accurately assess their own students. From our experience, this (along with subject-matter preparation) is sufficient base on which to build their knowledge of assessment. I'm not sure what you mean by intuitive understanding, exactly; I know that several students have told us that they can no longer think about planning instruction without thinking about assessment as part of instruction. It seems, for many, to have become part of their instructional schema. As you say, the article seems good for those who already know something about assessment: That is to whom it is directed. We don't have our students read the article, we have them learn by doing with feedback. Taylor: We know that learning in a University context is imperfect and that our students will continue to learn after they take our course. At this time, students take the assessment course during the 2nd or 3rd quarter of a 5 quarter sequence. Our students have an opportunity to use their work in the field after the course is completed and to give us feedback about how things are going. We literally have students tell us "thank you" in the halls during the quarters following our course. They claim that the thinking they learned to do in the course helps them develop strategies to focus their teaching better and to plan their teaching better. One even said, "My students even thank you." As Susan said above, rather than this course resulting in superficial coverage of assessment concepts and skills disconnected from other elements of teaching, the course is set up so that they grapple with the meanings of assessment concepts in a meaningful context - their own plans for teaching subject matter. I guess I'd have to say that I don't know what "intuitive" means in the sense you are using, but a habit of mind about how to stay clear about the goals of instruction, how instruction helps students reach those goals, and how assessment actually assesses for students' learning of those goals seems to me to be a powerful "intuitive" process.
As a parent, much of what I see in my own children's school experiences is random or text book driven. My children are learning more about how to please the idiosyncracies of teachers than they are substantive conceptual or procedural understandings (or even social understandings). The kinds of assessments they have reinforce a notion that science (or social studies, or English) is a list of facts to be memorized or is a teacher whim. I ask them "what did you learn from that experience" fairly regularly and am dismayed by the responses. Garlikov: As Susan knows from my EDPOLYAN/EDPOLICY writing, this is one of the things about schools that frustrates me the most. Some of it is due, of course, as you say, to assessment techniques that give the impressions they do, but I also suspect many teachers (and many adults in general) really DO think that these subject matters really are some specific body of facts that need to be learned or memorized. So they may actually be assessing in ways that reinforce what they intend to be teaching. Which, if so, is, of course, disappointing to me.
Taylor: I hope that what we give our students is a way of thinking that helps them, not only be technically better assessors, but better, more focused, more fair teachers who use assessment as ways to assess student learning and to communicate to students about what is important to learn. Because our students' first passion is helping students learn, many can embrace this view of assessment more easily than they can a view that portrays assessments as tools for dispassionate observers and graders. The spin off is also surprising. Today, in reviewing file for a teacher education scholarship, the winner was one whose cooperating and supervising teachers both stated that his teaching was focused, his goals for instruction were clear to students, and his assessments were "eagerly embraced" by students because they "knew what the purpose was."
Garlikov: Thanks for answering my questions. I think you pretty much addressed what I asked, but I have one more simple question, and then a much more important question.
Simple question: Do you think that your students understand that the kind of crucial feedback you give them when you go over their assessment plans, etc. is what they also need to see their students as doing if and when their students complain about particular items or scoring, etc.? That is, can your students generalize from the kind of thing you are doing in this regard, or is it that they pay attention to you because a) you are the teacher so they pay attention to your claims of invalidity (or some other sort of flaw), or b) you can articulate the flaws in their assessments extremely well and cogently. In short, are you able to get them to see that whenever one of their students might complain about the reliability or validity (or fairness or whatever) of a particular evaluation tool, they need to really listen and try to figure out whether there is any merit to the claim? Nolen: For some, yes. Probably for most in theory. We talk some about the power dynamics involved in these things, and in both the preceding ed psych course and this course we emphasize listening hard to students. (In fact, their major project for the previous course is to do just that: Listen hard to two students talking about what they learned in two consecutive class periods, and trying to explain why that's what they learned.) We are not the only ones in their program who model this, though not all profs do as much to encourage revision and rethinking because of time required.
Garlikov: Difficult question: *I* don't know how to construct good "formal", individual exams (or paper assignments) about philosophical/conceptual/logical issues that measures what I wanted students to learn. I only get some ideas about what students might have learned by continuing back-and-forth dialogue that further probes answers they give, questions they ask, and comments they make as we go along. This almost never ends up giving me the impression that their initial answers gave me; and often I am left with the vague feeling that if we carried it even further, they would either change or they would give me an even different/better understanding of what they know. So I am never happy with a time-slice assessment except in those few cases where students either seem to have learned nothing about a particular issue or where they seem to express remarkably perceptive, genuinely independently discovered, reflective views. Since much of what I think is important in schools IS of a philosophical or conceptual or logical nature, how can teachers in general design formal assessments that can be said, or shown, to accurately reflect what students really know or understand? And how can one tell such assessments do that.
An example of the problem is almost any discussion on EDPOLICY [an educational listserv forum to which Susan Nolen and Rick Garlikov subscribe], where it takes a number of responses back and forth for everyone to even be clear about what is being asked or responded to; if it occurs even then. The initial tendency is to feel that someone is dead wrong or terribly misguided in some way, perhaps in some cases not even very bright. But it usually turns out that they were making a fairly cogent interesting point that was just difficult to express in some way that everyone could understand it. Yet teachers or assessment "instruments" often don't give students a chance to clarify, discuss, argue, clarify some more, etc. Or do you have a way around that? Nolen: We just try to model and talk about listening to students, especially trying to understand the responses that seem off the wall. We give examples from our own teaching (both in the u. and in public schools) of times we found great insight lurking in what seemed initially to be a wacky answer. And we try to model this during the (many) discussions we have in our classes. Capturing some part of the essence of a discipline, including the habits of mind or approaches as well as the big ideas and questions, is something our students are encouraged (required) to struggle with in several courses.. It often seems to come to a head in the assessment course, where they have to be very clear about what is important to learn and how they will know when their students have learned it, AND how that learning (and assessment) is what Bruner calls "intellectually honest" or what Schwab calls "true to the discipline." They will continue (we hope) to struggle with this throughout their careers, as we do.
Garlikov: You both mentioned you were not certain what I meant by "intuitive" understanding when I wrote:
It SEEMS to me ... that if you cover all the things in your course that you discuss in the paper, that kids will be hard-pressed to get an intuitive understanding of testing or evaluating students -- though they will probably get an intuitive understanding of the problems of testing or evaluating students. You did answer it in a way that pleases me, however. Catherine's additional comments particularly helped in light of what Susan had already written about the "details". I take it now that, although when your students leave your course they have some particular techniques and methods for assessing students in ways that are valid and instructionally re-inforcing and useful, the most important thing you give them is a sense for how assessment needs to work in classroom teaching, and what the essential pitfalls and problems of assessment are. That is why I distinguished between an intuitive understanding of testing on the one hand, and an intuitive understanding of THE PROBLEMS (and issues) involved in testing. The latter is I think (1) the most important thing you could give your students, and (2) the only ingrained ("intuitive") thing you were likely to be able to give them. I didn't think that in one course you could teach pre-service teachers how to intuitively make up flawless assessments for their students. I had had the FEELING as I had read the EPAA paper that there was a possibility you were claiming that you taught your students how to make up wonderful or perfect tests with no problem at all. But from your responses, I see that what you do (which is what I had hoped you were doing) is to give your students a really good understanding of what KINDS of things need to be done in evaluating students, and a really good understanding of why those KINDS of things are important. Insofar as you helped them learn how to design some specific assessments, that is good; but it is better that you have helped them understand the concept of assessment, since the specifics may change for them as they get into situations different from any you may have anticipated, or as their instructional ideas and principles change. Thanks for answering. And 'Good For You Guys' for doing such a good job teaching this sort of thing, and for writing the EPAA paper about it.