Reasonable Decisions in Portfolio Assessment: Evaluating Complex Evidence of Teaching

Abstract

A central dilemma of portfolio assessment is that as the richness of the data available to readers increases, so do the challenges involved in ensuring acceptable reliability among readers. Drawing on empirical and theoretical work in discourse analysis, ethnomethodology, and other fields, we argue that this dilemma results, in part, from the fact that readers cannot avoid forming the data of a portfolio into a pattern—a coherent "story" or "stories"—in order to evaluate it. Our article presents case studies of readers independently evaluating the same portfolios. We show that even readers who hold a shared vision of effective teaching and who cite much the same evidence can, nonetheless, develop significantly different "stories." Our analysis illustrates that some portfolios are more ambiguous than others and are thus more likely to result in such divergent readings. We argue that more fine grained understandings of portfolio ambiguities and disagreements between readers over "stories" can help us respond to the challenges posed by the rich data of portfolio assessments.