Exploring Candidate Interaction with a High Stakes Teacher Performance Assessment: The KPTP and the Kansas Laboratory

This qualitative study explores the experiences of a cohort of pre-service teachers completing a high-stakes teacher performance assessment (HSTPA), the Kansas Performance Teaching Portfolio (KPTP), during their final year of teacher preparation. The inquiry asks whether the act of completing the assessment modified candidate conceptualizations of good teaching, and, if so, in what ways. Data were gathered via indepth interviews and content analysis, and data were analyzed via constant comparison. The study found that completing the KPTP was having some impact upon participant conceptions of good teaching, prodding them to broaden their understanding of the work of teachers to include not just dispositional and relational aspects of teaching, but elements of technical teaching practice. The article concludes with recommendations for policy, research, and practice.


Introduction
In a famous dissenting opinion for a 1932 United States Supreme Court case, Justice Louis Brandeis argued It is one of the happy incidents of the federal system that a single courageous State may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country. (Gardner, 1996, p. 475) This argument, drawn from Federalist #56 (Madison, 1788(Madison, /1952 has become known as the statesas-laboratories metaphor, and has been interpreted to mean that the inherent sovereignty of state governments allows them to experiment with a variety of legal and policy solutions from which other states and the federal government can watch and benefit (Kronk Warner, 2015). Education policy in the United States has historically been representative of this metaphor, as the history of local and state control of education has created a multifaceted local and regional system of educational experimentation (Organisation for Economic Co-operation and Development, 2013). High-stakes teacher performance assessments (HSTPAs) likely exemplify the states-as-laboratories metaphor in action in the field of education. HSTPAs are a particular kind of summative, external performance assessment used by institutions or entire states to determine whether teacher candidates will be eligible for teacher licensure. The high-stakes part of the acronym is in reference to the fact that failure to pass the assessment will render a candidate ineligible for licensure, regardless of that candidate's successful completion of other licensure requirements, such as university coursework or student teaching (Wilson et al., 2014). HSTPAs are increasingly being adopted across the United States, both at the institution and state levels, to serve as gatekeeping exams into the teaching profession (Knight et al., 2014). The most common of these, the educative Teacher Performance Assessment (edTPA) (Stanford Center for Assessment, Learning, and Equity, 2017), has been adopted by educator preparation programs (EPPs) in 40 states, with several states requiring that all teachers pass the exam to receive certification (American Association of Colleges for Teacher Education, n.d.). The rise of the edTPA has led to the creation and adoption of rival exams by both states and educational testing companies. As mandated HSTPAs have become the norm rather than the exception across the country, the teacher education professoriate has grappled to come to terms with them. The edTPA, at least partially due to its ubiquity, has proved particularly polarizing, with one camp arguing that the edTPA has the potential to bring badly needed coherence and rigor to teacher preparation and the other arguing that the edTPA decontextualizes the work of teaching and narrows the teacher preparation curriculum. While the debate rages on, this paper posits that an essential component of that debate ought to be a consideration of HSTPAs more broadly, particularly those that have been in consequential use far longer than the edTPA. One such long-running HSTPA is the Kansas Performance Teaching Portfolio (KPTP), which has served a gatekeeping role for teacher licensure in that state since 2008.
As states and institutions across the country move rapidly toward adoption of some form of HSTPA to serve as gatekeepers to the teaching profession, it is concerning that, beyond the literature on the edTPA's precursor, the Performance Assessment for California Teachers (PACT) and the nascent literature on the edTPA, there is little peer-reviewed research on other HSTPAs like the KPTP or the California Teaching Performance Assessment (CalTPA) that actually have a longer history of consequential use (California Commission on Teacher Credentialing, 2015;Kansas Division of Learning Services, 2015). To return to Brandeis' metaphor, these states have been teachers" (Hollins, 2012, p.1) and help teacher candidates internalize that vision. To explore whether the KPTP can achieve those goals, this study was guided by the following questions: 1) Does the process of completing an HSTPA modify candidate conceptualizations of the work of teaching or what it means to be a good teacher? 2) If the process of completing the HSTPA does modify candidate conceptualizations, in what ways does it do so?

Literature Review
This study informs the scholarly knowledge-base within two strands of literature: literature relating to the design and implementation of HSTPAs, and literature exploring conceptualizations of good teaching and their importance to teaching practice and candidate learning.

HSTPAs
Advocates of HSTPAs have long called for the development and implementation of "a barlike exam" to gatekeep entrance into the teaching profession (Mehta & Doctor, 2013, p. 8). These groups and individuals look to the implementation of such exams in the fields of medicine and law at the beginning of the 20th Century as central to the professionalization of those fields and the increased quality of practice that came with that professionalization (Darling-Hammond & Hyler, 2013).
Most literature supporting the adoption of HSTPAs is predicated on the argument that rigorous gateway assessments can help to professionalize teaching by defining the essential competencies of good teaching. The jointly maintained American Association of Colleges for Teacher Education (AACTE) and edTPA (SCALE, 2016) website argued that the edTPA recommended passing score should serve as a bare minimum standard of competency agreed upon by teacher education professionals, but that individual states might choose to impose an even higher mandatory score for licensure. Butler (2015) maintained that the level of detail and rigor of completing the edTPA would help prospective teachers understand the depth of knowledge and skill required for successful teaching, and should increase the respect given to the teaching profession. Likewise, Hollins (2012) contended that the edTPA was a promising contribution to professional teaching because it was one of the first assessments to truly measure candidate ability to manage the complexities of teaching practice, not just their knowledge of teaching. Other proponents of HSPTAs have emphasized their potential for providing program improvement data to teacher preparation programs (Bastian et al., 2018;Darling-Hammond, 2010, Miller et al., 2015Peck & McDonald, 2013;Peck et al., 2014) and for creating opportunities for faculty collaboration and intellectual engagement (Lachuk & Koellner, 2015;Peck et al., 2010;Warner et al., 2020). Lastly, Ahmed (2019) argued that the mandate of an HSTPA could be leveraged by pre-service teachers to resist scripted curricula, and, therefore, better serve the students in their field experiences.
However, HSTPAs have also been questioned and criticized both by scholars and practitioners in teacher education. General concerns about HSTPAs include questions about their role in narrowing teacher education curricula (Dover & Schultz, 2016;Madeloni & Gorlewski, 2013), in redefining academic disciplines in ways that may not coincide with the disciplinary consensus (Hochstetler & McBee Orzulak, 2019;Warner, 2019), and in minimizing the importance of culturally responsive and critical multicultural education (Kleyn et al., 2015;Liu & Milman, 2013;Madeloni & Gorlewski, 2013;Picower & Marshall, 2016;Tuck & Gorlewski, 2016). As the most prevalent HSTPA, the edTPA has also critiqued for its connections to Pearson, a for profit corporation; for its significant cost to teacher candidates; for its use of external scorers; for its role in shifting the locus of educational control from the local or state levels to the national level (Greenblatt, 2018;Greenblatt & O'Hara, 2015;Margolis & Doring, 2013).

The Shared Ancestry of HSTPAs
While they differ in many details, HSTPAs, from the edTPA to the PPAT to the KPTP, share similar conceptualizations of the work of teaching, enshrined in the architecture of the assessments and derived from common ancestry. The family tree of today's HSTPAs begins with the portfolio assessment component of National Board for Professional Teaching Standards (NBPTS) certification (Cochran-Smith & Power, 2010, Pecheone et al., 2013. The NBPTS was constituted in 1987 to both develop standards for accomplished teaching and to create an assessment and certification system that would allow for identification and recognition of accomplished teaching (Serafini, 2002). It began certifying teachers in 1994.
The NBPTS portfolio consists of three sections: 1) Differentiation in instruction, which is composed of selected student work samples demonstrating learning growth over time, plus written discussion of instructional decision-making that helped facilitate that growth; 2) Teaching practice and learning environment, composed of video recordings of teaching accompanied by written analysis of teacher engagement with students and impact upon learning; and 3) Effective and reflective practitioner, requires gathering and analyzing student data and demonstrating the ability to use that data collaboratively with families and other stakeholders to impact student learning (NBPTS, 2017).
Current HSTPAs tend to have similar architecture, requiring candidates to assemble evidence in the form of student work samples, videos of practice, and student data, then analyze and explicate that evidence by responding in writing to specific prompts. As shown in Table 1 below, generally, HSTPAs center upon similar competencies: planning instruction based upon knowledge of learners and learning environments, engaging in and explicating that instruction, and analyzing and reflecting upon the efficacy of that instruction based upon data and evidence (ETS, 2017;Missouri Department of Elementary and Secondary Education, 2017;Nelson, 2016;SCALE, 2016). In her analysis of the edTPA, Sato (2014) argued that teaching within the assessment is conceptualized as an executive process in which the teacher is responsible for selecting worthwhile learning goals for his or her students and then orchestrating learning activities for the group of students in an effort to support them toward achieving those goals. (p. 429) Requiring comparable tasks, processes, and skills, HSTPAs seem to conceptualize teaching in similar ways, as a technical process of observation, planning, enacting, assessing, and reflecting. While this is a fairly common way of defining teaching, it is not the only way to do so.

Conceptualizations of Teaching
Several scholars have explored the tension created by competing and seemingly contradictory definitions of teaching that have shaped educational discourse and practice (Chen et al., 2012;Jackson, 1986;Pillen et al., 2013;Wang et al., 2011, Warner, 2016. Lefstein (2005) provided a succinct overview of this tension in his analysis of conflicting visions of technical and personal teaching. In the former, teaching is conceptualized as a "a technical method, which can be 'identified, disseminated and universally adopted'" (Lefstein,p. 334). In the latter, teaching is "viewed primarily as a relationship, which is constituted by how teachers and students interact holistically" (p. 347). Lefstein noted that the two sides of this dichotomy were frequently considered to be in conflict with each other, but believed that a more constructive approach would be to explore ways to address both conceptualizations. He argued, Purely technical teaching would lead to inflexible, unthinking, insensitive and amoral educational activity. An absolutist personal approach would deny teachers essential technical tools and important scientific insights, thereby confining them to the horizons of their own experience. Either vision, embraced exclusively and in its totality, becomes a dangerous hallucination. (Lefstein,p. 347) While Lefstein described ways that such a synthesis might occur, at present, much discourse about teaching is still firmly divided along ideological lines (Cochran -Smith et al., 2013;Sato, 2014;Lewis & Young, 2013;Wang et al., 2010). At least part of the conflict over HSTPAs discussed in the previous section is rooted in this very ideological disagreement over the nature of teaching itself. If good teaching is primarily defined as a holistic development of personal and transformative relationships, then no series of standardized tasks is likely to be able to capture its complexity or measure its impact. That is, if good teaching is personal, then HSTPAs are not measuring good teaching. On the other hand, if good teaching is a technical process, then a standardized performance assessment is a perfectly reasonable way to measure teaching competence. If Lefstein is correct, however, and good teaching is a synthesis of the technical and the personal, then a truly valid assessment of teaching competence would need to capture that synthesis.

Theoretical Framework
Assessment is conceptualized as fulfilling multiple purposes within the scholarly literature and the field of education at large. Within the field of teacher education, assessment tends to be either conceptualized as assessment of learning or assessment for learning (DeLuca & Volante, 2016). Assessment of learning is the older, more traditional approach, in which assessment tasks are summative and evaluative, intended to allow the administrators of the assessment to measure the achievement of individuals at the end of a learning experience. Assessment for learning is a newer construct that envisions assessment as an ongoing process in which formative diagnostic feedback from assessments allows for the adjustment of instruction throughout the learning cycle. In such a conceptualization, the act of taking an assessment itself is understood to be a learning activity.
Performance assessments, particularly the portfolio-based performance assessments upon which the HSPTAs were ultimately based, were initially conceptualized as formative experiences (Shulman, 1998). In fact, Shulman's work on portfolio assessment at Stanford in the 1980s eventually formed the intellectual basis of the NBPTS portfolio and, from there, the underpinnings of today's HSTPAs. Shulman (1998) argued that, that "whatever its effectiveness as an assessment form, the portfolio approach provided dynamite educational experiences" (p. 31). The claim of many proponents of HSTPAs is that they have the power to serve both as formative learning events and summative assessments of candidate mastery (Margolis & Doring, 2013;Meuwissen & Choppin, 2015;Peck et al., 2014). That is, proponents claim that HSTPAs are both assessments of learning and assessments for learning.
As detailed earlier in this paper, HSTPAs are certainly being used in teacher education programs across the United States as summative assessments of learning. This study was not intended to interrogate the validity of using HSTPs in that manner, though recent studies trouble and complicate the use of HSTPAs and the edTPA in particular for making predictions of teaching effectiveness (Choppin & Meeuwissen, 2017;Gitomer et al., 2019;Goldhaber et al., 2017). Instead, this study was intended to interrogate the assertion that HSTPAs can be assessments for learning by interrogating the experiences of a cohort of preservice teachers as they completed an HSTPA to determine whether the process of completing the HSTPA was shaping their understanding of the work of teaching.

Methods
This study is based upon a larger research endeavor to understand the experiences of a group of teacher candidates navigating their final year of professional teacher education coursework. To explore my research questions regarding the potential of HSTPAs to modify candidate conceptualizations of the work of teaching, I designed a qualitative instrumental case study, where a "particular case is examined mainly to provide insight into an issue" (Stake, 2005, p. 445). Building such a picture required a mixture of qualitative methods including semi-structured interviewing and content analysis (Marshall & Rossman, 2006). As much as possible, the findings of this study have been reported in the words of my participants in order to faithfully portray their beliefs, feelings, and tensions. Participant responses should be interpreted not as objective reality, a construct that the paradigm informing this study rejects as essentially unknowable (Lincoln & Guba, 1985), but as accurate representations of reality as they perceive it, and which might be perceived quite differently by other stakeholders such as policymakers or teacher educators.

Context
This study took place within the English teacher education program in the School of Education at a university in Kansas, Center University (pseudonym). Primary focus was on the experiences of a cohort of teacher candidates who graduated in the Spring Semester of 2015. These 13 pre-service teachers were navigating their final year of coursework during the time of this study. Prior to study initiation, all participants had completed the majority of their general education and English content courses, as well as approximately 30 hours of Education coursework. At study initiation, during the Fall Semester of 2014, all participants were enrolled in an advanced English methods course and an advanced teaching practicum. Upon successful completion of these courses, teacher candidates student-taught full-time in the Spring of 2015.

Participants
Participant selection for this study was based upon the concept of purposeful sampling (Lincoln & Guba, 1985). Since this study was not intended to produce contextless generalization, random sampling was not appropriate. Instead, sampling decisions were made in order to achieve maximum variation in information and perspective (Lincoln & Guba, 1985).
First, I was particularly interested in the constructions and experiences of secondary preservice teachers. The experience of secondary teachers is unique because they are often thought to be content experts first and teachers second; this impacts their understandings of what it means to be a good teacher and their sense of who they are as teachers (Cooper & He, 2012). Second, I focused on secondary English pre-service teachers because in addition to the tension discussed with all secondary teachers above, the experiences of English teachers are unique in that they "are often seen as responsible for the affective and attitudinal education of secondary students in ways that other content area teachers are not" (Hochstetler, 2011, p. 259). This emphasis on emotional work may have a significant impact on pre-service teachers' understanding of what good English teaching looks like.
Therefore, the initial participants for this study were the 13 students who made up the majority of the cohort of pre-service English teachers at Center University in the 2014-2015 academic year. The cohort was relatively homogenous, with most students having grown up and attended school in the Midwest. Twelve of the students were female, and only one was male. All were undergraduates, and most were traditional college students, in their fourth year out of high school at the time of the study.

Researcher's Role
As a methods instructor, field experience supervisor, and researcher, I assumed many roles in the context of this study; this presented both advantages and limitations (Yin, 1994). I was the instructor for the methods course that all of these students took during the fall semester of 2014, and I was also the university supervisor for 4 of the 13 participants in their advanced teaching practicum. During the Spring of 2015 I was the university supervisor for nine of these teacher candidates while they student taught. Being so deeply embedded in the context under study offered the kind of "firsthand involvement" and full "immersion" that are essential elements of qualitative research (Marshall & Rossman, 2006, p. 100). Throughout the course of the study, I interacted with these participants on an almost daily basis from August 2014 through May 2015, giving me the "prolonged engagement" (p. 301) that Lincoln & Guba (1985) argued was at the heart of rigorous and trustworthy qualitative research.
However, there were some drawbacks to my position as well. Several of my roles required me to evaluate my participants, which established an inherent hierarchy of power between us, and which held the potential to influence the ways in which participants responded to my inquiries. I attempted to mitigate these effects in a variety of ways. First, as with any ethical research, my participants were recruited voluntarily through a process of informed consent, and no member of the cohort was pressured to participate if he or she preferred not to do so. Second, the vast majority of the work that my participants did on the KPTP actually occurred within their spring seminar, a course with which I was not involved in any way. Though I was supervising some of these participants during student teaching, I was not the instructor of record for student teaching, as a clinical faculty member recorded final student teaching grades for based upon input from the university supervisor and the clinical supervisor of each student teacher. Lastly, my study focused on pre-service interactions with the KPTP, an instrument that was designed and scored not by me, but by readers outside of my institution. Clearly communicating these facts to the participants as part of the informed consent process helped minimize the impact of my status as an evaluator upon their responses to my inquiry.

Data Sources
I utilized two major data sources for this study. The first, and primary data sources were group and individual semi-structured interviews, which occurred at specific points throughout participants' interactions with the KPTP. I drew additional data from documents collected within the context of the teacher education program at Center University, with the most important of these being completed participant KPTP tasks.
The KPTP is composed of four tasks 1) Contextual Information and Learning Environment Factors, 2) Designing Instruction, 3) Teaching and Learning, and 4) Reflection and Professionalism. For task 1, candidates gather demographic, personal, and academic information about the students within their student teaching placements, and use that information to develop profiles of an entire class, a subgroup of students within that class, and two individual focus students. In KPTP task 2, candidates use draw upon the profiles developed in task 1 to design a unit of instruction for their students, including discussion of differentiation in instruction and assessment for the subgroup identified in task 1, and for the individual focus students. For task 3, candidates are intended to teach the unit they designed in task 2, keep a daily log of observations about their instruction, compile and analyze assessment results from the unit, video themselves teaching several of the lessons in the unit, watch those videos, and critically evaluate their own instruction. They are then supposed to use those data sources in task 4 to help them reflect on the efficacy of their instructional practice and to plan future professional development.

Data Collection
Data collection occurred in four phases over the course of one academic year. Each phase involved interviews and content analysis of participant-generated documents. The first phase, which occurred prior to candidates' introduction to the KPTP was intended to help me understand how participants initially thought about teaching and teachers. The second phase of the inquiry was timed to capture tensions and responses in participants' initial interactions with the KPTP at the beginning of their student teaching semester. During the third phase, participants were asked to reflect back on their responses in the previous two phases, and to summarize their experience completing the KPTP. Finally, the fourth phase was designed as a formal member check.

Phase 1
During Phase 1, which took place during the fall semester of 2014, I sought to gain information about my participants' constructions of good teaching and teachers. The data sources in this phase were collected prior to participant exposure or interaction with the KPTP, which provided me with a baseline for comparison as I moved into the later phases of the inquiry. The information I gathered in this phase helped me to understand the issues and questions that I should explore in the following phases. Participants were asked to respond to the essay prompt, "From your perspective, what is good English teaching?" I collected and read these essays, performing a content analysis on them. The content analysis involved unitizing data from the essays and then coding each unit that made any sort of statement about good teaching with a descriptive code (Skrtic, 1985). This was the beginning of the codebook used throughout the project. Based upon this content analysis, I divided the cohort into four focus groups made up of participants who shared similar initial constructions of good teaching.
Next I interviewed each group to gain further information and elaboration about the constructions of good teaching that these participants held. These interviews lasted approximately one hour per group, and involved questions like, Tell me about the best teaching you think you've done, and highlight what specifically within that example qualifies it as good teaching and Tell me about the best teacher(s) you've had or seen. What were these teachers doing specifically that made them good teachers? I transcribed these interviews and then unitized and coded the data, using the codes developed in the first part of Phase 1 and adding new codes as necessary. Table 2 in the Findings section of this paper provides the complete code book from Phase 1 of the project, and the number of data units for each code.

Phase 2
Phase 2, which took place at the beginning of spring semester of 2015, involved analysis of participants' initial interactions with the KPTP. During this phase, I performed a content analysis of participant KPTP tasks 1 and 2 looking for elements relating to participant constructions of good teaching. I then compared the constructions emerging from tasks 1 and 2 with the initial constructions established in phase 1 to determine whether changes in construction were occurring. Finally, I interviewed participants individually using tasks 1 and 2 as stimulus documents to gain further information and elaboration about emerging changes or tensions. Individual interviews lasted approximately 45 minutes, and included questions like In our previous interview, you described

Phase 3
Phase 3 took place toward the end of the spring 2015 semester, as participants finished their KPTP submissions. First I performed a content analysis of KPTP tasks 3 and 4 and compared constructions of good teaching emerging from these documents with those established in phases 1 and 2, to determine whether changes in construction were occurring. Then I interviewed participants individually using tasks 3 and 4 as stimulus documents to gain further information or elaboration about these changes and tensions. Individual interviews lasted approximately 45 minutes, and included questions like Where in your KPTP is your best teaching reflected? and Did the need to do the KPTP change how you taught in your field placement?

Phase 4
Phase 4 took place at the very end of spring semester 2015. At this point, I compiled a case report, which I then provided to a member check group drawn from the ranks of the original participants, an essential component for the trustworthiness of my study (Erlandson et al., 1993;Lincoln & Guba, 1985;Skrtic, 1985). Five of the thirteen participants elected to participate in the member check. After these participants had read the report, they were brought together to discuss whether the constructions represented in the case report matched their own constructions of the situation. Following the member check, I then revised the case report based upon the data gathered during the member check itself.

Data Analysis
Data collection occurred in four phases over the course of one academic year. Each phase involved interviews and content analysis of participant-generated documents. The first phase, which occurred prior to candidates' introduction to the KPTP was intended to help me understand how participants initially thought about teaching and teachers.

Inductive Analysis
Initial data analysis for this study was inductive rather than deductive (Lincoln & Guba, 1985). My primary analysis processes were constant comparison and open-coding, allowing themes to emerge from the data (Glaser & Strauss, 1967). This process allowed me to delve into the many interconnected layers of participant constructions of good teaching and understandings of the KPTP without initially imposing pre-determined categories.

Deductive Analysis
Once themes and categories were established through open coding in the initial inductive analysis, those themes were then analyzed deductively through the lenses of scholarship on definitions of good teaching (Patton, 2002). In particular, as participant constructions of good teaching became increasingly complex in the later phases of inquiry, the multifaceted framework of good teaching exemplified by the InTASC standards (Council of Chief State School Officers, 2013) became a useful tool for organizing and interrogating participant constructions of good teaching. Since the InTASC framework forms the construction of good teaching that guided development of the KPTP, comparing participant articulations of good teaching with categories of the InTASC framework helped to determine the degree to which the process of completing the KPTP shaped participant constructions of good teaching.

Trustworthiness
In order to ensure that my research was both rigorous and ethical, this study followed the guidelines for meeting trustworthiness criteria established by Lincoln & Guba (1985): credibility, transferability, dependability, and confirmability. Credibility refers to the believability of the study, which is demonstrated via persistent observation, triangulation, and in-process and summative member checks. Persistent observation means "identify[ing] those characteristics and elements in the situation that are most relevant to the problem or issue being pursued and focusing on them in detail" (Lincoln & Guba, 1985, p. 304). Constant comparison and emergent design allowed me to meet this criterion. I triangulated my data collection by collecting at a variety of points in time (the four phases previously described) and by collecting through various methods and sources (group interviews, individual interviews, documents). Member checks were built into the study throughout, and, as detailed above, the final phase of the study was formal summative member check.
Transferability refers to whether or not the readers of a study are able to determine the degree to which the results are applicable to their own contexts. Purposeful sampling, as described earlier in this section, and detailed description allow readers to make transferability judgements. Dependability denotes the consistency by which a researcher follows established research practices within the naturalistic paradigm. This can be demonstrated by the keeping of complete records of research procedures, referred to as a dependability audit trail and the review of those records by an external party. Lastly, confirmability signifies that the interpretations made by the researcher can be linked directly to the raw data collected in the study. Like dependability, this is best demonstrated by a review of interpretation and data by an external party, a process referred to as a confirmability audit.
To meet the requirements of dependability and confirmability audits, I employed the peer debriefer audit recommended by Erlandson et al. (1993). A peer debriefer is someone who is of the same academic rank and background, with whom the researcher can meet periodically throughout the research process to discuss emerging analysis and design, and who can give constructive criticism at a variety of points in the process, offering alternate perspectives on data and serving as a "devil's advocate" (Lincoln & Guba, 1985, p. 309). I recruited a peer debriefer, of similar academic rank, familiar with the teacher preparation program at Center University, but not with the particular participants in this study, and who had training in qualitative research generally and naturalistic inquiry specifically. We met on a monthly basis to discuss my research process, analysis, and decision-making, and both of us kept "written records of each encounter, partly for the sake of the audit trail…and partly for reference by [me] as [I] later [sought] to establish just why the inquiry emerged as it did" (Lincoln & Guba, 1985, p. 309). This continuous record of reflection and decision helped to ensure that each step of the emerging design was documented and rationalized, and allowed my peer debriefer to audit and question my interpretations.

Findings
The following section details the changes in participant conceptualizations of good teaching, both because and in spite of their experiences with the KPTP. Confirming what Chong (2011) argued, participants were entering their field experiences "with clear images of what teaching entails and how they [saw] themselves as teachers" (p. 220). However, this inquiry revealed that completing the KPTP was having some impact upon participant conceptions of good teaching, prodding them to broaden their understanding of the work of teachers to include not just dispositional and relational aspects of teaching, but elements of technical teaching practice as well. While early participant responses centered on the dispositional and emotional work of teaching, teacher knowledge of students, and the ability to use that knowledge to motivate and to form trusting relationships, as the pre-service teachers moved through the process of completing their KPTPs, their descriptions of good teaching became more complex. As one participant described, the KPTP influenced her by making transparent "how much work it takes to do it [teaching] and every little thing that goes into teaching and… It made me reflect on how I do all [these different things], but I didn't realize that I did [them]. That was kind of cool." Her comment indicates that through the process of completing the KPTP, she was able to view teaching through a different lens than she had before, a lens that was more analytic than intuitive, more atomistic than holistic.
To a greater or lesser degree, almost all participants experienced a similar shift in awareness, with corresponding implications for their conceptions of good teaching. In the initial phase of this inquiry, participants had been confident in their abilities to distill good teaching into a relatively small number of dispositional traits, like being playful and caring, and interpersonal skills, like empathizing and nurturing. By the later phases, however, good teaching seemed much more complex to them. As one participant said, "I think there are a lot of things that go into being a good teacher, and you kind of have to have all of them [to be a good teacher]." The sections that follow are organized based around the common themes that demonstrate the evolution of participant understandings of good teaching throughout the study. The first theme, personal teaching, dominated participant descriptions of good teaching in the early phases of the inquiry. As the study progressed, participants increasingly emphasized the second theme, technical teaching, in their documents and interview responses. The third theme, flexibility and adaptability, represents the understanding of good teaching that a few participants seemed to have constructed by the end of the study in order to reconcile their dedication to personal teaching with their increasing appreciation for technical teaching. Within the sections on each theme, salient quotes from participants are included to illustrate particular elements of the themes. Each quote was selected as representative of ideas voiced by multiple participants.

Personal Teaching
Relevant, fun, and caring-these are the words that participants used most often to describe good teaching, forming the first emergent theme in this inquiry, that of good teaching as personal teaching. At the beginning of the study, prior to interaction with the KPTP, as a whole, the cohort seemed to agree that the primary characteristics of good teaching were engaging students in fun and interesting experiences, establishing relevance of content and activities, and developing personal connections and relationships with students. Table 2 below details all codes relating to definitions of good teaching from Phase 1, the number of data units relating to each code, and the number of distinct participants whose construction of good teaching included material labeled with that code. As Table 2 makes clear, the three top codes, all key elements of personal teaching, were each between two and three times more common than the most frequent code relating to technical teaching. All but one of the participants described establishing relevance as an essential element of good teaching at some point in Phase 1, versus only half that number for the top technical teaching code.
Early on, one participant explained that good teaching is "about making sure your students are engaged in the lessons, and that you choose materials that will interest your students….In my [best] lesson [what made it good] was that everyone got out of their seats and everyone participated and was involved, mostly because it was fun." Another participant praised one of her former teachers who "used to wear like hats for different subjects…like, this is going to require critical thinking, so I'm going to put on my hard hat…it's like kinda cool. You know, like, teachers wear many hats…it was fun." The majority of participants at some point identified fun as an essential quality of good teaching, particularly at the beginning of the inquiry.
Along with fun, relevance was an important quality of good teaching for these participants. According to one participant, "An effective English teacher makes the literature and literacies relevant and applicable to students' past, present, and future lives." Another agreed, stating "it is imperative that English teachers make reading and writing relevant to students." One of the first important lessons that one participant felt she had learned in her pre-student teaching field experiences was that it was necessary "to try and get my students thinking about how issues in the novel might match up to issues in the world we live in today. Once my students were able to see how the novel could still connect to their own lives, they became more interested in hearing what I had to teach them." Participants argued that learning could not happen unless students were able to connect to the material, and that good teachers knew their students well enough to help them make those connections.
In order to know students well enough to facilitate connections, these participants believed that good teachers must truly care about their students. Throughout the study, participants used the words care and connections to describe the best teachers they had observed and the best teaching they personally had done. In her Phase 1 interview, one participant told the following story to illustrate this quality: My homeroom teacher, I was a competitive dancer, and we were somehow in the same city over a weekend. She had like a wedding and I had a competition, and she took time out of her weekend to come and watch me, and she took pictures and put them on the board…It was really cool.
Another participant at the beginning of the study simply stated that the most important part of a "teacher's job [is] to make meaningful connections with students." A third participant indicated that the best teacher she had ever observed "was always going out of her way to make sure that there was a personal connection" between her and her students. Across the board, these participants argued that good teachers were able ensure that students, in the words of one participant, did not feel like "little boxes to be checked."

Technical Teaching
As the study progressed, a second theme, technical teaching, emerged from participant discussions of good teaching. In particular, participants began to emphasize the importance of backwards design, detailed planning, and classroom assessment. Table 3 below details all codes relating to definitions of good teaching from unitized data gathered in Phases 2 and 3, the number of data units relating to each code, and the number of distinct participants whose construction of good teaching included material labeled with that code.  Technical  4  3  Multiple modalities  Either  3  3  Graphic organizers  Either  3  2  Involving all students  Either  2  2  Test preparation  Technical  2  1  Discipline/authority  Technical  1  1  Self-discovery  Personal  1  1  Clarity  Technical  1  1  Coverage  Technical  1  1  Content expertise  Technical  1  1 Note: Top three codes by frequency from Phase 1 of the study are represented with bold-face to highlight shift in relative emphasis over the course of the inquiry.
While Table 3 makes clear that participants continued to believe that good teaching had to be fun and engaging, with data units coded to that description being almost three times more common than the next most frequent code, the other two elements of personal teaching emphasized in Phase 1 shifted in relative emphasis. In contrast, the assessment cycle, only mentioned by a single participant in Phase 1, was the second most frequent code by Phase 3, with all but one participant indicating that a cycle of assessment was essential to good teaching.

Backwards Design
In Phases 2 and 3, participants focused heavily upon a cycle of establishing objectives, creating assessments to measure those objectives, planning instruction to teach those objectives, delivering that instruction, administering the assessments, and then determining the next set of objectives based upon the data gathered from the assessment, very similar to the process of instructional design or backwards design described by Wiggins & McTighe (2005). In Phase 1, however, only a single participant emphasized the importance of assessment within her description of good teaching. She described participating in a unit in her practicum placement, the field experience that pre-service teachers at Center University complete during the semester prior to student teaching, where she and her cooperating teacher analyzed student writing samples at the beginning of the unit to determine specific communication needs and then "created writing, reading, and speaking assignments that could augment what they needed to work on." Task 2 of the KPTP, however, requires pre-service teachers to respond to prompts and design their lessons within a framework of pre-assessment, instruction, and post-assessment. One participant explained this cycle in the following way: "you've got to start with where you want them to end up. Which is another way of thinking about how everything you do should be working toward the product or assessment you want them to be able to perform on at the end of the unit." A number of participants indicated that they had not previously conceptualized the work of teaching in this cyclic way. Another participant said the KPTP helped her to plan a whole unit and really do a lot of backwards design, considering standards and the needs of your students…it just really makes you think through very thoroughly all of those little details that should be considered in a unit with your students in the content.
A third participant noted that the KPTP "made you lay out the objectives; made you lay out the preassessment; made you lay out everything bit by bit by bit," and that this was a more systematic process of thinking about instruction than she would probably have considered prior to completing the KPTP. A different participant explained that the KPTP reinforced for her the importance of "always matching up the objectives to the standards to that summative assessment and then building everything in between…[because students] will get something from [the lesson] if we are not [planning instruction that way], but they will get so much more from it if we are really following that formula." Another participant indicated that the KPTP "has shown me that in order to plan, whether I am doing a 50 page plan for a unit or not, in order to plan effectively and teach effectively you do have to put a lot of thought into it" before getting up in front of the class or even selecting the activities that will be enacted.

Detailed Planning
The KPTP definitely played a role in developing participant understanding of and belief in the importance of detailed instructional planning. When asked where in the KPTP they were able to demonstrate good teaching, participants almost invariably pointed to the lesson plans themselves as providing the best opportunity within the KPTP to demonstrate what they considered good teaching. One participant noted that the lesson plans were "probably the one part where you could see that I know kind of what I'm doing" and that "Being organized and planned" was essential to good teaching. Another agreed, saying, I really did like Task 2, the lesson planning, where you had to lay it all out and…what standards applied to it….I think that definitely shows good teaching, as far as keeping your formative assessments, summative assessments for every single lesson that you do tied to your standard.
A third participant echoed these thoughts, noting that the KPTP really helped her to see the importance of "making sure that [the lesson] is tied to standards from which you derive main objectives" and then "planning with the assessment in mind…[by] thinking of the end of your lesson or unit before you plan anything, [and asking yourself] 'what do you want the outcome [of the lesson] to be?'" These comments indicate a growing awareness of a conception good teaching discussed in the literature as organized, sequenced, and systematic (e.g. Hunter & Russell, 1981).

Assessment
Participants also emphasized the KPTP's role in helping them explore assessment as a larger process including pre-assessments, formative assessments, and summative assessments. One participant noted that the KPTP "got me thinking about assessment more just because that was so prominent in the [KPTP]…I already formatively assess[ed] a lot in my classroom, but pre-assessing, I really didn't think about it until the [KPTP]….I will definitely pre-assess…more [in the future] and continue formative assessment and summative assessment." Another agreed, saying "I am a little more cognizant of being intentional with assessment…because it is so emphasized in the [KPTP]…it sort of reaffirmed that you have to give multiple assessments, which you work from [and] learn from. [You need to] view them as assessments of your own teaching and [as well as] what kids are learning." A third participant came to realize the importance of planning and establishing "clear expectations [for assessments] because when I didn't have those you could tell, and the kids were like, 'I don't know what I'm doing.' So, clear expectations-having a rubric or telling them what you are going to be grading something with" is an important part of good teaching.

Flexibility and Adaptability
The final theme of flexibility and adaptability emerged as a means for a few participants to reconcile their conviction in the importance of the first theme, personal teaching, with their growing awareness of the significance of the second theme, technical teaching. Good teaching, then, according to these participants, meant not just having a solid plan of instruction and assessment, but possessing the skills and habits of mind necessary to change that plan in real time depending upon student responses and needs. Toward the end of the study, one noted that good teaching was "being able to adapt, being flexible." Another agreed, saying, "[Good teaching is] so much improvisation, really. You never know what students are going to say, or what direction you go. I think being able to think on your feet is definitely important. That was one of the components of good teaching. I think the ability to do that and to be flexible is important." The two skills of writing concrete plans and of adapting instruction "on the fly" might seem to be unrelated, but for these participants, there was a definite connection. One participant explained this connection, saying "I took spring break to plan through the end of my time. I think doing that helped me adjust what I needed change on the fly because of assemblies and things like that. I was able to get rid of this and do this instead. Instead of scrambling last minute." Another argued "the necessity of being flexible in instruction. I think that when I am putting it down on paper, [I] really do see how much things [may] need to change [depending on student need]." Later in the interview she reiterated that good teaching really did require "that flexibility-having a really specific plan, but having that specific and structured plan for the purpose of being able to change it in live time if you need to [for your students]." In this sense, flexibility and adaptability seemed to be key attributes that allowed participants to form a bridge between their initial conceptions of good teaching as relevant, fun, and caring, and their new appreciation for the technical skills of planning instruction and assessment that are so prevalent within the KPTP.

Discussion
This section highlights the shift in participant conceptualization of teaching as a result of interaction with the KPTP, which I argue contributes and advances the dialogue around the role of HSTPAs in teacher education. During the first phase of the study, participants had described good teaching as primarily an interpersonal activity, highlighting the development of teacher-student relationships and teacher knowledge of students. These more technical elements of teaching, while missing from Phase 1 discussions of good teaching, rivaled the more personal elements of relevance, fun, and care in Phase 2 and 3 discussions.
It is important to note that the only shared experiences that all participants had between Phase 1 and Phases 2 and 3 were beginning student teaching and completing their KPTPs. They had finished their university coursework by the end of Phase 1, so they encountered no new emphasis on technical teaching or backward design there. Working just with their mentor teachers during student teaching also cannot explain this observed shift in participant emphasis, as the planning processes of mentor teachers varied significantly from participant to participant. Some mentor teachers developed detailed lesson plans with associated assessments, while others just moved methodically through district-adopted textbooks and pre-packaged curricula. Still others were satisfied with only a few notes about topic and activity jotted on a weekly calendar. Given such wide variation in student teaching experience, it seems likely that the common involvement with the KPTP was responsible for shifting participant recognition of elements like backward design that are essential to successful KPTP completion.
On the whole, the experience of completing the KPTP influenced this study's participants to shift the criterion by which they judged the efficacy of their practice. Prior to undertaking the KPTP, participants judged the efficacy of teaching by the quality of the relationships produced within the practice-relationships between the teacher and the student and between the students and the content. Care was important because it positively impacted the relationship between student and teacher. Relevance was important because it positively impacted the relationship between the student and the subject matter. Fun was important because it positively impacted both relationships. The relationships were themselves important because of their potential to spark transformative experiences (Jackson, 1986). Working with the KPTP, however, shifted the criterion that participants referenced when judging good teaching, from transformative relationships to direct gains in student learning.
These results suggest that supporters of HSTPAs may be correct that the act of completing such an assessment can indeed serve as a learning experience to help candidates internalize a common vision of teaching (Butler, 2015;Darling-Hammond & Hyler, 2013;Hollins, 2012;Stillman et al., 2015). The results of this study also suggest that critics of HSPTAs may be correct-the act of completing high-stakes, task-based assessments centered upon analyzing the planning, assessing, and enacting of learning experiences may be encouraging a narrowing of candidate focus from the personal and relational to more technical, process-product approaches (Dover & Schultz, 2016;Madeloni & Gorlewski, 2013). The candidates in this study, all of whom to a greater or lesser extent began the study viewing good teaching as firmly rooted in Lefstein's (2005) personal perspective came to appreciate the opposing technical perspective. They credited the act of completing the KPTP as forcing them to undertake that intellectual work. This is not to say that any participants indicated that they enjoyed completing the KPTP. In general, they found the process onerous, and the task-based writing repetitive. The many separate prompts stymied their abilities to craft cohesive narratives, which left them feeling that their final products did not completely capture the nuances of their teaching practice. Despite developing a new appreciation for the technical aspects of teaching, they expressed frustration that the personal aspects they still saw as so very important were not allocated at least equal space within the mandated assessment structure. Only a few participants seemed to synthesize the personal teaching they brought to the KPTP with the technical teaching enshrined within the KPTP to form a new understanding of good teaching as flexible and adaptable.

Limitations
Since it was conceptualized as a naturalistic inquiry into the experiences of particular participants in a particular context, the study was never intended to be generalizable-though insights from the study may be transferable to other contexts (Skrtic, 1985). Results of this study should not be interpreted as being universally representative of the experiences of all pre-service teachers completing high-stakes teacher portfolio assessments. Rather, this study was intended to provide readers with access to the experiences of these 13 pre-service teachers at Center University as they attempted to navigate their final year of teacher preparation and the concurrent HSTPA. The participants in this study were all prospective English teachers, a discipline that previous scholars have recognized as being particularly concerned with affective dimensions of teaching (Hochstetler, 2011). The experiences of teacher candidates from other certification areas might very well be different. I invite readers to reflect critically and consider whether the findings of this study may provide insight into their own contexts.

Implications for Policy
State-specific HSTPAs are inherently more contextually responsive than their nationallystandardized counterparts. While the task architecture of the KPTP, for example, is similar to that of national HSTPAs, as described earlier in this paper, the procedures and processes for implementing and scoring it are determined by the Kansas Division of Learning Services rather than by an external testing company like ETS or Pearson. The state collaborates with university-based teacher licensure programs to recruit scorers familiar with the strengths and challenges of teaching in the different regions of Kansas. It sets a cut score based upon a target of proficiency negotiated with educational stakeholders across the state. Most importantly, the KPTP includes a process for remediation for those candidates who fail to meet the required level of proficiency (Nelson, 2016). Using the scores and justification statements provided by the KPTP scorers, teacher education faculty are allowed to meet with candidates whose submissions fail to meet the cut scores and establish a remediation plan for them. This plan can include the assignation of additional learning experiences prior to the rewriting of the previous KPTP or the development of an entirely new KPTP submission. The revised or new submission is then scored by trained KPTP scorers (Nelson, 2016). While nationallynormed HSTPAs allow resubmission of tasks, candidates are only provided with numerical scores, not commentary, and resubmission requires the payment of a significant fee.
Policymakers interested in mandating HSTPAs for teacher licensure may consider the results of this study when deciding whether to adopt a national assessment like the edTPA or PPAT or to develop a state or local exam like the KPTP. If the primary motivation of such policymakers is to ensure that a unified vision of competent teaching practice is codified and understood by teacher candidates prior to their reception of licensure, then this study suggests that a cheaper and more contextually responsive HSTPA may fulfil that purpose.

Implications for Practice
This study suggests that completing an HSTPA is a learning experience in and of itself. That learning experience centers on the understanding of good teaching upon which the HSTPA is based. Teacher educators may wish to consider the degree of compatibility between the view of good teaching that drives their particular educator preparation programs and the view of good teaching guiding any HSTPA that their candidates may be required to complete, in order to either leverage or mitigate that learning experience.

Implications for Research
Participant critiques of the instrument and the process of completing the assessment raise additional questions about whether the learning that candidates achieved was worth the time and frustration of completing the assessment, and whether that learning will result in lasting changes to teaching practice. Additionally, this study raises questions about what kind of identity tensions may have resulted for candidates as they undertook the heavy intellectual work of modifying deeply held conceptualizations of teaching during the process of completing a high-stakes gateway exam, the results of which would determine whether they were able to enter the profession at all.

Conclusion
Both supporters and critics of HSTPAs have focused on their potential to reify the conceptualization of teaching held by teacher candidates. Supporters see this potential as an advantage, arguing that a wide-spread unified vision of teaching is an essential element in the professionalization of the teaching workforce and the improvement of education nationwide. Critics also see HSTPAs as vehicles for defining teaching, but worry that the definitions encapsulated in HSTPAs like the edTPA and the KPTP are overly technical, ignoring what they see as equally or more important personal and political aspects of teaching. This study suggests that both camps may be at least partially correct. This research also suggests that, at least one of the goals of supporters of the edTPA, the development and diffusion of a unified vision of teaching with "student learning in the center of a three-step teaching cycle: planning, instruction, and assessment" (Sato, 2014) can be achieved with an HSTPA like the KPTP that is cheaper and more responsive to local contexts. Further research will be necessary to determine both the strength of and longevity of the learning experience created by the KPTP and the degree and impact of conceptual tension engendered by the KPTP. education policy analysis archives editorial board