“ Seeing the Whole Elephant ” : Changing Mindsets and Empowering Stakeholders to Meaningfully Manage Accountability and Improvement

School accountability and improvement policy are on the precipice of a paradigm shift. While the multiple-measure dashboard accountability approach holds great promise for promoting more meaningful learning opportunities for all students, our research indicates that this can come with substantial challenges in practice. We reflect upon the lessons learned from our recent research on the CORE districts’ use of multiple measure data dashboards. Our research indicated that a shift to greater flexibility and locally determined capacity building efforts brings its own set of challenges. Building on this foundation and drawing upon the expertise of a central practitioner-leader, we explore these issues through one promising case: the use of these multiple-measure accountability systems in one CORE district, Oakland Unified School District (OUSD). OUSD’s approach suggests that districts have substantial agency to help modify the mindsets of practitioners and the community, by modeling the values of inquiry, openness, and flexibility. By loosening the reigns, districts can give school stakeholders the space and authority to meaningfully manage their own accountability and improvement.


Seeing the Whole Elephant": Changing Mindsets and Empowering Stakeholders to Meaningfully Manage Accountability and Improvement
School accountability and improvement policy are on the precipice of a paradigm shift.Most importantly, the federal Every Student Succeeds Act (ESSA) requires a more comprehensive assessment of school performance and a less prescriptive, local approach to school support.While the multiple-measure, locally-led accountability approach holds great promise for promoting more meaningful learning opportunities for all students, our research indicates that this can come with substantial challenges in practice.We reflect upon the lessons learned from our recent research on the CORE districts' use of a multiple-measure accountability system (see Marsh, Bush-Mecenas, & Hough, 2016, 2017).Building on this foundation, we explore a promising case: the use of these multiple measures to drive school improvement in one CORE district, Oakland Unified School District.

Lessons from CORE
As discussed by Bae (2018), six California districts (Oakland, Fresno, Long Beach, Los Angeles, San Francisco, and Santa Ana) developed an innovative accountability system under a waiver from No Child Left Behind (NCLB) resting on three key tenets: 1) the importance of local control and local solutions to problems, 2) a focus on shared responsibility and support over compliance, and 3) the belief that districts can best build capacity through peer-to-peer collaboration focused on data.At present, the CORE districts (now including Sacramento City and Garden Grove) continue to work together to improve outcomes for the million students they serve, the majority of whom are minority students who are socioeconomically disadvantaged.Together, the CORE districts have created a unique school performance measurement system that includes holistic measures of academic growth, social-emotional learning, and school culture/climate.While no longer implementing their waiver since the passage of ESSA, the districts use this measurement system to inform and deepen California state's accountability and measurement system.Thus, the CORE districts present an opportunity to learn how to effectively utilize multiple measures of school quality, develop shared accountability, and grow capacity for schools and districts to improve.
In 2015, our team of researchers working with Policy Analysis for California Education (PACE) examined how administrators understood, implemented, and responded to CORE's new accountability system (see Marsh et al., 2016Marsh et al., , 2017)).Overwhelmingly, we found that district and school administrators greatly appreciated the more holistic approach to measurement, the emphasis on support over sanctions, and the transparency of the CORE approach.Further, they reported using measurement system results to inform decision-making around resource allocation, stakeholder communication, and improvement planning.However, despite these benefits of a multiple measure accountability system, the districts reported several key challenges: First, we found that the CORE system needed to be modified to suit local needs.While this may have allowed districts to create a more relevant and meaningful accountability system, it also revealed a tension between shared accountability and local variation.While many agreed that locally defined and relevant measures were valuable, some reported prioritizing high-stakes standardized measures given their limited resources and time.
Second, many in the districts were concerned with unintended consequences.Bae (2018) points to the value of transparency as a means of "incentiviz[ing] the behaviors we seek to encourage (e.g., a focus on a holistic view of student learning and development, the incorporation of noncognitive indicators) and safeguard [ing] against the behaviors we seek to discourage (e.g., a shortsighted focus on standardized test scores in two subject areas, the narrowing of the curriculum to tested subject areas)" (pp.21-2).Our research indicates that CORE's multiple-measure system still evoked strategic behaviors that might invalidate measurement and preclude learning, among some actors.For example, some district leaders reported that administrators asked them to share the "right" answers on social-emotional skills and school culture-climate surveys, in order to achieve a higher school rating.Others worried that the methods of reporting of student suspension were altered to lower suspension rates.Moving away from compliance mindsets towards a learning orientation proved challenging.
Finally, new measures take a long time to be fully understood and acted upon.While CORE's measurement system holds promise for improving local systems, efforts to grow capacity around the measurement system were still a work in progress.Administrators believed that additional capacity needed to be grown around understanding the new social-emotional learning and school culture measures, as well as how to actually develop and improve social-emotional learning and school culture.The following case presents insight into how districts can substantively shift mindsets to advance the intent of multiple-measure approaches.The quotes throughout this narrative come from our practitioner colleague and co-author David Montes de Oca, former Senior Deputy Chief of the Office of Post-Secondary Readiness for Oakland Unified and current Deputy Chief of Improvement for the CORE Districts.

Changing Mindsets in Oakland
Oakland Unified School District has embraced and expanded upon CORE's multiplemeasure accountability approach, including: broad outcome measures, a mix of state and local accountability, measuring Opportunities to Learn (OTL), data dashboards, and School Quality Reviews.Together, these measures, and the structures they are embedded within, are intended to help stakeholders in "seeing the whole elephant, taking the blindfolds off and seeing the schools."1OUSD worked to avoid the potential pitfalls identified by Bae (2018;e.g., low parent and community support, compliance-oriented staff responses, and inadequate capacity to address improvement needs): we ask, how?
In executing local school improvement, OUSD took a proactive approach to redefining the school's purpose, identifying highly localized, formative performance indicators in collaboration with all stakeholders, and responding to practitioner feedback on district hindrances to improvement.After exiting state receivership in 2009, leaders in Oakland took a bold move: to reimagine the district as a collection of full-service community schools committed to meeting the academic, social-emotional, and physical health needs of students and the community.Leaders met with about 400 stakeholders for over a year to understand how to measure quality in full-service community schools using explicit quality standards, measured in qualitative and quantitative ways.Engaging focus groups of students, parents, and educators, OUSD defined four values: 1) attention to and responsibility for the whole child, using multiple measures for a comprehensive view of how a child is doing; 2) impact, represented by calculating growth on every measure (including surveys, chronic absence, etc.); 3) equity, ensured by weighting groups of students, such as identifying the lowest-performing racial/ethnic group as an indicator across most metrics; and 4) college, career, and community readiness, represented by the linkages between measures and ultimate student readiness for life after high school.
This regular stakeholder engagement certainly meets Bae's recommendation of meaningful stakeholder engagement, and likely enhanced buy-in and commitment to the reform.To break down the persistent compliance orientation of school staff, OUSD then began pivoting from accountability as a "hammer and feeling particularly judged" toward using measurement as a "flashlight" to promote "learning, improvement, growth and the idea that we're constantly seeking to get better."Specifically, OUSD meaningfully enhanced the annual "snapshot" measurement structure with more regular, formative metrics, while allowing schools to design their own indicators to use for engagement in cycles of inquiry and improvement that involved "testing" ideas.
They also challenged school leadership teams to identify barriers to achieving their improvement goals.At times, this meant challenging district policy and practices.For example, school leadership teams identified the frequency and scheduling of certain assessments as a hindrance to their improvement efforts.As a result, district administrators modified the assessment calendaring process in response to this feedback.Together, these actions represented a substantial shift in mindsets for school staff: from top-down accountability metrics to locally defined formative indicators, from set-in-stone initiatives to trying out potential solutions, from being restricted by district policy and practice to being empowered to make the district work for schools.
Of course, OUSD faces ongoing challenges in managing accountability and building capacity for improvement, which are likely to test all states and districts embracing multiple-measure approaches to accountability and support.Essentially, all such districts face a balancing act, addressing inherent challenges in the system: the tension between accountability and continuous improvement; the candor to admit that few individuals in the school system possesses a core understanding of equity and how to promote equitable outcomes; and the continual sense of "discomfort, trepidation, and risk" among stakeholders that is necessary to motivate continuous improvement.The aspiration is to distribute these challenges and the weight of this heavy lift across "many hands and hearts." Oakland's approach suggests that districts have substantial power to help modify the mindsets of practitioners and the community, but this involves modeling the values of inquiry, openness, and flexibility.By loosening the reigns, districts can give school stakeholders the space and authority to meaningfully manage their own accountability and improvement, to give them "empowered agency about being able to do something about it right now and every day."