District-Wide Effects on Data Use in the Classroom

In the present study, an examination is conducted in three school districts of how data are used to improve classroom practice. In doing so, we explore the effects that attitudes toward data, principal leadership, and computer data systems have on how data are used to affect classroom practice. Findings indicate that educators are ambivalent about data: they see how data could support classroom practice, but their data use operates in the presence of numerous barriers. Many of these barriers are due to principal leadership and computer data systems; these barriers often have negative effects on attitudes toward data and disrupt the progression from using data to inform classroom practice. It is hypothesized that many of these barriers can be removed through effective district policies to improve structures and supports for using data.


Introduction
During the last 10 years, the field of education has witnessed a substantial increase in studies which examine how educators may use student data to help improve their practice.This research has shown a number of factors that facilitate classroom data use, such as collaboration, principal leadership, personnel supports, and effective technology (e.g., Anderson, Leithwood, & Strauss, 2010;Wayman & Stringfield, 2006;Copland, 2003;Lachat & Smith, 2005;Marsh, McCombs, & Martorell, 2010).
Much of the research on educational data use has focused on case studies and empirical descriptions.Thus, there are few established causal links between student achievement and the use of data (Hamilton, Halverson, Jackson, Mandinach, Supovitz & Wayman, 2009).This notwithstanding, there is reason to believe that the effective use of data may improve schooling.For instance, the effective use of data is often cited as part of more general school improvement initiatives (Ingram, Louis, & Schroeder, 2004;Stringfield & Datnow, 2002), teachers often report changes in practice based on data use (Wayman & Stringfield, 2006;Datnow, Park, & Wohlstetter, 2007), and studies are emerging that statistically correlate student achievement to interim assessment administration (Wayman, Shaw & Cho, 2011;Carlson, Borman, & Robinson, 2011;May & Robinson, 2007).
Thus, there remains a need for further study of how data use at all levels of a district affect classroom practice, and in contexts not chosen for their proclivity in using data.The present study responds to that need through an examination of three districts of differing size and context, none of which were chosen for their excellence in using data.Accordingly, the goal of this study is to examine how practices at every level of the district affect the use of data in the classroom.We focus our research questions on four areas that prior research suggests may be particularly important: (1) How do educators commonly use data?(2) What are educators' attitudes toward using data?(3) How do principals lead faculty in using data?(4) How well do computer data systems support educators in using data?

Research on Educational Data Use
Educational research has noted the wide variety of factors that influence educational data use.In the present study, we examine four areas that arise frequently in research: attitudes toward data, principal leadership, and computer data systems.In this section, we provide a brief overview of research in each of these four areas, followed by a model that conceptualizes how these factors work together to promote effective data use.
Educators' use of data.Research on educational data use notes that uses vary by role, but has focused primarily on how teachers and principals use data.For instance, research in exemplary settings has shown teachers using a variety of student-level data to group students, re-group students, and adjust instruction based on data (Wayman & Stringfield, 2006;Datnow et al., 2007;Lachat & Smith, 2005).Research in exemplary settings has shown principals using both student-and building-level data to make policy decisions and support faculty (Copland, 2003;Datnow et al., 2007).Research has provided little detail on data uses of central office staff and instructional coaches, with the exception of studies such as Honig and Coburn (2008) and Marsh et al., (2010).
Educators' attitudes toward data.Educators in rich data-using contexts often report that educators have positive attitudes toward data when supported by a culture of data use (Wayman & Stringfield, 2006;Copland, 2003;Datnow et al., 2007;Lachat & Smith, 2005;Knapp et al., 2006).Educators in these studies typically believed that using data helped them improve practice and resulted in improvements in student learning.Educators in other contexts often display more tempered attitudes.For instance, teachers have sometimes been shown to be suspicious of data initiatives, often separating data use from their own judgment (Ingram et al., 2004;Valli & Buese, 2007;Young, 2006).Educators in all roles are sometimes hesitant to use data for fear that it will require a large time investment and little practical return (Wayman, et al., 2009a).
Principal leadership.Successful school-based data initiatives are almost always marked by principals who are employing practices such as setting clear expectations for data use, involving entire faculties, and making time for teachers to use data (Wayman & Stringfield, 2006;Copland, 2003;Datnow et al., 2007;Halverson, Prichett, & Watson, 2007;Knapp, Swinnerton, Copland, & Monpas-Huber, 2006;Supovitz & Klein, 2003).On the other hand, studies of contexts not known for data use describe a wide range of principal involvement (Anderson et al., 2010;Wayman et al., 2007;Young, 2006) -some principals in these studies were effectively leading faculty in data use, but most were not.Those who were not were often characterized by these studies as either disinterested in data use or lacking a set of strategies that could foster effective faculty data use.
Computer data systems.Nearly every district in the United States has some form of computer system for managing student data (Means, Padilla, DeBarger, & Bakia, 2010).When implemented effectively, these systems have been shown to facilitate many facets of educator data use, such as collaboration and rapid turnaround of data (Wayman & Stringfield, 2006;Lachat & Smith, 2005;Long, Rivas, Light, & Mandinach, 2008).Other research chronicles the struggles of educators unsupported by effective data systems.These studies have shown computer systems often lack integration, are inefficient, and are hard to use (Means et al., 2010;Wayman et al., 2007;Wayman et al., 2009a).
Conceptual framework.Many scholars have presented models to describe how the use of data may improve education (e.g., Copland, 2003;Mandinach, Honey, Light, & Brunner, 2008;Supovitz, 2010).While the details of these models differ, they all share the same core logic: data use provides information that educators may employ to change practice.
The research base on educational data use shows mixed results in the application of this model.While some studies show this progression to happen successfully (Wayman & Stringfield, 2006;Datnow et al., 2007;Lachat & Smith, 2005), others show that inappropriate use of data has actually hindered educational practice (Wayman, et al., 2009a;Vasquez Heilig & Darling-Hammond, 2008;Valli & Buese, 2007).
Prior knowledge, information, and other elements may influence how certain data are noticed, prioritized, and used (Alavi & Leidner, 2001;Coburn, Honig, & Stein, 2009;Tuomi, 1999).Thus, we posit that the progression from data to knowledge to practice is influenced by a variety of elements.The present study focused on three such elements: attitudes toward data, principal leadership, and computer data systems.Figure 1 provides a graphic representation of our logic: In this conceptualization, educator use of data provides information that influences classroom practice.However, educator uses of data are influenced by an educator's attitude toward data, principal leadership, and the access they gain to data through data systems.It is important to distinguish data from information.Data are the raw inputs (e.g., student test scores or teacher observations) that educators may access about their students; information is the processed outcome of these data.Thus, educators access data from their computer data system and process it into information through their uses of data.This information is used to make changes to classroom practice.
While this is a building-level model, it is deliberately not role-specific, allowing for roles such as central office or instructional support to influence how practice occurs in the classroom.For example, how an instructional coach's uses of data influences teachers' classroom practice will be influenced by the coach's attitude toward data, the leadership of campus principal(s) and the data systems that provide access.It is also important to note that the model does not assume that data use leads to improved practice -instead, the model allows for the possibility that these or other influences might result in data use that leads to more or less effective practice.

Method Introduction
The present study is drawn from a larger three-year project designed to help three school districts improve their use of data by employing a systemic focus called "the Data-Informed District (Wayman, 2010;Wayman et al., 2007).It is important to define three terms used in this project.First, we use a broad and encompassing definition of data, meaning anything that helps educators know more about their students (e.g., formal assessments, tests, quizzes, and student background data).Second, data use means the actions in which educators engage as they collect these data, organize and analyze them, and draw meaning from them to inform practice.Third, we often use the term effective data use to distinguish between data use practices that benefit educators in their practice (and which thus benefits student learning) from other data use practices that have been shown to actually hinder educational work (Wayman et al., 2007;Earl & Fullan, 2003;Valli & Buese, 2007;Young, 2006).
Data were collected in three districts in Texas1 during the 2009-2010 school year.Districts were not selected for their success at using data; in fact, district leaders volunteered for this study to improve their districts' data use.Boyer School District was a district of approximately 8,000 students that mostly served a non-Latino White population,2 less than five percent of who were economically disadvantaged.Gibson School District was a district of approximately 25,000 students of various ethnic backgrounds,3 half of whom were economically disadvantaged.Musial School District was a district of approximately 45,000 students of various ethnic backgrounds, 4 a third of whom were economically disadvantaged.Districts varied in their student achievement: in a typical year, the percent of students meeting standards on the state exam was consistently greater than 95% in Boyer, approximately 75% in Gibson, and approximately 85% in Musial.The overall state rate was typically about 82%.

Procedure
We employed mixed-methods in conducting this study.Phone and in-person interviews were conducted with individuals, site visits were made to schools to conduct educator focus groups, and a confidential online survey was made available to all educators in each study district.In the following sections, we describe our procedures for collecting the qualitative and quantitative samples.
Qualitative sample.Qualitative data were collected through individual interviews and focus groups.These were conducted using a semi-structured protocol that focused discussion on ways data were used and accessed, specific data systems employed, and wishes for future data use.All qualitative interviews were recorded and subsequently transcribed for analysis.
At the central office level, individuals were identified through a review of central office positions.This list then was discussed with our primary district contacts to ensure proper coverage.Additionally, many interviewees were asked to suggest other individuals to interview.Central office educators were interviewed by telephone or in person.
At the building level, teachers, principals, and other building staff participated through focus groups conducted during site visits to 19 campuses.These schools were chosen to be representative of each district in terms of size, location, and socioeconomic makeup (see Table 1).During school site visits, two focus groups were conducted in each school.One focus group consisted of the principal and individuals they chose who were familiar with data use in their school (e.g., assistant principals or instructional support staff).On the same day, a teacher focus group was conducted in that school.Teacher focus groups included three to five teachers who were selected by the principal from a randomly generated list of seven to nine teachers. 5High schools had many more teachers on staff than other schools; to ensure that we fully understood data use in the high schools, we conducted two teacher focus groups at those sites.The qualitative sample consisted of 197 total participants.Table 2 provides a description of this sample, disaggregated by educational role and district.(Wayman, Cho, & Shaw, 2009b), a 67-item instrument assessing a variety of factors, including attitudes toward data use, support for data use, instructional practices, technology, and specific ways in which data were used by the respondent.The survey was given online and made available to all educators throughout each district.Participants were not allowed to leave any items blank, so there were no missing survey data.Survey response rates were 50% in Boyer, 62% in Gibson, and 41% in Musial.The quantitative sample consisted of 3,101 individuals across the three districts.Table 2 provides a description of this sample, disaggregated by educational role and district.

Measures
Comparison categories.Educational role and district experience were each used to compare educators on aspects of data use.We defined four roles to be used for comparison: (1) campus administrators (principals and assistant principals), (2) central office staff, (3) instructional support staff (campus staff such as counselors, school psychologists, and instructional coaches), and (4) teachers.On the survey, district experience was collapsed into four categories: (1) 5 years or less, (2) 6 -11 years, (3) 11 -20 years, and (4) 20 years or more.
Data use questions.Selected survey items were singled out to help describe how district educators used data.One block of 14 questions asked how often participants engaged in specific data uses, such as identifying individual students who need remedial assistance, setting school improvement goals, and evaluating district achievement trends and performance.These items were set on a four-point Likert scale with the following options: less than once a month, once or twice a month, weekly or almost weekly, and a few times a week.Each response option was numbered 1 -4, with 1 corresponding to less than once a month.
Another block of questions asked how often participants used specific computer systems in their district.These items were set on the same four-point Likert scale as the questions above.
Four scales measuring attitudes and uses of data were formed from groups of survey items.The individual survey items within these groups asked how much the respondent agreed with a statement, offering the following options: strongly disagree, somewhat disagree, somewhat agree, and strongly agree.Each response option was numbered 1 -4, with 1 corresponding to strongly disagree.To create each scale, responses were averaged across the group of items in that scale.Scales thus ranged from one to four.These scales have been shown to be valid and reliable in other samples (Wayman et al., 2007;Wayman et al., 2009a).
The Attitudes Toward Data scale was a four-item scale that asked participants whether they liked data, found it useful, and whether it helped them.The alpha reliability of this scale ranged from 0.838 to 0.8936 .The Data's Effectiveness for Pedagogy scale consisted of five items that asked about the contributions that data can make for improving educational practice (e.g., helping to plan instruction, reveal new insights, or identify learning goals).The alpha reliability of this scale ranged from 0.930 to 0.949.The Principal Leadership scale consisted of five items that asked how much participants agreed that their principal or assistant principal(s) supported data use (e.g., encouraged it as a tool to support teaching, made training available, were examples of effective data users).The alpha reliability of this scale ranged from 0.909 to 0.910.The Computer Data Systems scale consisted of four items asking about computer systems, such as whether the participant's systems were easy to use or provided ample data.The alpha reliability of this scale ranged from 0.943 to 0.951

Analyses
Qualitative analyses followed methodology suggested by Miles and Huberman (1984).Drawing upon prior research on educational data use, an a priori list of potential analytic themes was generated.As qualitative data collection progressed, these themes were updated and refined during research team meetings.This collaborative and inductive process resulted in a conceptually coherent set of themes that was used for coding interviews and focus groups.The research team used this set of themes to code participant responses.Themes were examined by role and school level to identify emergent patterns and explanations regarding educator data use.
Quantitative analyses were conducted as follows: for the block of 14 questions that asked about uses of data, mean responses were ranked for each role.This produced a ranked list of the frequency of data use for each of the 14 questions.For the four survey scales, ANOVAs were used to compare means responses by role.Significance was set at the 0.05 level and effect sizes (partial eta-squared) were computed.When role was statistically significant, Tukey post-hoc tests were performed to identify significant mean differences.Due to space restrictions, simultaneous confidence intervals for mean differences were computed but are not presented in this paper.For the questions that asked about frequency of computer use, we classified each system as an assessment system, student information system, data warehouse, or "other" system and dichotomized responses into weekly use or less.Percentages of weekly use were presented by role for each type of system.

Results
Besides differences in demographics, size, and economic makeup, our districts also presented diverse contexts in the ways data were approached and used.In the Boyer school districts, "data use" was viewed by most educators as the examination of state test data.Since almost every Boyer student passed the state test, many educators believed the state tests were irrelevant to their work -and thus, so was data use.Consequently, efforts to use data in Boyer were confined to small groups of interested educators.The district was attempting to implement some procedures around formative assessments, but was having trouble building momentum.
In Gibson, state test performance was strongly emphasized, but as part of a larger, curriculum-based initiative.Curriculum was divided into segments and a locally-built benchmark examination was given to students at the end of each segment.These examinations were tied to curriculum but were also intended to prepare students for performance on state tests.
In Musial, there was a very strong focus on state test performance.District leadership used a variety of ways to communicate the importance of state test performance.Musial also implemented a set of locally-built assessments tied to state exams.Teachers were expected to examine these periodic assessments to improve instruction, with an eye toward performance on state tests.Musial had recently hired a central office administrator whose role was to support data use throughout the district with a strong focus on working with building-level educators on using data to improve instruction.
In the following, we provide analysis of how these districts used data in their varying contexts.Sections are provided corresponding to our four research questions: (1) uses of data, (2) attitudes toward data, (3) principal leadership for data use, and (4) computer data systems.

Uses of Data
The ways data were used varied by role in all three districts.We use the following sections to provide descriptions of how data were used in four roles: central office educators, instructional support specialists, campus administrators, and teachers.
Central office educators. 7Central office educators, in line with their responsibilities for large numbers of students and teachers, tended to use data for monitoring district and campus progress, providing feedback to campus personnel, and in support of broad-scale campus efforts.Many of these uses were centered on attempts to help building-level educators improve practice.For example, Gibson and Musial central office educators provided feedback to campuses about particular goals; campus personnel were expected to use this feedback to make adjustments in areas identified for improvement.As another example, central office educators in Boyer performed item analysis on behalf of teachers and helped inform departments and teachers about academic areas in which various grade levels needed attention.
Instructional support specialists.Despite a variety of titles and intended responsibilities, the ways that instructional support specialists used data were consistent across the three districts.Persons in instructional support roles indicated that they used data in three main ways.First, they used data to identify and help teachers address the needs of individual students.Tables 3 through 5 show that instructional support personnel in both Musial and Boyer ranked the use of data to identify student learning first (M=2.91 and M=3.14, respectively); in Gibson, it was a close second (M=2.70).Second, instructional support personnel used data to collaborate with teachers.Interview data revealed that instructional support personnel used a variety of data (e.g., primary reading data, math inventories) to aid teachers in forming small groups of students or to help teachers prepare lessons that target specific skills.Further, Tables 3 through 5 show that discussing student progress or instructional strategies ranked high in survey data: it was the highest-ranking survey variable in Gibson (M=2.74),second for Musial (M=2.81), and third for Boyer (M=2.95).
Third, interview data suggested that instructional support personnel used data to help teachers reflect on practice.This included both monitoring and diagnosing of aggregated groups, as well as intervention and support with individual teachers and students.
Campus administrators.In each district, our data showed that campus administrators often focused their data use on struggling students.This issue came up first and frequently in nearly every administrator focus group.On the survey, campus administrators ranked the use of data for identifying the needs of struggling students and for developing recommendations for intervention as the top two most frequent uses of data (see Tables 3 through 5).Administrators reported far less frequent use of data to identify the learning needs of students who were performing adequately or beyond, ranking this use last among all survey options in each district (see Tables 3 through 5).
In addition, administrators also reported some frequent non-student based issues to which they applied data.One such use involved teacher feedback and evaluation efforts: administrators in Gibson and Musial described classroom "walkthroughs" and how they collected and reported data from this process.Administrators in each study district also described using various data to gauge the fidelity of curriculum implementation (i.e., whether teachers were on schedule and assessing the rigor of their teaching strategies).
Teachers.Across the districts, teachers reported a variety of uses of data, including using data to help struggling students, group and regroup for instruction, reteach particular concepts and skills, and adjust instruction.Similar to administrators, teachers in interviews often focused on using data to support struggling students.This use also ranked first or second for surveyed teachers in each district (see Tables 3 through 5).Teacher comments less frequently focused on the needs of students who were performing adequately or who were excelling in the classroom.Such use of data ranked ninth among teachers in all three districts (see Tables 3 through 5).
Teachers also discussed various ways that they used data to change how or what they taught; Tables 3 through 5 show these items rank highly.In interviews, this often related to grouping, such as forming instructional groups of students or deciding what to teach to the entire group.Teacher comments in each district indicated that their attention to groups occurred infrequently during the year, such as at the start of quarters.Although surveyed teachers reported frequently using data to adjust instruction for individual students (see Tables 3 through 5), we heard little mention of this in focus groups.
We also found that teachers only rarely talked about using data as a centerpiece of meetings with instructional support personnel or with campus administrators.Such use of data ranked near the bottom for surveyed teachers in each district (see Tables 3 through 5).In focus groups, teachers either did not mention this type of data use or described it as happening infrequently.

Attitudes Toward Data
Survey and interview data revealed that participants were generally positive about data and its potential, even in the face of consistently present barriers.These barriers made many educators ambivalent about data -they saw value in data, but were hesitant because of hardships and other concerns.In the following two sections, we present results describing positive attitudes held by participants, followed by results that outline the barriers that temper these attitudes.
Positive attitudes.In each district, we found educators to be generally positive about data and what it could do for their practice.Participants in all roles averaged at least 3.00 on the Attitudes toward Data and Data's Effectiveness for Pedagogy scales, with some roles approaching the maximum of 4.00.In interviews, participants often spoke positively of data and their potential.
While positive, teachers often displayed more skepticism about data than did those in other roles.Table 6 shows that teachers' views were significantly different from those in other roles on the two scales.Table 7 shows that in each district, teachers ranked significantly lower on the two scales (excepting Musial central office administrators, who were similar to teachers on both scales).Campus administrators and instructional support personnel consistently ranked high, if not significantly so.In fact, campus administrators in Gibson and Musial averaged near the maximum on the Effectiveness scale.Educators in all roles showed slightly more optimism about the effectiveness of data than in their personal attitudes toward using it.Interview data captures the details of these positive attitudes, which differed by district.Educators in Gibson and Musial had greater exposure to data and showed attitudes more similar to each other than to Boyer educators.Throughout Gibson and Musial, participants who were positive about data saw it as a way to support professional judgment.For them, data contributed to instruction (e.g., providing feedback about individual students, lessons, programs, or learning issues) and was an important part of reflecting collaboratively about issues of mutual concern.A Musial teacher described this attitude thusly: "Say you administer a common assessment, you go to team meeting and talk about it: the strengths you are seeing, the weaknesses.'[We discuss] How can we change our instruction to make this concept more clear?'" In comparison, Boyer educators described their positive attitudes more vaguely than educators in Gibson and Musial.Boyer educators spoke generally, as evidenced by this teacher: "Working with another grade level teacher [on student data] before the school year is valuable.I get to learn what helps certain students."Further, Boyer administrators typically focused their positive attitudes not on themselves, but on benefits for other educators.For example, central office administrators valued data as a tool for teachers and campus administrators, but they rarely reported using data in their own work.Similarly, campus administrators were positive about data but focused on the work of teachers, not themselves.
Barriers tempering positive attitudes.The positive attitudes above were often tempered by day-today difficulties in using data, such as problems with computer systems, lack of time to reflect on data, and the labor-intensiveness of using data.In line with survey results, teachers in all three districts described these barriers with more negativity than did educators in other roles.In Gibson and Musial, where data were in more frequent use, educators described barriers in greater detail and breadth than did Boyer educators.
Teachers in Gibson and Musial expressed concerns about the role that data was playing in their district; these concerns were not seen in Boyer.For example, some Musial teachers felt that data were being used to inappropriately compare or encourage unnecessary competition among campuses.One said, "We are pressured to meet standards, pressured by the data to meet standards, absolutely.They make it clear that that's very published and very public." While teachers were most vocal about data barriers, non-teachers (campus administrators, central office educators, and instructional support educators) also shared these concerns.For example, non-teachers in Gibson were especially concerned about their lack of integrated computer data systems.They were concerned about challenges accessing the right data and in sometimes having to rely on others to get data for them.As another example, non-teachers in Musial also expressed concerns about the difficulties of data access.Additionally, they worried about the kinds of conclusions that might be drawn from data, such as data only serving to confirm expectations, rather than expanding knowledge.A few also felt that data they had personally collected were more informative to decision making.Non-teachers in Boyer were concerned about two issues.One was lack of time -particularly, how labor intensive data use could be due to the lack of integration in their data systems.The other was a general concern that some educators might undervalue data's role in improving practice since the district already had high levels of student achievement.

Principal Leadership for Data Use
Principals across the districts seemed to hold the benefits of data use in high regard (see Attitudes Toward Data above).Further, Table 7 suggests that participants across all three districts were generally positive about principal leadership for data use, as indicated by averages on the Principal Leadership scale.These views varied by role in Gibson and Musial (see Table 6), where more expectations were placed on principals to use data.In both districts, principals and their instructional support staff scored significantly higher on the Principal Leadership scale than did teachers and central office administrators.Central office administrators in both districts scored significantly lower on this scale than all other roles.
Nevertheless, qualitative data show that faculty struggles with data use were often connected to the leadership of their principals.These data show considerable variation in principals' leadership behaviors for data use.Further, we observed principals in Gibson and Musial more involved in leading for data use than were principals in Boyer.In the following two sections, we present detail on the positive and negative leadership approaches observed.
Positive leadership strategies.A few principals in our study had established structures that promoted regular, consistent data use in their schools.With the exception of one principal in Boyer, these principals worked in Gibson or Musial, and were more common at the elementary than secondary level.
A few principals were particularly active in developing robust collaborative routines.Not only did they support teacher-to-teacher collaboration, they also worked directly with teachers on data-related activities and used a collaborative, collegial style in setting expectations and plans for using data.Their teachers reported planning with their administrators at team meetings and described their administrators as committed to communicating with teachers about data.An exchange between Musial focus group teachers described this perspective: Teacher 1: He'll come and sit down with a team.Teacher 2: He'll pare it back for you.
Teacher 1: Yeah, he'll sit down and say, "Well, let's talk about this and how it works."He wants to know how we're doing things, what's the growth, he's very serious about that.
Similarly, a handful of campuses stood as examples for how administrators and teachers could be in strong agreement about how and why data can contribute to improving schooling.Notably, we heard talk at these campuses that suggested data use was a substantive part of everyday work, fostered by leaders and sustained by the interactions and activities of faculty.
Examples of other structures included explicit time set aside for faculty to examine data use, varied forms of collaboration (e.g., grade-level teams, vertical teams, interdisciplinary teams), and instructional coaches trained to support teachers in using data.Further, principals that appeared to be more skilled at using data were better able to help faculty learn to use data systems, lead discussions on the meaning of data, and promote effective questioning.
Negative leadership strategies.Unfortunately, many more principals were employing few of these strategies, or were using strategies that had negative effects.In these schools, teachers directly expressed frustration with their administrators' attempts to lead for data use -their negativity often was expressed first toward data explicitly, but frequently transitioned toward comments critical of their principals' leadership behaviors.
Many teachers were critical of the data skills of their principals and some said their principals did not engage in any significant data analysis.One Gibson teacher described how this affected faculty data use: There are leaders on this campus who don't know how to analyze the data.And so you have pockets of teachers who really good at using and analyzing data, and then you have pockets of teachers who just follow along with what those folks are doing, but they have no idea how to access any of their data.Some principals and faculty described a directive, "top down" approach to leading the use of data.Many of these campuses were quite involved in using data, but there was little evidence of collegiality.In fact, teachers at these campuses more frequently spoke of data use as a monitoring exercise, as this Gibson teacher did: "They want 'proof' that we're analyzing data -they don't just let me go analyze my data.I don't need to write anything down, I need to absorb it.In my opinion, they want stuff written down so it's proof that I've looked at the data." We also heard of leadership behaviors that seemed punitive in nature.For instance, teachers at one Musial campus said that their principal posted lists of teachers whose failure rates were greater than ten percent.Another example came from a Gibson campus, where teachers reported being called in to discuss data with their administrators when assessment scores were not up to par; these teachers reported feeling that the tone of these discussions was punitive.
Our data revealed few instances in any district of principals structuring time for faculty to examine and reflect upon data.There was no evidence that principals were limited by district policies in their ability to provide time for data.In fact, there were instances where district policy enabled faculty collaborative time (e.g., PLCs, common planning time), but principals rarely specified that their faculties spend that time using data.As a consequence, it was common for teachers at all three levels to describe their use of data occurring before school, after school, at lunch, or on their conference period.Further, there were few examples where principals provided structure for how the time set for using data could be spent (e.g., guiding questions, analysis protocols).
Teachers often reported mixed feelings about their interactions with administrators.In Gibson and Boyer, some teachers worried that administrators did not respect other demands on teachers' time.This was especially so for teachers who were asked to perform burdensome and redundant activities, such as hand copying data in preparation for meetings or creating paper records for administrators as "evidence" of collaboration.In Musial some teachers were concerned that collaborative data events were more about creating "lockstep" and predetermined practices, rather than about innovating solutions for students.
Finally, we were struck by the disconnect we observed between teachers and principals at many schools in each district.At some schools, it felt as if administrators and teachers were talking about two completely different schools.For example, we frequently encountered schools where administrators cited clearly articulated expectations for using data, but teachers were unable to articulate these same expectations.As another example, administrators in one Gibson school described themselves and their faculty as avid users of data and provided many examples and artifacts showing how their school used data.However, teachers at this school were negative toward data, reported using it infrequently, and were suspicious of their principal's motives around using data.

Computer Data Systems
We defined a "data system" as any computer-based tool that helped educators examine student data.Under this definition, the data revealed a diversity of systems employed in each district: 39 distinct systems were found in Boyer, 58 were found in Gibson, and 68 were found in Musial.
While the variety of systems was large, many provided data that was fed into to three general types of systems: (1) student information systems (SISs), which typically handle day-to-day student information, such as scheduling, course grades, and demographic information; (2) assessment systems, which rapidly organize test results; and (3) data warehouses, which integrate data from a variety of systems and provide a comprehensive, longitudinal view of student performance.
In the following, we first provide three sections presenting an overview of data systems in each district.Next, we offer a section that describes the uses of these systems, followed by a section outlining issues in integrating these systems.Finally, we provide a section that outlines educator attitudes toward these systems.
Systems in Boyer.Of the study districts, Boyer was the least advanced in its data technologies.Their SIS was not technically an SIS, but an electronic gradebook built to handle grades and other student data.Boyer employed two separate assessment systems: one geared toward collecting, analyzing, and reporting district-or state-level test results (e.g., district benchmarks and state achievement tests), and the other geared toward providing regular, formative assessment data to teachers.Boyer had no data warehouse to integrate various systems.
Systems in Gibson.Gibson employed an SIS for day-to-day information, as well as an assessment system for providing assessment data.There existed a data warehouse in Gibson, but it was not user-friendly in the ways that it integrated or presented data; consequently, the data warehouse typically was used only to organize and access district-and state-level tests.
Systems in Musial.Musial also employed an SIS for day-to-day information and an assessment system for providing assessment data.Although Musial technically had a data warehouse, it did not exist for many educators in a practical sense: due to site license issues, most educators (and all teachers) did not have direct access to the data warehouse.The data warehouse provided reports that pulled data from many disparate systems, but access problems had created bottlenecks in accessing these reports.Further, when reports arrived, they typically were in the form of a large Access or Excel file.
System uses.Table 8 shows the number and percent of educators that reported weekly use of their district's various systems.In each district, the SIS was the most prevalently used system.
Otherwise, these numbers reflect the diversity of technological contexts in the three districts -no clear patterns were apparent across district or role.Although the distinctions between SISs, assessment systems, and data warehouses may seem clear, we found that districts sometimes attempted to "stretch" the intended use of their systems to meet locally desired ends.Thus, systems might be used in ways for which they were not necessarily designed.For example, the Boyer electronic gradebook was being used by central office members as an SIS and doubled for some teachers as both an SIS and gradebook.The Gibson data warehouse provided another example: although it could provide data integration, it typically was applied as an assessment system.
We heard some instances of data systems facilitating work, but much more commonly heard how technical limitations suppressed data use.For example, many Musial educators described the "information overload" they felt with the large Excel files or stacks of printouts they received.Gibson educators often experienced a slow turnaround time for feedback regarding district-based assessments.Boyer educators rarely described using their systems in ways that were not compliance oriented (e.g., state achievement reports or student documentation).
System integration.In every district, many educators described the desire for easy to access, integrated information about students.They wanted the various systems in their district to connect in a way that offered one-stop access to all data, as described by a Musial principal: "This district wants to be data driven.They value it.They have tons of data, but not one good solid system." District administrators in each district recognized this problem and each district was addressing it in a different way.Boyer administrators were considering a data warehouse and saw participating in our study as one way in which they might prepare themselves for the "next level" in data system implementation.Gibson was developing plans to obtain a more dynamic data warehouse that could provide more refined, comprehensive, and timely views on students.Musial was working to expand licenses to allow teachers access to the data warehouse.This work included developing a user-friendly system interface, as well as tasking a central office administrator to help ensure that educators' interests were represented in the new interface.
Educators in each district were going to great lengths to compensate for lack of integration.Several educators in each district described using self-built and maintained systems (e.g., Excel, Access, or paper and pencil) in order to aggregate and analyze their students' data.Although more work-intensive and less powerful than the output from an integrated system, the resultant analyses were often described as useful.A Gibson assistant principal described one such self-built system used in his school: Every teacher has a folder and a corresponding spreadsheet that has every single one of their students on it.Each student has their ethnicity next to their name, along with the class period that they're in.And what this allows the teacher to do is enter the data after an assessment is taken, and the teacher will then be able to sort through this to see how students in different classes did by ethnicity and (standard).And then they'll be able to target their intervention by specific (standard) in a special intervention period, where the students go to a particular intervention based upon need.So it's a quick way for teachers to have an at-a-glance look at how the kids are doing on what they're teaching in the classroom.We uncovered these self-designed systems at every level of every district, sometimes shared across teams or buildings via email or a school common drive.In some cases, "power users" were assigned the task of querying, assembling, and distributing these makeshift approaches to integrating data.
Attitudes toward data systems.Table 6 shows that scores on the Computer Data Systems scale only differed by role in Gibson.Table 7 shows that teachers and instructional support staff in Gibson perceived their data systems more positively than did campus administrators and central office personnel.
Interview data in all three districts revealed stronger negative attitudes than did survey data.Typically, educators in all three districts were highly frustrated with the amount of time it took to access and analyze data, and most of their frustration resulted from a lack of system integration.Lacking an integrated data system, educators in each district reported that the time burdens of generating and compiling data left little time to actually use the data.For example, educators in each district reported spending too much time pulling together data from disparate systems and some reported hand-copying data to ready it for use.This was especially so in Gibson and Musial, where some described using their time to fill out forms that could otherwise have been pre-populated by computer data systems.One Gibson teacher described one such example, saying, "We took the [data system] printout and copied the same information in a watered-down format onto another sheet of paper … yes, it divided things I was strong in and weak in, but I could have done that on the printout."Educators in every district saw such labor-intensive routines as untenable in light of today's advancements in technology.

Discussion
The goal of the present study was to examine how practices at every level of the district affect the use of data in the classroom.We addressed this goal by exploring uses of data, attitudes toward data, principal leadership, and computer data systems.Educators were ambivalent in their attitudes toward data -they often could see how using data could improve practice, but were negative about the barriers that made data use hard for them.Many of these barriers related to principal leadership and data systems.Some principals in this study were shown to be employing multiple leadership strategies to facilitate faculty data use; in those schools, attitudes were good and data supported educational practice.Unfortunately, more principals were employing few such strategies; in those schools, educators reported negative attitudes, and had difficulty using data to improve practice.Results also showed a range of hardships and wasted time caused by inefficient computer data systems.Most importantly, these factors together affected how data use translated into knowledge and practice.
Fortunately, many of these barriers are ones that can be handled structurally.Formal district policies for using data were largely absent from our results, so we suggest that districts can support and facilitate classroom practice by attending to data-related district policy.That is, districts can improve how data inform practice if leaders are intentional about writing and codifying district policy to alleviate the barriers found in the present study.In the following sections, we provide discussion of five areas that district leaders might consider when creating policy: context, attitudes, principal leadership, computer data systems, and focus on all learners.

Context
Context can be important in considering district data use policies.However, our results suggest that, regardless of context, districts may be dealing often with similar policy issues.We identified a few findings that were considerably different from district to district; other differences existed only in details specific to each district.
One finding from this study depended strongly on context: the interaction of context and policy environment seemed to affect how data were used.Gibson and Musial were averageachieving districts looking to improve, so there was a stronger response to accountability ratings in these districts than in Boyer.The attention given to accountability ratings in Gibson and Musial sometimes seemed to distract principals and other leaders from the bigger picture of teaching and learning.Educators in the high-achieving Boyer district did not view their students' achievement in terms of accountability.Still, accountability had an effect on data use in Boyer, because many educators saw data use as accountability-related and thus believed that data use did not apply to them.Thus, the accountability system created little motivation in Boyer to engage in data use.
For most other findings, context mattered only in detail.For instance, regardless of district, we found that educator attitudes toward data were often ambivalent because of barriers, but the barriers themselves often depended on district.As another example, all districts had integration problems with their data systems, but the responses to this problem varied by district.

Attitudes
Attitudes toward data have been shown by prior research to be important: in exemplary contexts, studies have shown attitudes toward data to be positive and educators believed that data use positively supported their practice (Wayman & Stringfield, 2006;Datnow et al., 2007;Lachat & Smith, 2005).However, studies in these contexts rarely present the daily barriers shown in our data.Our data showed attitudes toward data were remarkably positive, even in the face of these barriers.Positive perceptions of data's potential seemed to be what kept educators using data, because many thought data could help them in their practice.Consequently, educators sometimes even constructed time-consuming workarounds in order to use data.
This notwithstanding, it must not be ignored that educator attitudes in our study often were negative toward what data use did to them.At first glance, this seemed related to role: survey data indicated that teachers were consistently more negative than administrators or support staff about using data.However, in triangulating survey data with interview data, we came to interpret these attitudes as related to everyday practice, not role: attitudes toward data were mostly related to the barriers and problems educators faced.
We surmise that teachers were typically more negative than other roles because the nature of how they worked with data made them subject to more barriers.To contrast, instructional support staff used data frequently, but were often more positive.Instructional support jobs often carried natural structures that enabled them time and other supports, so it is possible instructional support staff did not face the day-to-day barriers that teachers did.It is reasonable to conclude that if the activities associated with data make it harder for one to do one's job, then one would be understandably negative about data -regardless of role.
In considering how attitudes might shape district policy, there is a bright spot in these results: attitudes seem to be influenced by structural barriers and these structures can be affected by district policy.District policies can be written to improve a number of complaints found in our study, such as time to use data, learning opportunities, expectations for data use that align with improved practice rather than accountability compliance, principal leadership, and data systems.

Principals
Numerous studies have suggested that the principal is important to the success of a data initiative (e.g., Wayman & Stringfield, 2006;Copland, 2003;Datnow et al., 2007).In the present study, the connection between a principal and faculty appeared to greatly affect how data were used; this connection seemed to depend on the strategies employed by the principal.In schools where principals intentionally employed strategies to support data use to improve student learning, teachers typically reported better attitudes and more effective data use.In schools where principals employed fewer or negative strategies, attitudes were worse and data use was reported to be less effective.In these schools, teachers were often left to deal with data on their own -not just in terms of the meaning they drew from it and subsequent change in practice, but also in structures, such as time and data access.
District policies can address principal leadership strategies in a number of ways.Some are relatively simple.For instance, policies could provide for professional development that helps principals learn how to use data-related leadership strategies.Or, policies could contain a list of the types of strategies that district leaders believe may be most effective in the district context (see Wayman, Spring, Lemke, & Lehr (2012) for a research-based inventory of such strategies).Policies such as these may seem obvious, but we saw no evidence that any of our districts had addressed principal leadership in any structured way.
While these policies are an important first step, the current study provides evidence that simple policies such as these -those that only mandate action -are not sufficient.Policies must additionally provide elaboration and support for how these actions may be implemented.For example, our findings highlighted district policies that provided building-level educators time to collaborate (e.g., PLCs, common planning time).Despite these structures, principals rarely encouraged or required this time to be allocated to analyzing data.In this case, principals were not limited by district policy in providing faculty time to use data, but were not fully supported by this policy because it only created collaborative time.Thus, even in the presence of these policies, many teachers still reported insufficient time to use data.These policies would have been more effective with regard to data use if they had been written to also provide support for what went on in that time.For instance, a policy could require that this time be dedicated to data use a certain number of days per week and could provide guidance regarding key problems to work on.

Computer Data Systems
Results of this study, like others (e.g., Wayman & Stringfield, 2006;Lachat & Smith, 2005;Means et al., 2010), make it clear that educators need integrated, one-stop access to the data they want to use.Integration was so important to the educators the present study that many improvised time-consuming methods to connect various pieces of data.The results of this study additionally showed how ineffective data systems affected attitudes toward data and the progression from data to information to practice.
Clearly, any district that chooses to make data an important part of their educational experience should enable some way to integrate data so that educators have one-stop access to all the data that are available.But our findings show that the use of these systems is often more complex than merely enabling the hardware.For instance, users and districts were sometimes applying these systems in ways different than envisioned by system designers.Cho and Wayman (2012) suggested a reason for this: system use is dependent on the sense that the user makes of it.They further suggest that preconceptions about data systems, data, and their utility for practice serve as a lens for determining what is meaningful about a system. .Thus, educators may have certain expectations, potential uses, or kinds of practices in mind before accessing data.This may promote some system features as worthy of use or adaptation, while obscuring the significance of others.
Consequently, district policies for data systems should be twofold: Besides providing the means for providing one-stop, integrated access, policies must also attend to how the system will fit into the everyday work of the educator.We posit that data systems should be at the center of every district policy about data use.For instance, a policy dealing with professional learning should outline how the data system will be used to support skills educators gain during professional learning sessions.Or, a policy supporting principal time structures should also describe how the data system may be used to make use of this time more productive.

Focus on All Learners
Our results showed data use being focused primarily on struggling learners.We believe this is probably the case in most districts -while prior research has not explicitly identified or quantified this result as in the current study, almost all of the uses of data described in prior research involve struggling learners (e.g., Anderson et al., 2010;Wayman & Stringfield, 2006;Datnow et al., 2007;Lachat & Smith, 2005;Marsh et al., 2010;Young, 2006).Focusing on struggling learners is an absolute necessity; this fact has been underscored in the current accountability climate.However, we also caution against narrowing data use to focus on one group of students, because narrowing toward one group does not fit how schools work.Instead, we suggest that a data initiative will be more successful for struggling and non-struggling students alike if it is focused on formative opportunities for every student throughout the district.District data policies should be written to describe how this focus on all students will raise the achievement of struggling learners through increased differentiation or other sound practices.

Conclusion
Most of the educators in our study wanted data to be used to support classroom practice.Unfortunately, they faced barriers and problems that sometimes made this hard to do.Our personal experience and our reading of the literature indicate that many districts across the United States are in the same situation as our study districts.
While we found that the influences on data use were numerous and complex, we also believe that many influences can be made positive by effective district policy.Unfortunately, our districts had very few -if any -policies related to data use.On the bright side, some of these policies can be easily written.Further, a little bit of policy may go a long way toward improving conditions around data use.
If districts are to write effective policy, they must keep in mind that, at the heart, the findings of this study speak to how data use fits (or does not fit) into the everyday work of education.Thus, it is likely that district policies grounded in compliance will not be effective, but policies that react to and support the ways that educators work will be effective.
We believe that the effective use of data can be a strong contributor to educational practice.However, we believe this hypothesis cannot be confirmed until contexts are created where a variety of factors support data use district-wide.This study has presented new information toward making these supports a reality and we hope to see such supports become commonplace in the coming years.

Figure 1 .
Figure 1.Conceptual Framework of Data Use at the Building Level.

Table 3
Respondent Rankings of Data Uses in Boyer, By Role Note.Mean response is shown in parentheses and only uses specific to the role are included.Note.Central office is not included because few uses applied to that specific role.Note.Teacher: n=284.Campus administrator: n=13.Instructional support: n=21.

Table 4
Respondent Rankings of Data Uses in Gibson, By Role Note.Mean response is shown in parentheses and only uses specific to the role are included.Note.Central office is not included because few uses applied to that specific role.Note.Teacher: n=1117.Campus administrator: n=62.Instructional Note.Mean response is shown in parentheses and only uses specific to the role are included.Note.Central office is not included because few uses applied to that specific role.Note.Teacher: n=1117.Campus administrator: n=62.Instructional support: n=82.

Table 5
Respondent Rankings of Data Uses in Musial By Role Note.Mean response is shown in parentheses and only uses specific to the role are included.Note.Central office is not included because few uses applied to that specific role.Note.Teacher: n=1215.Campus administrator: n=52.Instructional support: n=146.

Table 6
One-way ANOVAs for Survey Scales by Role, for Each District Note.Independent variable is role.Note.Central Office administrators are not included for Boyer because of insufficient response.

Table 7
Means of Survey Scales, Disaggregated by District and Role Note.Means are not presented for Boyer Central Office because of insufficient response.Note.Significant pairwise role differences are noted by superscripts within the table.t = teachers, ca = campus administrators, co = central office, and is = instructional support

Table 8
Number and Percent of Participants Using Data Systems Weekly or More, Disaggregated by District and Role Note: Percent of participants reporting weekly or more use given in parentheses.Note.Boyer does not have a data warehouse Note.Figures are not presented for Boyer Central Office because of insufficient response.