Introduction

As growing diversity has been matched by increasing disparities in educational outcomes between a number of groups—poor and affluent, white and minority, urban and suburban—a patchwork of uncoordinated programs have been introduced at all education levels by a variety of government entities. This ad hoc form of policymaking often results in programs that at best do not reinforce one another and at worst actively work to undermine each other. For example, the move to base access to Title I resources on economic rather than academic need was an important shift in emphasis that allocated resources more equitably. Unfortunately, a number of local programs in many urban school districts relied on the annual testing funded by Title I as an important component in assessing overall system performance and often used the data to inform local goals. A more promising response, which has developed over the past decade, has been what is referred to as systemic or standards-based reform. Systemic reform, according to U.S. Deputy Secretary of Education Marshall Smith, is typically based on state-level reforms that implement more rigorous content and performance standards across grades and disciplines.² Systemic reform requires that curricular material and assessments be aligned with these standards. Preservice teacher education and teacher professional development must also support these goals. Finally, funding, technology, physical plant, and human resources must all be allocated in such a way that each group has equal access to the things it needs in order to succeed. To improve student achievement and to close the equity gap are the ultimate goals of systemic reform efforts, but supporters of systemic reform believe these goals can only be achieved by improving all aspects of the educational system.

Major systemic reform initiatives at the national level seek to strengthen teacher education, reinvigorate the development of high-strength curriculum, and promulgate disciplinary standards. Some of these programs operate under the auspices of the U.S. Department of Education and the National Science Foundation.³ The majority of reforms are state-level initiatives to improve the performance of students in specific subject areas or grade ranges such as elementary reading or middle school mathematics (Armstrong, 1999). Individual districts also engage in reform initiatives that seek to motivate particular schools or groups of students to meet individual standard components. Finally, individual schools and classroom teachers work to develop lesson plans to teach concepts based on grade-level standards. These trends and the consequent emergence of more data-rich environments raise the need for new technologies and new management techniques for coping with complexity. Three technical issues in particular appear to be especially crucial.

First is the problem of accurately identifying policy targets. Identifying the target group and the desired outcomes for a particular reform is necessary if one is to describe, analyze, and locate reform efforts within educational systems. Existing organizational identifiers such as school, classroom, or demographic data are often inadequate to isolate for study or evaluation students or teachers who participated in particular programs or received distinct treatments. The ability to accurately compare analytically distinct groups is vital if one is to assess the impact of systemic reform.

Second is the problem of managing data. It is evident that successful systemic reform will depend on access to and effective use of large amounts of data. This means that the quantity, timeliness, and level of detail of the data needed from decision-support systems will only increase. Proponents of systemic reform point to the importance of a process model and evaluation framework for assessing programs (Clune, 1998). The process of systemic reform must include actors from all levels and it must include an awareness of resources and barriers confronting actors across educational roles. The focus on process also points to the importance of quality indicators. Quality measures might include a detailed analysis of curriculum goals and well-understood, publicly available education standards. High quality instruction demonstrates an alignment between standards, curriculum, and assessment. The difficulty of measuring differential interventions is compounded by the need to gauge the quality and thoroughness of new, robust curricula and to obtain more detailed analyses of student progress.

Third is the problem of metrics, i.e., multiple-choice versus authentic-performance tasks. Demands for more authentic assessments have led to calls for new metrics of student performance relative to standards, as well as an emphasis on measuring a particular student's attainment of individual proficiencies.⁴ The new standards-based approaches have led to the creation of new sorts of testing that go beyond the typical "fill in the bubble" standardized assessments. The introduction of multifaceted testing regimes (including traditional standardized tests, as well as performance- and portfolio-based assessments) both increases the complexity of the individual instruments and creates new requirements for the underlying data systems that must both record and provide an analytical environment for the data. The increasing sophistication of assessment practices calls for a parallel development in the arena of information-management strategies.

The problems described above are the technical aspects of fundamental research questions—What to study? How to aggregate the data? And what are the appropriate measures? As our understanding of the educational process deepens, our technical capacity to collect, manage, and analyze data must keep pace.

This article seeks to apply insights from the growing body of literature on knowledge management (based primarily on research coming out of United States and European business schools)⁵ to the specific case of systemic reform in U.S. education. Knowledge management (KM) strategies can serve as valuable tools for decision makers at all levels of the educational system. In this paper, I focus on outlining the distinctive features of knowledge management, identifying the characteristics of successful KM efforts and exploring the usefulness of KM for making important educational decisions. I conclude with a discussion of the role of information technology (as a component of a KM system) in the implementation of high-stakes accountability for 8th grade students in one particular urban district, that of Milwaukee, Wisconsin.

I. Knowledge Management (KM): Definitions and Scope

[I]t is perhaps easiest to understand knowledge in terms of what it is not. It is not data and it is not information. Data are objective facts, presented without any judgment or context. Data becomes information when it is categorised, analysed, summarized, and placed in context.

Information therefore is data endowed with relevance and purpose. Information develops into knowledge when it is used to make comparisons, assess consequences, establish connections, and engage in a dialogue. Knowledge can, therefore, be seen as information that comes laden with experience, judgment, intuition, and values. (Empson, 1999)

The progression from data to knowledge can be seen both as a temporal process in which data, imported into a system's architecture, aggregates individual facts into summaries and averages that are then presented in an appropriate context. In an educational setting, this might be a report of student test performance by grade, ethnicity, race, and gender. The addition of deeper contextual information about local school leadership, particular organizational characteristics, or other less quantifiable factors can be combined with mechanistically generated test-score results to describe variance in outcomes that could not be extracted from the more traditional reports. It is this application of personal knowledge and of well-designed models that differentiates information systems from knowledge systems.

Knowledge Management as the Use of Data and Information

In districts, the student data system provides the core of such a system. However, this focus on information systems and tools for aggregation and on KM as the application of information should not imply some type of computer-based system that is somehow imbued with deep contextual knowledge of the organization. This is not just a question of the technologies employed. An important role is played by institutional culture. For example, a district with a collaborative model of interaction between schools will typically display a far greater capacity to develop robust analyses of school-level processes and needs than a district lacking such a model. Davenport has identified several important questions that may be helpful in efforts to clarify an organization's approach to the use of knowledge management:

Does an organization's culture reward decisions and actions according to how people use and share their knowledge? Or is it content with the widespread use of intuition and guesswork at the expense of organizing people and processes to apply the best knowledge, experience, and skills to projects and tasks? (Davenport & Davenport, 1999)

The Davenports point to the importance of organizational culture in enabling or blocking the use of knowledge. Cultures that support knowledge accumulation and application will be the most effective, efficient organizations. Organizational structures and processes provide a window into the value a knowledge management system will return to any implementer. A willingness to engage in problem-solving processes and share information with "outsiders" is an important resource for enabling knowledge management efforts.

The Objectives of Knowledge Management

1. Create knowledge repositories– a) external knowledge (competitive intelligence, market data, surveys, etc.), b) structured internal knowledge (reports, marketing materials, techniques and methods), and c) informal internal knowledge (discussion databases of ‘know how' or ‘lessons learned').

   In an educational setting, curriculum aids might be thought of as knowledge repositories. For example, the Milwaukee Public Schools Curriculum Design Assistant⁶ (CDA) is both a source of documentation—standards, learning goals, etc.—and a repository for instructional plans based on this documentation. These lesson plans can be stored in the system and shared with others electronically to provide a knowledge base for a wider audience.

2. Improve knowledge accessthrough a) technical expert referral, b) expert networks used for staffing based on individual competencies, and c) turn-key video conferencing to foster easy access to [geographically] distributed experts.

   Examples of this sort in public school education are probably rare, but the Community of Science online database⁷ is a data and communication resource that functions well in education research: it links researchers, research intuitions, and funders together. The purpose of the Community of Science online database is to reduce the barriers for those seeking funding for research, as well as to reduce the difficulty of funders in locating qualified researchers.

3. Enhance the knowledge environment –a) change organizational norms and values related to knowledge in order to encourage knowledge use and knowledge sharing, b) customers may be asked to rate their provider's expertise.

   This objective focuses on the creation of a technological environment that will contribute to the social transformation of an organization. Anderson Consulting used this sort of approach to radically shift norms of information sharing and use among its consulting staff (Graham, Osgood, & Karren, 1998; Greengard, 1998). In the consulting business, there are traditionally strong norms about keeping personal expertise personal—it represents a large portion of individual competitive advantage. Anderson Consulting wanted to reverse this behavior and reward those who shared information with other consultants within the organization. The company began to make participation in an online, e-mail-based, problem-solving environment mandatory. Eventually the bar was raised further and pay and promotion were linked to the number, quality, and immediacy of an individual's responses. This approach was draconian, but it was successful in building both a compelling repository of problem-solving information and shifting or overturning a strong norm against sharing information. This approach models an important tool for accumulating and diffusing successful educational practices. The call for methods of replicating successful programs and school initiatives that "beat the odds" could be addressed with a system that improves communication within educational systems.

4. Manage knowledge as an asset– a) attempt to measure the contribution of knowledge to bottom line success. (Davenport, DeLong, & Beers, 1998, pp. 45-48)

   While this final KM objective sounds the most compelling, it is also the most difficult to operationalize. Even firms with excellent data-management practices and sophisticated conceptions of return on investment have difficulty assessing the return on intangibles. Learning Landscape⁸ is an example of one such effort in education. This system was an outgrowth of the Connecticut Academy's NSF-sponsored Statewide Systemic Initiative. The Connecticut Academy worked together with the consulting firm, KPMG, to develop a data warehouse environment based on the National Center for Education Statistics' core data elements. The stated purpose of the project was to transform data into knowledge to improve student achievement and teacher quality. This project was abandoned after it became clear that the development costs would exceed what most districts would be able to afford. A new firm has been established, EdExplore, which has the same KM goals but is much more limited in scope and has implementation costs in line with district resources. EdExplore remains focused on bringing cutting-edge approaches to data warehousing to the evaluation of student and teacher performance.⁹

The fourth goal—determining the return on investment in knowledge management—is closely related to efforts that attempt to determine the return on investment from spending on early research and development. Many analysts have struggled with this vexing issue, which remains only partly addressed. There are so many unquantifiable, human elements in a KM system that it may be very difficult to come up with metrics that are generalizable.

II. Knowledge Networks and Educational Systems

The arguments of the authors cited thus far in support of knowledge management systems provide very little insight into the exact analytical approaches one would use in any particular scenario, since so much depends on the organizational form, sector, or level of analysis. The discussions in the literature focus instead on families of tools designed for pattern detection and predictive modeling. It is important to bring these together with the experience of those working in the specific domain to identify the important dimensions of knowledge in that field, since we have defined knowledge as the application of information in context.

A crucial feature of educational systems is that they are made up of a number of nested systems or organizations. In analytical terms, this can also be described as levels or units of analysis. In education, these levels range from the federal level, through the state, district, school, classroom, and student levels. Reporting and analytical needs differ from level to level as do the relevant time scales. For example, there are classroom needs for lesson planning and local testing. At the school level, Title 1, free lunch, state-mandated testing, and other mandated programs are focal points of data-management issues. The reporting requirements are as numerous as the funding sources at each level of the organization. Analytical needs differ, but are present at every level of the system. A robust knowledge management system must reflect the information and knowledge management needs of all levels. In particular, data must be gathered at a level of aggregation appropriate to the user with the most fine-grained analytical needs.

The Level-of-Analysis Problem

The importance of the level of analysis comes to the forefront of any effort to describe and apply KM principles. For example, district-level functions might include analysis of the quality of the data in the system (this may be in formal terms of validity and/or reliability, as well as of alignment of the metrics in use and the learning or performance standards). District-level analysis might also focus on the curricula that are in use across the district and assess their relative effectiveness. These district-level functions call for nested hierarchies of approaches to systems design that support collection of the relevant data and aggregation of these data to the appropriate level of analysis.

The identification of distinct levels in an organization with different KM needs is even more vital if one considers the increasing demands technology places on individual units in an organization. Boisot argues that the traditional neoclassical economics concept of a linear relationship among the different factors of production fails to adequately explain the returns of knowledge (Boisot, 1998). Boisot points out that as one moves up within an organization, the number of elements that must be integrated in order to produce a good or service has probably not increased dramatically in recent years. However, the pace at which new technologies and needs are introduced and old technologies become obsolete has increased dramatically (Boisot, 1998). This rapid pace of change in the factors of production places a premium on the ability of individuals within an organization to track change and respond to it. This ability is one of the key features of successful KM.

Unfortunately, this is an area in which traditional information management systems (such as SASIxp™ or ABACUSxp™ from NCS or eScholar from IBM and Vision Associates) are particularly weak in general.¹⁰ Data warehouse systems are excellent tools for making complex selections of data from many different sources. However, there are no good inferential or predictive models in place within commercial school decision support systems for modeling school or student behavior in real time. This is not because there is not a rich understanding of student achievement. Rather, the cost and complexity of real-time KM systems represent a major obstacle to schools or districts taking advantage of this technology. Schools have lagged behind industry in replacing paper-based clerical and business functions (often referred to as back office functions) with technology-based systems. They often lack the physical infrastructure necessary for communication between numerous geographically dispersed buildings.

Increasing Access to Computing Power

There are certainly budget, security, and management implications in broadening access to sophisticated computing technologies. However, the need for information transcends district governance issues. Moves toward greater local autonomy and responsibility—whether in response to school vouchers, charter initiatives, or other pressures—mean that there will be an increasing need for local analytical capacity. The need for well-informed local decision-making at the classroom level is not unique to any particular organizational form, however, such as block scheduling, or multi-age classrooms. It exists in both flat and hierarchical organizations.

The level at which decisions are made may differ across organization types, but there must still be robust data-collection and knowledge-dissemination mechanisms at all levels. This is particularly important given the different uses of individual data elements at different levels of aggregation and at different levels of the organization. Even at the school level, one can demonstrate the utility of comparing groups such as students who are bussed versus not bussed, or students participating in a specific after-school program versus students who do not. Traditional aggregations of data in such instances prove inadequate.

Student Data as the Basic Unit of Analysis

At the same time, however, districts need to integrate increasing demands for accountability, both fiscal and educational. Increasing scrutiny at the district level implies an entirely different focus for information system design. In this district accountability model, data are aggregated up to higher levels, such as the school or program, and are used to justify administrative or managerial decisions that affect large numbers of students and schools. This does not simply mean a reliance on central capacity. Knowledge management should be responsive to needs at all levels of the system. The problem is that the events and outcomes being monitored all tend to occur at the lowest levels of aggregation—the student.

Even in the most centralized system, instructional outcomes occur at the individual level. However, most district accountability measures and goals are aggregated to higher levels of the organization. Data on student and teacher absence, student program participation, teacher professional development activities and test scores need to be available at the lowest level for aggregation to any meaningful unit—classroom, grade, neighborhood, or other coherent grouping. This means that only individual-level data will serve. This is important because reform efforts must work on multiple levels. It is also vital that the system be transparent in order to combat efforts to cheat or otherwise tamper with the data. Data collected at the student or classroom level need to be aggregated and fed upwards to higher levels of the educational accountability structure to inform system decisions made at those levels. A system without proper security in place could be subject to manipulation by school-level actors. Again, this is not an argument for or against centralization, per se. Rather, one must recognize that there are knowledge management decisions that are appropriate and possible at each level of an organization. Data structures and analytical tools should reflect this reality.

This divergence between existing data systems and data needs points to a gap between new technological capabilities and the policy environment that would enable schools to actively collect relevant information and put it to use in a KM system. When the model of change targets student learning, it is essential to focus on identifying those innovations (technical and organizational) that best serve schools and classroom teachers. Most existing models of school governance do not provide for data analysis at the classroom level. Often there is also a gap between technical capacity and technical feasibility. Most educational systems have very little technical capacity. For example, teachers may not have easy access to the Internet, or district computer staff may not have much experience supporting the analysis of problems relevant at the classroom level. There must be collection and delivery systems in place. There must also be analytical support for complex problems that are not solved by looking at simple bivariate comparisons. What technological capacity does a school system need? That depends on the system's goals.

Making instructional decisions in an individual classroom requires a sophisticated understanding of student attributes and of what individual curricular units can provide. The ability to generalize to a larger population is not a relevant concern in this context. At the school level, however, a principal may be very interested in how a new professional development activity is affecting student performance relative to other classes in the same building, or in comparison to other schools across the district that did or did not participate in the same training. The tools and data needed for within-classroom analysis are quite different from those necessary for a cross-school comparison. Differing forms of organizational analysis call for diverse forms of KM system implementation. One faces similar challenges at any level of aggregation, from the classroom through the district level to the federal level.

This discussion does not directly address the existence or quality of data needed for analysis. The lack of reliable data for school and student evaluation continues to be a vexing problem. Without a solid analytical model and the appropriate KM infrastructure to collect and aggregate the data, no district of any size would be able to implement a robust evaluation system. For example, many districts embrace an assessment policy that focuses on individual grade cohorts. This has led to the construction of information systems and testing plans that do not support longitudinal analysis of individual students. However, a system that allowed for accurate and fair evaluation at all levels of the educational system would differ radically from the current model that separates data use and evaluation (occurring at the state and district level) from the people who actually do the teaching and learning (schools, teachers, and students). A fair, consistent knowledge management system would provide an environment for implementing an appropriate stakes model at each level, rather than focusing high-stakes measures on teachers and children.

Variations in Organizational Approaches to Knowledge Management

Donald Marchand argues that organizational reform efforts can often best be described by the way in which they operationalize their KM strategies. He describes four important dimensions of such strategies, as shown in Figure 1 (Marchand, 1999). On the horizontal axis, managing costs and risks are straightforward concepts. On the vertical axis, adding value implies increasing ones return on investment. New realities can be conceived of as both creating new services or goods and creating relationships with new customers. Points near the center of Figure 1 represent little or no activity along an axis, while points farthest from the center represent best practice.

Figure 1. How Information Creates Business Value (Marchand, 1999).

In this example, Organization A is an organization that sees innovation in information technology as a way to reduce risks and costs. A school district following this strategy might implement a new in-school, computer-adaptive mathematics assessment to reduce outside testing costs and to help minimize instructional time lost to testing. Such a system might also help address the problem of mobile students who may not be included in once-a-year standard assessments. Such an assessment would be an element of a comprehensive program to reduce the risk of litigation associated with differential outcomes between poor and non-poor students by providing the opportunity to test adaptively and at smaller time intervals.

Organization B, on the other hand, perceives the value of improvements in information management as coming from adding value or enhancing existing interactions and creating new information services and products. A school following this strategy might invest in workgroup computers and a high-speed Internet connection. This technology would be used to support teaching a chemistry class in a virtual, web-based environment that would allow for the interaction of commercial and university subject-area experts as outside advisors and mentors for student projects.

Marchand argues that an effective organization with a well-designed information management plan will have a clear understanding of its place on these dimensions and will plan for its development needs accordingly. It seems clear that an organization with a mature understanding of the importance of information for decision-making might map its strategy onto Marchand's graphic as a circular structure, Organization C. However, the costs involved in moving a complex organization along multiple axes simultaneously would almost certainly be prohibitive. It is important to recognize the tradeoffs between adding to and protecting what already exists. A balanced strategy would reflect this.

Mapping one's own organization on this framework can provide some important insights. No single effort can conceivably score high in all four dimensions, but a comprehensive knowledge management strategy ensures that, as a whole, investment in appropriate technological innovation addresses all of an organization's major goals.

Marchand discusses another dimension not explicitly captured in this graphic. Is a particular knowledge investment required for operation, is it essential for competition, or is it likely to bring distinction as a unique actor in the relevant market? Most investment in operational data systems tends to be directed toward the first two areas, day-to-day function and competition. It is the concept of standing out from the crowd that helps to drive truly innovative transformations of KM systems. In his research, Marchand finds that managers "rarely say that significant portions of their investments are focused on applications that give them distinctive competencies with customers" (Marchand, 1999). The immediacy and relative clarity of an organization's current operational needs tend to drive most KM investment strategies. This is no less true for school districts. Any district will have a working payroll or bookkeeping system. There is no other alternative. These systems must be operational and deliver substantial value. Investments in long-term payoffs, such as those often encountered in improving educational systems, typically suffer by comparison. A system for tracking curriculum development and delivery is typically not characterized by a sense of urgency. The rationale for building such a system is less clear-cut than for the other, more traditional, operational data systems. Operational systems track and manage day-to-day transactions but are of little use for planning or evaluation purposes. However, as is the case with basic research in the manufacturing sectors, there needs to be substantial investment in a knowledge management infrastructure for any organization to reap long-term payoffs. The return on investment from knowledge is small in the short term but can have a huge impact in the long run.

If one applies Marchand's dimensional framework to schools and asks how information creates value in an educational setting, the level of analysis becomes extremely important. For example, one focus of a school district, because it performs the business functions of any large organization system, would be on managing risks and reducing costs. District administrators would also be interested in producing increased learning, but they could only have an impact through indirect effects. One model of district action would be an active structuring of incentives and resources to create an environment in schools and classrooms that enhances learning. Bureaucratic structures are good at routine tasks and can provide infrastructure at a reasonable cost by taking advantage of economies of scale. Individual teachers, on the other hand, are engaged in directly adding value to the educational process and creating new realities for their students. This is a case of functional specialization. The problem is that actors at each of the levels need to be cognizant of the knowledge and system dynamics that define the other levels.

Even within a district office, there will be differences across organizational boundaries. Curriculum support staff might be heavily focused on "creating new realities" by aligning district curricular resources with new standards or goals. This might include a "training on demand" streamed video system for delivering professional development when and where it suits individual teachers and coordinators. Using technology to overcome the boundaries of time and space can create many new opportunities. A legal services team might be focused on reducing risk: for example, a district might put all Individualized Educational Plans (IEPs) online with a system for tracking interventions and student performance. This might be one response to concerns that special education students are receiving uneven access to services. This system could provide both a tool for demonstrating compliance and encouraging school staff to stay on top of student needs by making school-level activities more transparent. This drastic over-simplification highlights the difficulties one encounters when trying to design a system that meets the needs of actors at multiple levels in the educational setting.

It should be made clear that it is not necessary to build one monolithic knowledge management system. Indeed, it would be extremely difficult to build a complex system that would be adaptable enough to respond consistently to a changing educational policy environment. A resource that has been developed in this regard is the Schools Interoperability Framework, which is an important acknowledgement of the heterogeneity of data-acquisition and knowledge- management systems in use today.¹¹ Any realistic district information system must be made up of a matrix of interlocking systems that serve different functions and different user communities.

Risk versus Investments in Information Technologies

. . . the risks are even greater when, as with many Internet applications, real business innovations are also being looked for. The main reasons for disappointment cited also remain stubbornly familiar: incomplete definition of business requirements, insufficiently detailed technical specifications, changing business requirements, and lack of business user input during development. (Feeny & Feeny, 1999)

Unrealistic expectations are as likely to be attached to smaller-scale applications of technology as they are to large technology systems. One smaller-scale technological "fix" that is frequently advocated is the availability of computers in classrooms. Proponents argue that computers are valuable tools and resources for both teachers and districts. However, what frequently goes unrecognized is that, in order to live up to the expectations and serve the needs of both teachers and district administrators, computers must become an integral part of the classroom environment. Technology used as just one more add-on activity will have very little educational impact other than perhaps increasing keyboarding skills. There must also be sufficient computing resources in the classroom with links to district-level data systems that allow individual teachers to make meaningful queries in real time. If one believes that frequent measurement is critical for gauging the value added by a particular educational strategy, the ability to record and evaluate data in real time is crucial. District or state level analysis, on the other hand, would be a centralized, top-down approach that would have little use for complex data structures and would only rely on school-level technology for acquiring data. This distinction in interest between consumers of data at the different organizational levels also reflects the need for a more global understanding of the importance of the timeliness of data collection and rapid turn-around for results that are to aid decision making at the classroom level. The tension between district and classroom needs remains a troublesome barrier with respect to both turn-around time and the expected payoff for the time invested. Districts need data that are not immediately useful in the classroom (but must be collected there) and classroom teachers routinely assess their students and vary their interventions based on those assessments.

Finally, there is a serious question about the validity of the measures of student learning used at both the school and district level. Teachers often question the alignment of standardized tests with enacted curricula. There are also concerns about the consequential validity of using such tests to make high-stakes decisions about the progress of students and the retention or pay of teachers. By the same token, while a classroom assessment may be valid within that classroom, the reliability of such measures is too low to be useful for comparing student progress within a school or across a district. This is a problem that technological innovations can address, but not in purely technical terms. The design of new assessment instruments may be enabled by new delivery and recording technologies. The rapid growth of computer-adaptive testing and its immediate scoring and reporting of results represent an enormous change from the typical "fill in the bubble" examinations. Wide access to digital video has made the production of multimedia portfolios of student work a reality in many non-affluent schools. This is an area where the landscape is changing rapidly. As distance technologies improve to allow teachers to collaborate with subject-area experts, master teachers, test constructors, and others, it will become easier to work together on test quality and comparability. The social organization of classroom support structures will need to change as well, but the hope for coordination at a higher conceptual level can be realized.

III. Expected Benefits from Technological Innovations

• Enthusiasm for knowledge: Knowledge seeking and accumulation are encouraged and rewarded. A spirit of inquiry drives people. Curiosity is seen as an important asset.

• Staying ahead in knowledge: Having the drive to continue to learn and expand capabilities. Anticipating customer needs is the focus—not responding to them.

• Tight coupling of complementary skill sets: Tear down internal boundaries and operate in teams. There need to be boundary spanners in each area to make these external connections—don't make everyone a generalist.

• Iteration in activities: you never achieve perfection. Iterative improvements are the only constant. Developing core capabilities is more like gardening than building something—things need constant care and need to be turned under and replanted from time to time.

• Higher order learning: Don't just learn from operational needs. Listening too hard to current customers (problems, etc.) can blind one to the needs of potential customers in new markets. "For every activity," the manger asks, "what is the potential knowledge-building import of this action? Is it part of a larger pattern to which I should be devoting attention? If not, should it be? If it should not be, why am I doing it?"

• Leaders who listen and learn: Leaders at all levels need to be knowledgeable about the organization's technologies. Eager learners are the most effective managers. (Leonard-Barton, 1995, pp. 261-266)

Leonard-Barton discussed the importance of organizational culture for supporting knowledge management activities. It is also important to consider the involvement of critical users in the creations of a KM system. David and Leslie Willcocks Feeny (Feeny & Feeny, 1999) approach the imperatives for successful implementation of information technology-based innovation from a different direction. They argue that users of a technology must be the focus of any development or change effort, not its target. For example, a centralized reporting system for student attendance might provide detailed output that is designed by a district staffer to satisfy a state reporting requirement. Local schools, however, might have multiple additional needs that are not served by such a report. This is often the case when technology is developed by a central bureaucracy to serve its own KM needs. A KM system for student assessment will look very different if a district office designs it for its own use from the way it would look if it had been designed by groups of classroom teachers to support instruction.

The Feenys also point out that needs and requirements are not static in a rapidly changing complex system. Traditional approaches to system development are too linear to adequately address the dynamic, multidimensional nature of successful knowledge management systems. Another important characteristic of successful KM projects is the presence of high-level non-information technology supporters (e.g., managers not in the technology or computer departments). Since the adoption of KM technology is largely a social process, it is vital that senior operational managers support the project and show that they value active participation. These caveats seem to be particularly important for projects that attempt to integrate the needs of multiple levels within an organization. High-level support that focuses on an overriding need—such as improving student test scores—can be very effective in overcoming traditional barriers to cooperation that are often encountered in bureaucratic organizations. Needs that can be defined more broadly—that appeal to organization-wide norms or goals—are best articulated from upper levels of management and have the best chance of being widely accepted if they are sponsored by someone with no parochial interest in one system or another.

Others have addressed the problem of integrating human and technical systems. Karl Eric Svieby has referred to information technology as the primary hygiene factor in KM: "IT is for KM like a bathroom is for a house buyer . . . essential because without it the house is not even considered by buyers. But the bathroom is generally not the vital differentiating factor for the buyer" (O'Dell, Grayson, & Essaides, 1998). Technology is important for efficient transfer of vital knowledge, but delivers its benefits only as it supports human communication and knowledge construction.

O'Dell and her colleagues also provide important insights about some general rules-of-thumb for KM systems. They argue that "the more valuable the knowledge, the less sophisticated the technology that supports it" (O'Dell et al., 1998). For example, large data warehouses and data mining tools typically yield low-value knowledge, while a low-tech help desk delivers high-value knowledge. This is the difference between looking at pages of tabular data on the one hand and statistical analysis and advice from an expert on the other. The expert brings personal experience, context sensitivity, and technical skill and combines it with the data at hand to produce integrated knowledge as an output. The important point of this example is that the expert interprets information—data that has been systematized. It is the aggregation of information and expertise that produces knowledge.

O'Dell, Grayson, and Essaides also suggest that the low-tech/high-tech split reflects the fact that "tacit knowledge is best shared through people; explicit knowledge can be shared through machines. Or, the more tacit the knowledge, the less high-tech the solution" (O'Dell et al., 1998). District-level information systems often contain a great deal of explicit knowledge about students and schools. Tacit knowledge is that uncodified knowledge that is based on personal experience, absorption of organizational norms, and other factors. Explicit knowledge is information that has been written down or recorded in an information system. This might seem like a simplistic distinction, but it has important implications for decision-making and the reform process. This does not mean that the accumulation and transmission of tacit knowledge is not possible. Rather, it means that knowledge management systems must have imbedded in them some portion of the critical tacit knowledge needed to interpret information in the system at hand.

For instructional decisions, teachers and school-level administrators, for example, often operate on the basis of tacit knowledge about an individual student or group of students. These data are much more difficult to aggregate and transfer. The primary problem is not technical. Rather, it is the difficulty of developing relevant metrics for a wealth of anecdotal data. Another important example of the importance of tacit knowledge is the practice of using individuals as the focal point of reform efforts. School districts often use successful principals and other administrators as agents of change. Administrators that have been able to "turn a school around" are seen as a valuable commodity. The literature on KM refers to this process as one of using mobile intellectual capital to bring expert skill to bear on a particular local problem (Albert & Bradley, 1997). The value of intellectual capital is often the tacit knowledge about how one manages curricular changes or fosters a positive school climate. This process of conveying tacit knowledge about such a complex task is one example of a knowledge system.

The drawback is that tacit data is not easily transferred and successes at one location are not easily replicable to another. Some KM system-designers attempt to imbed the interpretation of experts in the outputs of the system; for example, one might present a bar graph of mean scores on a particular set of assessments. A more knowledge-rich presentation might include a representation of error bands around the mean, or provide a comparison to scores of the same students in a prior assessment. It is not merely that the information presented should be contextualized. It is important that the contextualization be done in a way that makes a valid comparison and enhances the explanatory power of the measure in question.

As the Feenys, Davenport, and others suggest, there are distinctions along the continuum from data to knowledge. What these authors do not provide is a detailed understanding of how one applies this continuum to an educational setting. In order to bring about the senior management participation that Feeny refers to above, it is necessary to establish the payoff of the investment in knowledge management at every level of analysis. If the unit of analysis is the student, then the other questions are derived from that. The analytical framework should focus on the individual. The data structures in this case must be available at the individual level and be sufficiently broad for meaningful diagnostic use. Making the linkages clear between different levels of the organization and building methods of capturing and using tacit knowledge are two characteristics that must remain in the forefront of any design effort.

Efforts to Reform School Data-Management Practices

The founders of the SIF emphasize the need for comprehensive, consistent data management from a market-driven point of view. They argue that it is impossible to provide sophisticated applications if each individual school district pursues its own data-management strategy.¹⁴ The challenge the SIF has set for itself is based on the efforts of the business information technology community to move from data-management to knowledge-management systems. While the issues involved in successful knowledge management are largely absent in the literature on educational administration and assessment, an important and growing body of work is emerging from business schools around the world. These works range from thinking about the role of experts in organizational learning (Albert & Bradley, 1997) to multi-dimensional representations of the lifecycle of knowledge (Boisot, 1998). The Financial Times recently ran a three-month series reviewing the current thinking in academia about knowledge-management systems.¹⁵ This series does an excellent job of making very complex models of organizational development and impact assessment accessible to a broad audience and has helped to inform our discussions with decision makers in Milwaukee.

The second important strand of work comes from the National Center for Education Statistics (NCES, 2000).¹⁶ This effort produced a comprehensive, standards-based model for school data system definitions. Unlike a product created by a particular vendor, district-level, state-level, and federal education administrators developed this model. Rather than being market-driven, the NCES Forum on Educational Statistics focused on the decision-making needs of school- and district-level administrators. This focus on users of data turns the traditional approach on its head. Most major school software systems—such as the offerings for National Computer Systems (NCS)—are driven by a lowest-common-denominator approach, where the package provides for the minimum needs for the maximum number of possible users. The NCES data elements, on the other hand, are developed to a high level of specificity and are intended to be extremely flexible and encompass the widest possible use.

Both of these efforts point toward the importance of standardized acquisition and the use of data for day-to-day and long-range decision making. The issue of standardization is particularly important when the focus is on evaluation. Increasing demands by outside funders and state and local agencies for data on program impact continue to raise the analytical burden placed on the district. In traditional transactional student data systems, the focus is on managing schedules and tracking attendance and grades. Reporting is designed using a top-down approach that is focused on district, state, and federal reporting requirements. What the NCES proposes is a much more flexible design that would support very fine-grained inquiry from any level of the organization.

IV. An Example of Mismatched Rationalities: The MPS Case Study

The District's Technology Strategic Plan

Classroom Management in a Decentralized Organization

Instructional time can be increased by reducing teacher time spent on classroom management tasks like attendance and grade record keeping. A single point of data entry (the teacher) should distribute that information across the school. New technology can then make available that data and integrate all other data relevant to a particular student to assist staff with decision making and the provision of services.

MPS has taken steps toward redesigning the student information database maintained at the district level. In addition, a site-based transaction-oriented database system is required. The two databases together can exchange relevant student information to provide better support.

School Management in a Decentralized Organization

Decentralization has imposed staggering new responsibilities on school management personnel at the same time that the complexity of client needs has increased. School-based technology will help address these challenges.

MPS Accountability in a Decentralized Organization

Systemic integration of reporting data at both the school and district level is required to tie together school educational plans, school accountability measure reports, district monitoring reports (MPS report card), state reports, and federal reports.¹⁸

The RFP laid out global system requirements that addressed some of the major shortcomings of the existing system. These requirements included an integrated security model, an import-export facility, and a user-friendly query-and-reporting capability. These prerequisites are important features that the legacy system lacks. The document goes on to elaborate on the current situation (at that time) and projected needs in all of the major data subsystems. The distance between the existing capabilities of the system and the projected end points were sometimes quite significant. One of the most positive elements of the RFP was the theme of data-based and data-driven decision making.

One of the major considerations to be faced in designing a database is to understand the questions that will be asked of the data. Much of the RFP is focused on improving the timely collection and reporting of student data: attendance, guidance interventions, discipline, and grades. It is also clear that data collected by the new School Management System (SMS) will be used to evaluate individuals, programs, and processes. The SMS system was purchased from a systems integrator and is being adapted to the needs of the district. The needs of the district require universal access—the ability to access a particular set of records from any location—and real-time longitudinal elements that track changes as they occur over time.

The shift from a centralized data storage and reporting system to a responsive, pervasive decision-supported system is a difficult challenge. The client/server topology recommended in the Technology Strategic Plan and required in the RFP provides a division between processing power and data accessibility that reflects the needs of actors at different levels in the system. The proposed system incorporates the two primary models of client/server system design. First, individual school administrators and teachers will be able to query the central data repository from their own computers. Second, the data queried can be downloaded to a local computer for further manipulation, or for combining with local data. The central data store might also supply "what if" datasets that allow for the development of contingency planning based on changes in important systemic variables.²⁰ Most importantly, the system being developed will allow people at a distance from the central office to become sophisticated consumers of student and system process data.

The Case of 8th Grade High Stakes

One of the important responses to the introduction of a multi-method assessment was the formation of the Middle School Principals Collaborative. The principals from 12 of the 23 middle schools initially formed this group. The group has since grown to include all middle school principals. One of the central duties of this group (in cooperation with district administrators) has been to work out the details of designing, implementing, and evaluating the proficiency standards and assessment structure of the district's middle schools. Over the course of the following two years, as the group grew to include all schools, the district and participating schools began to negotiate what metrics were to be recorded to demonstrate student proficiency. While several different methods were discussed, they all revolved around weighted averages of multiple measures.

At the district level, the units responsible for implementing the recently developed Technology Strategic Plan were building a number of new applications. Two of these efforts were of particular interest for school administration. The first is called the Student Management System (SMS). The SMS was to be a new transactional system for managing student information. This would include enrollment, attendance, grades, discipline, program participation, and other elements. SMS was intended to replace a mainframe-based system—portions of which were over 15 years old. The other important school information system to be developed was a data warehouse for student assessment and other outcome data. This system was intended to provide an analytical resource for studying programs, assessing school effectiveness, and generating reports for external accountability.

The intention was to make the SMS and the Data Warehouse available at all levels in the district. The distribution of the SMS from the district down to the classroom level was designed to accomplish a number of things. First, data entry was spread across a wider set of district personnel. Teachers would be able to take attendance in their classrooms and record assessment data directly. The teachers would also be able to check on student program status themselves. The system further allows teachers to record lesson plans and other data to capture more fine-grained data about classroom practices. Planners hoped to be able to integrate much of this data into the Data Warehouse for later analysis at higher levels of aggregation. This would make it possible to develop a better understanding of such factors as the impact of new curricula and changes in professional development.

The Data Warehouse was intended to provide local access to assessment and program participation data extracted from SMS and combined with test data from external vendors, referral data from special populations support systems, and other standalone data systems. System designers proposed developing different methods of interacting with the Data Warehouse that would support both differing data-use needs and differing technical skill levels of the system's users.

During this same period, we were working with staff members of the Office of Research and Assessment to help them develop their support for databased decision making in schools. Since much of our work is focused on the district level, we felt that it was also important to examine best practices for local data collection and manipulation. To this end, we have been working with Derek Mitchell of CRESST²¹ and the Quality School Portfolio (QSP)²² to consider the critical elements of good, school-level, information-system design.

On April 26, 1999, we participated in a district-wide review of the status of the MPS Student Management System and Data Warehouse projects. At this meeting, we also presented the QSP tool as an avenue for forming additional insights into school-level student data decision making. MPS deputy superintendents, most department heads, and representatives from the University of Wisconsin-Milwaukee and Alverno College attended this meeting. The meeting covered the actual progress to date of the ongoing design efforts, as well as the pressing data needs of Milwaukee's middle schools. Representatives of the Middle School Principals Collaborative also presented their homegrown approach for tracking student progress. This initiative was developed as a direct response to the district's inability agree on a set of proficiency metrics and provide the specifications for a data management system to deliver the needed data in a timely manner. One important outcome of this meeting has been the growing sense of urgency regarding the delivery of useful analytical data for decision making at the school level.

The QSP presentation served to provide both a background for discussing the needs of site-based decision making and an overview of information-systems planning across the district. The major areas of thrust behind QSP (school action plans, reporting processes, data-based decisions, and accountability) are by no means unique to this software package and reflect the needs of site-based decision makers in any field. We discussed how QSP might be used to accelerate the development process of the Data Warehouse by providing a conduit through which site-based managers could funnel back their own analytical models. We discussed our interest in the research on how schools store, analyze, and retrieve data in support of continuous improvement and other school reform models.

There were three significant outcomes of this meeting. First, the director of the MPS Department of Technology Services committed his staff to doing their best to get all of the middle schools wired and hooked up to the SMS system by Fall1999. He also committed the application development team to building and fielding a system for capturing and reporting the middle schools proficiency data that would be used to make promotion decisions for students who would be 8th graders in the 1999-2000 school year as soon as the specifications for this system could be finalized. The second outcome of the meeting was the decision of the Middle School Principals Collaborative to continue the development of its own school-based system in the event that the district would be unable to meet the Spring 2000 deadline. The leader of the effort expressed concern that given the other pressing technology initiatives in progress that it would be difficult for the district to build and deliver a system in such a short span of time would be difficult.

Finally, from out point of view, the most important outgrowth of this meeting was a decision by the then director of the Office of Research and Assessment for that office to build its own analytical database. This decision was driven by two different concerns. First, there was the recognition that a number of schools were under pressure to make decisions about preparing students for upcoming high-stakes assessments. Neither the existing mainframe system nor the new Data Warehouse being implemented has the capacity to provide the level of flexibility in reporting needed by schools. Second, there has been a growing realization that meeting day-to-day, operational data needs and answering questions about accountability require different interface, data-storage, and data- manipulation technologies and do transactional or compliance-focused systems. In MPS, this gap between operational and research needs has led district research staff to face the necessity for developing a different data management architecture to support these separate efforts.

At this point, there were potentially three different systems that might be in place by the end of the 1999-2000 academic year to track and report on the promotion status of 8th grade students. What was unknown at the time was that the recently elected (April 1999) school board was about to replace the superintendent. One month later, a new superintendent was in office; he replaced almost all department heads (the director of Technology Services retained his position) and one of the two deputy superintendents. In addition, the director of the Office of Research and Assessment was moved on the organizational chart to report to the director of Educational Services rather than to one of the deputy superintendents. These changes at the district level both halted the plans for the creation of a separate research data system and challenged the leadership of the technology services by replacing many of the experienced decision makers so that neither group was able to accomplish its goals for supporting the 8th grade graduation decisions. By the end of September 1999, both units formally informed middle school principals that they would not be able to provide any direct help in collecting data on student progress towards meeting graduation requirements, nor would they be able to do anything more in reporting on student retention. All middle schools were going to be responsible for notifying high schools of the status of each student at risk of not passing by the end of the summer school session.

Despite the political and technical upheaval occurring at the district level, the Middle School Principals Collaborative was continuing to meet to discuss tracking student performance and reporting student progress towards promotion to teachers and administrators. The middle school principal who had developed his own system for tracking this information formally offered his system to the group and agreed to both modify the system to fit several different school organization models and to train a small number of people to provide training support in turn to their counterparts in other middle schools. This effort was designed to use existing hardware and software and would run on either Windows or Macintosh computer systems. It was also designed with the expectation that it could be combined on a central server and managed by either the Principals Collaborative or some unit at the district level. The system went through several formal reviews and revisions and was used at the end of the school year to produce electronic files that were sent to the district's Office of Research and Assessment for review. These files were then uploaded to the Data Warehouse for use this fall to generate statistics for the district's accountability report.

The most serious dilemmas encountered by participants in these development efforts were not technical. The district had not completed its connection of the middle schools to its high-speed network, but the amount of data that needed to be communicated was trivial. The barriers all revolved around communication. As the end of the school year approached in the spring of 2000, I was asked by the manager of application development in the Technology Services department to become a formal member of the Data Warehouse development team. The team is explicitly responsible for making sure the warehouse contains the data necessary to produce the district's annual accountability report.

My involvement was partially based on my knowledge of desktop hardware and software, but the primary reason the team leader wanted me to participate was to overcome the communication barriers between the Department of Technology Services and the Research and Assessment Office. I was also the only person on the committee who directly represented school-level interests through my involvement with the two middle schools I was helping train to use the QSP tool described above. It became clear half way through the first meeting that the group had not developed a common understanding of the needs of the schools or of the limits of existing data systems in meeting the district's accountability needs. I was also surprised to find that the Data Warehouse team did not include, nor had ever included, a school administrator.

The team worked over the next two months to put together a plan for collecting the school-level data and putting it online. We also identified an alternative method for collecting the results of the summer school assessments of students who had not passed by the end of the regular academic year. The mix of technologies included installing the standalone system developed by the Middle School Principals Collaborative in all schools with 8th graders and requiring them to use it, repurposing a dormant mainframe system to collect alternative assessment data on summer school students, writing custom programming to aggregate the school data and load it into the Warehouse, and creating a new report on a discontinued report card system to provide high schools with accurate placement data for students who were or who were not retained in grade at the end of summer school. While the process was successful in the end, the resources used to respond to this emergency reporting need could have been better spent in other areas if the district's data management system had been in synch with the accountability requirements the educational system had placed on schools and students.

Preliminary Conclusions

The case study cited above points to a number of different areas for concern. First, ambitious systemic reform efforts call for radical changes in traditional school information systems. The dimensions of knowledge management strategy that Marchand maps out in Figure 1 provide a background for the difficulties MPS managers encountered when they attempted to make a sweeping overhaul of their information technology infrastructure. Even the most conservative deployment estimates of system designers in the original Technology Strategic Plan are more than two years in the past. Developers estimate that it will take another two years to complete the wiring and programming necessary to bring all 160 regular schools online. Some managers have indicated that the declining political support for the system and the ongoing burden of customization will probably lead to the development of a new system independent of the externally purchased SMS application currently being implemented by Milwaukee high schools.

The Data Warehouse is also several years behind schedule. There remain two major stumbling blocks to the development and use of the system. The first hurdle is the difficulty of producing assessment, enrollment, and program participation statistics that match those created by the existing combination of mainframe exports, SAS (statistical software) scripting, and hand manipulation used to produce the district's accountability reports. This problem can be traced back to the inherent complexity of the analytical puzzle of tracking a highly mobile population of students with an archaic information system that relies on a great deal of expert knowledge on the part of its users. This is simply a high-dimensional analytical problem that cannot be easily moved from one system to another. The other problem is that the new data system uses different data element definitions, field layouts, and formats. The new system is designed to take into consideration the improvements in methodology and changes in how one defines important school metrics such as value-added assessment and program effectiveness. Any comparison of analytical models between the two methods of producing accountability statistics requires the ability to engage in a sophisticated translation among approaches that are vastly different.

The other stumbling block the Data Warehouse faces is the method of access that school administrators and teachers will use to extract data for local analysis. The tool that was initially adopted is being used by district-level analysts in the Department of Technology Services and the Office of Research and Assessment, but was seen as too complicated for the average or casual user. Other solutions that rely on Web-based access are either not supported by the some of the desktop technologies in place, do not meet the security requirements the district must meet to protect student data, or are as complex as the tool the district is currently using.

Finally, the lack of strategic planning and strategic resource allocation continues to plague development efforts at the district level. It is possible to raise funds to support the infrastructure in providing high-speed video to several thousand classrooms, but it is difficult to find the resources to train the staff to use video effectively, or to build an evaluation system to track the impact of this technology on teaching and learning. It is the mismatch between resources available for high-profile technologies and the resources available to measure effective teaching that lead to the dilemmas encountered by Milwaukee Public Schools. The level- and unit-of- analysis problems alluded to above only exacerbate this mismatch between resource availability and needs. When both resources and the external requirements for annual accountability focus development efforts on the introduction of newer and newer technologies, school-level needs are bound to be short changed.

Notes

General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-0211. (602-965-9644). The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu .