Research and evidence in education decision-making : A comparison of results from two panCanadian studies

In this paper we compare the use of research and other evidence in the decision-making practices of two groups of education policy elites, situated in different contexts – provincial education ministries and school districts. Data are derived from two pan-Canadian studies: Galway (2006) and Sheppard, Galway, Brown and Wiens (2013). The findings show that policy decisions at the ministry level are informed primarily by political and pragmatic factors, personal and professional beliefs and local knowledge. The role of external research is shown to be relatively marginal and confined to quantitative studies and performance assessments. Decision makers at the school district level attend less to political and pragmatic influences relying more on personal beliefs, values and experiential factors supplemented by the advice of professional staff and in-house research/indicators. Results from both studies demonstrate limited reliance on external data and university-based research – the latter ranking 15th of 20 influencing factors. Education Policy Analysis Archives Vol. 23 No. 109 2 Consistent with Beck’s (1994; 1997) risk theory, we theorize that education po licy making in both contexts is influenced by both macroand micro-level factors, where choice of policy evidence is mediated by personal considerations and political risk factors. This suggests a weak policy development paradigm that is resistant to independent researchinformed evidence.


Introduction
We begin this paper with a short account of three incidents, which brought us to the realization that education policy making is a very public enterprise, where beliefs, opinions and politics are significant policy considerations.Several years ago, at the beginning of the school year, the front-page story in a major provincial newspaper bemoaned the fact that Francophone students had performed very poorly in provincial examinations and there was some speculation that the school board was placing linguistic concerns ahead of subject area competence in the recruitment of teachers.As the issue unfolded, and despite the absence of any supporting research, this 'conventional wisdom' seemed to become accepted as professional knowledge within the K-12 education community where we both held senior administrative positions.Some time later, one of us was purchasing symphony tickets when the artistic director of the orchestra came rushing out of his office with program notes for the season's inaugural concert.In the conversation that ensued, he highlighted the British Prime Minister's plan to redouble Britain's investment in music education.He then volunteered that music education was dying in Canadian schools, a message that later appeared in the symphony's program booklet and was reported in the local media.Then, a few weeks later, at a meeting of provincial education ministers, a colleague commented on a story from a regional newspaper in France.The article described the French Education Minister's vision for his country's education system, which emphasized the continuing relevance of core subjects and traditional methods, such as reading, writing and dictation.He noted that the shopkeeper who sold him the paper engaged in a lengthy monologue on what he felt was wrong with the Minister's plan.
While, in and of themselves, neither of these is particularly unusual, we were struck by the fact that all three represent very public expressions of interest in the education policy process.These incidents highlight the public standing of education in Western society; everyone seems to care about education and everyone has an opinion -from parents to shopkeepers, and yes, to symphony conductors.But, how do policymakers negotiate these often divergent and competing representations?In Canada's global-competitive environment the 'performance' of the education system seems to have become a standard against which special interests and the public measure the effectiveness of government in addressing public concerns relating to regional and national economic downturns, competitiveness and community/regional sustainability.These arrangements draw attention to education policy making as a high-stakes undertaking.However, the question of what policy makers consider to be valid evidence for policy development has not been fully examined and questions continue to be raised about the extent to which government policy is grounded in research-based knowledge.
In this article we begin with a literature review on research evidence and policy/decisionmaking in education, focusing on British, Australian and Canadian contexts.In the second section we describe the methodologies from two pan-Canadian studies that examine the use of research and other evidence in the policy formation practices of two groups of education policy elites, situated in different contexts.The first study examined the use of research and evidence by ministry level decision makers: education ministers and senior ministry bureaucrats (Galway, 2006).The second study considered the factors and evidence used by decision makers (school board trustees and superintendents) at the school district level (Sheppard, Galway, Brown & Wiens, 2013).In the last section we discuss and compare the findings and, following on earlier work in this area, offer suggestions for (1) improving the integration of research into the policy practices of education decision makers, and (2) strengthening the intersection of researchers, education practitioners, and policy makers.
Policy theorists have long argued that decisions about education should be tied more closely to research evidence.There is a growing push for ministers and other public sector policy makers to refer more of their policy decisions to independent research and these calls have been growing louder in recent years (DeBroucker & Sweetman, 2001;Fitz-Gibbon, 2003;Honig & Coburn, 2008;Naughton, 2005;Slavin, 2002Slavin, , 2004;;Thomas & Pring, 2004;Zussman, 2003).The challenge has been to effectively convey this idea to politicians, the media and the public, who often seem to base decisions on their own, different forms of evidence -popular sentiment or their own experiences and perceptions -rather than on research evidence (Levin, 2004b).Some researchers have accused policymakers of taking their lead from local pressures and political and practical considerations such as public opinion, conventional professional wisdom and media reports (e.g., Black, 2001;Cooper & Levin, 2010;Kwok, 2003;Leicester, 1999;Levin, 2003a;Ungerleider, 2003;Weissberg, 2002).Others have mentioned idiosyncratic factors such as dependencies, loyalties, associations, self-interest, personal values, beliefs, experiences, biases and fears (e.g., Honig & Coburn;2007;Reimer & McGinn, 1997;Stone, 2002).Reimer and McGinn (1997, p. xiv) describe some of the problems they experienced in their policy work with senior education decision-makers in the international context: We started as researchers, eager to get developing countries to improve their education systems.We were convinced that our skills in research and analyzing data would make an important difference.To our great dismay and chagrin, we found it difficult to get decision makers to listen to us.When they did listen, they often were critical of what we had to say and seldom followed our advice, it seemed.[…] Almost all their comments were negative, denying the validity and usefulness of the reports.… [T]hey drew up recommendations for action, many of which directly contradicted what we had recommended.
While many in the academy will identify with the frustrations expressed in this excerpt, it may be a little naive for researchers to expect policymakers to abandon their own perceptions and agendas in favour of a strictly linear technocratic approach to policy development.The seminal work of policy researchers such as Weiss (1979) and Lindblom (1980) raised questions about the direct or 'instrumental' influence of research-based evidence on policy, claiming instead that research utilization is indirect, long term and circuitous.As Levin (2001) comments, "[t]he political world is …shaped by beliefs more than facts" (p.14).A number of scholars have observed that in cases where research does impact policy, the effect normally takes place after an extended period -a phenomenon described by Weiss (1979) as the 'enlightenment' function of research.
Much of the critique of so-called "evidence-free" policy making can be theorized through the notion of a chasm between the academic and political/policy contexts.Nathan Caplan's (1979) early work on "two communities" theory suggested that researchers and policymakers operate in entirely different contexts with "different and often conflicting values, different reward systems and different languages" (p.459) Caplan (1979, p. 468) argued that the coupling of existing knowledge to user needs requires collaborative arrangements whereby the researchers and policy makers come to an understanding on "which aspects of a [policy] problem are to be decided on the basis of data-based knowledge and which ones are to be decided on the basis of non-research knowledge."Because university-based researchers operate according to their own, different accountability and reward systems, this has typically required that they follow their own research agendas while maintaining an appropriate level of independence from the government policy environment -even publicly criticizing government policy in some circumstances.As Levin (2004b) notes, "university-based research is driven by the university culture and reward system; it is primarily aimed at communication with other scholars and the rewards are related to research grants and peer recognition" (p.9).Moreover, the engagement of academics in research on specific policy problems, as in the case of commissioned research brings its own set of problems including serious ethical dilemmas relating to authenticity and independence.Strike (2006) observes that the relationship between government and researchers is "fraught with moral peril" (p.67).On one hand, there is the hope that researchers can advance social progress when they serve the needs of policy makers.On the other hand, commissioned studies are not subject to the same rigorous review standards as independent research: The research done at the behest of policymakers is often not refereed.It is often "quick and dirty" -done under time pressures and poorly funded.Moreover, policymakers often ask researchers to evaluate or assist in the implementation of policies and programs in which the policymakers have a vested interest.They may be uninterested in funding research that might fail to support favoured programs.The questions may be structured so as to assume the merits of current policy and policy aspirations.Most troubling is that, at the state level, those policymakers who fund research often have influence over the budget and reputation of the public universities in their states.Researchers may fear offending public officials, lest their access to funds and influence dry up.University administrators may be reluctant to have their faculties assume a critical or oppositional role, and policymakers are likely to look for sympathetic researchers.(Strike, 2006, p. 67) Government actors, for their part, have charged researchers with ignoring important policy areas in education (Blunkett, 2000;Galway, 2006).In the past, some critics have suggested that, with the possible exception of commissioned research, much of the independently-produced university research may be inaccessible or does not provide the answers decision makers are looking for when deciding among policy options (Cooper & Levin, 2010;Lindblom & Woodhouse, 1993;Neilson, 2001;Pring, 2000;Stone, 2002).In the mid-1990s an intense debate began in the UK around the quality and utilization of educational research.The ruckus was propelled by an article in the Times Education Supplement, wherein Alan Smithers (1995) argued that the only criterion for judging the value of educational research should be the extent of its usefulness to policy makers and practitioners (Dunseath, 2000).Hargreaves (1996, p. 7) later declared that education research was 'second-rate' and decried its inability to "make a serious contribution to fundamental theory or knowledge," charging instead that it cluttered up obscure academic journals, was irrelevant to practice and was uncoordinated with antecedent research.Two controversial British reviews of the quality and utilization of educational research followed.Tooley and Darby (1998) evaluated the nature and quality of educational research in Britain; based on a sampling of articles published in four British educational journals, the researchers concluded that only about one third of the articles could be said to have been based on sound research practice.Citing weaknesses in methodology, policy relevance, objectivity and quality of argument, the authors suggested that education research must become nonpartisan, pragmatic and relevant in its design, include a plan for dissemination, and be guided by an overall research strategy.The Tooley and Darby study was followed closely by a second review by Hillage, Pearson, Anderson and Tamkin (1998).Hillage and his colleagues put forward the widely contested view that the research-policy divide was due, in large part to an education research agenda that was 'supplydriven'; that is, the research originates with researchers and not policymakers and practitioners, a model described by Landry and his colleagues as the 'science push model' (Landry et al., 2001).They argued further that education research tended to be backward looking and evaluative, rather than forward-looking and exploratory, resulting in negligible policy impact.This idea overlaps with the notion that education research lacked a 'cumulative character;' whereby new studies help to construct a knowledge base by building effectively on earlier work (Davies 1999;Hammersley, 1998;Ince, 2008).
While some of the observations in the Tooley and Darby and Hillage reports have been without serious challenge several have been contested.Hodgkinson (1998) criticized Tooley and Darby (1998) for making a fundamental distinction between empirical research and non-empirical research and for evaluating qualitative studies by predominantly quantitative, positivist-realist criteria.He further charged that the review ignored basic ontological and epistemological precepts of research conducted in post-industrial traditions; for example, the fact that many qualitative researchers do not claim that their work is generalizable in a statistical sense, nor do they claim objectivity.
Several reviews describing the state of education research in Australia have been less critical of the research community.Bates (2001) undertook a meta-analysis of research reviews in Great Britain and Australia over the preceding ten-year period.He concluded that the Australian reviews, in comparison to those conducted in Great Britain, tended to deliver much more positive assessments of the value of education research, both in terms of its international reputation and its use among teachers and policymakers.Ribbins, Bates and Gunter (2003) account for this difference, in part, by the nature of the reviews.While the Australian work has tended to begin with educational practice to inform research, the UK reviews have tended to begin with the research and work forward to practice.
Between 1999 and 2000 five reviews focusing on research and practice in the Australian context were undertaken as part of an overall initiative sponsored by the Australian Department of Education, Training and Youth Affairs (DETYA) and the Australian Research Council.The first concluded that much of the Australian research base was devoted to issues of teaching and learning (Holbrook, Ainley, Bourke, Owen, McKenzie, Mission & Johnson, 2000).Connections between university researchers and the educational community were considered to be robust and included ministry-faculty research committees, the publication of research in teacher-focused publications and the secondment of academics to ministry staff positions.There was additional evidence, based on claims by practitioners, that findings from education research had permeated practice, although the review acknowledged that research utilization was often mediated through individual policy advocates.In a second study, Figgis, Zubrick, Butorac and Anderson (2000) examined the impact of research on policy, starting at the policy/program level and working backwards to determine whether selected policies/programs were founded on a research base.Their work put forward the notion that practitioners and policymakers were connected to research through 'connecting webs' or networks with multiple nodes.They suggested that the application of research in an educational context requires human intervention of motivated actors and the active marketing of research knowledge by its producers.A third study examined teacher practice to determine the extent to which it was reflective of a research base and concluded that research was, in fact, a substantial influence among several 'proximal' sources of influence on teaching practice (McMeniman, Cumming, Wilson, Stevenson & Sim, 2000).The fourth study examined citation rates of Australian researchers in international journals (Phelan, Anderson & Bourke, 2000).This bibliometric analysis demonstrated that Australian research was extensively cited in a range of knowledge domains including pedagogy, administration and policy development.A fifth study examined the influence of research on policy decision-making within the vocational education and training system in Australia (Selby-Smith, 1999).Of the senior education decision makers surveyed, two thirds claimed that strategic and political considerations were most influential in decision-making while research was often used to justify or support a decision already taken, variously described as the symbolic use of research (Landry et al., 2001), use of research to mobilize support (Nutley et al., 2003;Weiss, 1998) and use of research to build political support, (Honig & Coburn, 2008).Selby Smith (1999) made several conclusions with respect to the authenticity of research for policy decision-making in education.First, research does impact decision-making, but it exerts its influence on the broader long-term education agendaextending to the so-called 'big ideas' in education.He suggested that researchers be modest in their expectations about the impact of research on decision-making noting that "research is only one of a number of sources of information available to decision-makers and information from all sources is only one of many inputs into the decision-making process" (p.556).Second, because decision makers bring their own experiences and backgrounds to bear on their practice, "the extent of research impact is affected by the knowledge, attitudes and experience of decision-makers" (p.565).Finally, to be influential, research must be conducted in an environment of collaboration where robust linkages exist between decision makers and researchers at all stages of the research process.An "emphasis on linkages rather than dissemination alone increases the mutual responsibility of the parties, since enduring linkages are based on sustained mutual esteem, and understanding of the potential contribution of each party, and a commitment to collaboration for the good of the system" (p.565).
Prior to the mid-2000s there were relatively few references to the quality of educational research within the Canadian context.Most of the discussion is framed around the scope of the research effort, which was generally found to be somewhat lacking.Levin (2003a), for example, acknowledged pockets of strength, but claimed that the educational research enterprise generally, is small, short-term, narrow and fragmented.Although the research effort within faculties of education was growing, he concluded that there were few organized mechanisms to enable education research to be translated into knowledge for policy.Applied research in Canada, according to Levin (ibid.)had remained within the purview of government departments and third parties, such as privately funded institutes and think tanks.Willinsky (2003) similarly observed that the influence of academic research in Canada "pales before the impact of the government's own sources of information, whether from politicians, bureaucrats, or government professionals" (p.2).He noted, however, that the availability of academic articles through electronic means has facilitated research access among policy advisors and decision makers and this has acted as a "counterforce to policymakers' reliance on a small number of academic consultants as gatekeepers and sources for research" (p. 1).According to Landry, et al., (2001) part of the problem was that the body of work on research utilization was disparate -scattered across different social science disciplines and recorded in disconnected journals.There were also problems with narrow and overly prescriptive definitions of research utilization which had come to be associated with direct instrumental or linear uses of research rather than conceptualizing research use as a broader, more interactive and circuitous process.
In 2003, deBroucker and Sweetman compiled a series of papers by well-known Canadian educators and researchers with the goal of examining and evaluating the impact of Canadian education research impact to date.The authors drew three conclusions about the state of education research as policy evidence.First, compared to the health-care sector in Canada, education research is used to a far lesser extent; second, there is a pressing need to involve a broader set of people than researchers in considering research issues, and finally, there is a need to expand the indicator set to include a broader range of measures, including those indicators that are more difficult to measure.Shortly thereafter the then federal (Liberal) government established the Canadian Council on Learning (CCL) as an independent national agency with a mandate to produce, promote and disseminate educational research (Mason, 2012).Before funding was cut by a successor government in 2009, the CCL had made good progress in drawing attention to several key policy issues in education by completing systematic reviews and disseminating the research findings through a number of channels, most notably, an on-line repository of thematic reviews, research and data.Cooper et al. (2009) acknowledge a strengthening of capacity and cite several provincial initiatives aimed at increasing support for evidence-based policy and practice, but point out that integrated pan-Canadian strategies are more difficult because education in Canada is not a federal responsibility.
From the perspective of raising awareness of research in education, the fallout from these reviews and from the discussions and debates that followed was both positive and progressive.More recent work in this field has left behind the divisive debates around research quality and attempted to focus on defining and structuring the field of evidence-informed practice as a research area (Cooper et al., 2009;Nutley et al., 2003) defining and promoting high quality and highly relevant educational research (Earl, 2011;James & Pollard, 2008;Pollard, 2007Pollard, , 2008) ) and understanding the different forms and trajectories of research use both in education and other public sectors (Nutley et al., 2008;Rosenblatt & Tseng, 2010).A major advance in promoting this agenda in the UK was the Teaching and Learning Research Program (TLRC), a large-scale effort to spearhead a series of projects and thematic analyses intended to engage users in research-informed practice both to improve research quality and generate new knowledge (James & Pollard, 2011;Pollard, 2011).Pollard (2007) remarked that the program was intended to support high quality and highly relevant research that could "satisfy the ambition of 'Pasteur's Quadrant'" (p.640), a reference to Stokes (1997) categorization describing a form of research that marries pure basic research to applied research.The ten-year £40 million program, which concluded in 2011, generated a series of ten principles of effective pedagogy in four broad areas and is widely regarded to have been successful in translating academic research into userfriendly forms and engaging users in culturally and politically current educational discourse (Earl, 2011;Galvin, 2008;Hogan, 2011;James & Pollard, 2011;Pollard, 2008).
Other researchers have been working on understanding the different forms and pathways of research use.While those working in this field have argued about different perspectives on research use (for example, rationalist-decisionist/rational linear versus more ambiguous and interactive approaches) there is some agreement that there is a diversity of avenues through which research permeates and influences educational practice and pedagogy (Nutley et al., 2008).Landry et al. (2001) examined the utilization of social science research in Canada across four disciplines and conceptualized the process as cumulative rather than instrumental; dependant on the engagement of researchers with end users rather than the quality of the research products.Drawing on research in the social care field and cross-sector reviews, Nutley and her colleagues describe several models of research use, noting that both rational-linear and interactive perspectives could be found operating in each.This suggests that there is value in conceptualizing such models and perspectives as iterative and overlapping; their use being appropriate at different times and in different contexts (Nutley et al, 2008;Nutley et al., 2009).
A number of recent articles have begun to concentrate on research utilization as 'evidence into practice' with emphasis on the intersection of researchers and practitioners (Earl, 2011;Edwards et al., 2007;Nutley et al., 2003) including linkage mechanisms, research mediation efforts and new dissemination strategies (Cooper, 2013;Landry et al., 2001;Martin et al., 2010;Qi & Levin, 2013).We have already mentioned the work of Landry and his colleagues.Based on their study of more than 1200 social science researchers in Canada, they concluded that the most important determinants of utilization are "the mechanisms linking the researchers to the users, the users' context and the publication assets of the researchers" (p.334).Edwards et al. (2007) similarly found that establishing reciprocal relationships with users as co-researchers could enhance practitioner value for research.
As the field has grown and expanded from health to education and to other social sectors new terminology has come into usage: knowledge transfer, evidence-informed policy, data-driven decision-making and knowledge brokering, to name a few.Knowledge mobilization (KM) has been touted as a useful all-encompassing term because it conveys the notion of direction instead of random interaction and it "embodies the idea that the use of knowledge is a social process, not just an intellectual task" (Cooper et al., 2009, p.166).The more recent North American work on education research utilization/research impact has framed the connection of research and policy environments in terms of KM and knowledge brokering with governments, schools and school districts, and community organizations (Cooper, 2013(Cooper, , 2014;;Cooper & Levin, 2010;Cooper et al., 2009;Jbilou, Amara & Landry, 2007;Knowles et al., 2013;Qi & Levin, 2013, Wilson, Lavis, Traverse & Rourke, 2010;Yohalem & Tseng, 2015).Honig and Coburn (2008) were interested in which forms of evidence, if any, were used in school district offices and the purposes for which the evidence was used.They conducted a comprehensive review of relevant research literature but found a large proportion of the material in circulation were advocacy pieces leaving only about fifty peer-reviewed publications.The major findings suggest that the process of evidence use is complex, highly political and based largely on local knowledge.Along the same vein, Cooper and Levin (2010) surveyed a sample of superintendents, principals and other leaders to study how research use was perceived, supported and integrated into the operations and activities of school districts and schools across Canada.They found a strong interest in research use, but a weak and loosely coupled institutional research capacity that depended on "a few interested people rather than being deeply embedded in daily practice" (p.61).
Since the early 2000s, there has been a proliferation of interest in KM including more publications, new networks of researchers and practitioners, and more outlets and spaces for interchange on evidence and practice, but also a number of lingering problems (Cooper et al., 2009;Qi & Levin, 2013).Notwithstanding the contribution of Nutley et al. (2003) in suggesting a framework for organizing the evidence-into-practice research agenda, Cooper and her colleagues complain that that conceptually and theoretically, the field is still in disarray, there are too few empirical studies on most of the issues, and efforts to increase research impact have not been properly assessed (Cooper et al., 2009).Some of the Canadian work by this group and others at Queen's University and the University of Toronto is attempting to address some of these problems.In a recent study of research mediation in Canada, Cooper (2013) takes on the unenviable task of disentangling some of the cross-sectorial confusion around KM terminology and knowledge brokering models, hypothesizing about how research brokering organizations (RBOs) are situated in the education sector and creating a typology of Canadian RBOs (e.g., the Atlantic Institute for Market Studies and the Fraser Institute) in education.RBOs are grouped into four categories -government, not-for-profit, for-profit and member RBOs -and are defined by their orientation along seven common dimensions.This particular work is an important step towards redressing some of the conceptual problems in defining the field of KM raised by Cooper and her colleagues in their earlier work (Cooper et al., 2009).In an effort to assess KM efforts among knowledge producing organizations, Qi and Levin (2009) studied websites from 100 institutions and agencies (including governments, universities and faculties of education, and other organizations) in Canada and other English-speaking countries.The researchers judged the research sharing effort to be low to modest, product-oriented and passive (unidirectional) with no opportunity for two-way interaction.Cooper ( 2014) similarly looked at on-line strategies and social media as a linkage mechanism between producers and users of research.The websites of forty-four Canadian RBOs were examined to assess their on-line and social media dissemination strategies.The social media effort was considered to be modest -mostly small-scale Facebook pages and Twitter networks -and there were few avenues for two-way interchange.
The foregoing discussion highlights the thorny questions of what constitutes evidence for education policy and how education research becomes introduced into the policy process.Researchers and social policy critics still have divergent and incomplete conceptualizations about the relationship between research and practice.While this is an intriguing area, it has not been fully interrogated, the literature is still somewhat disconnected, and there are gaps in our knowledge base.As Cooper et al. (2009) have observed, after all of the activity related to KM "very little is known about how practice organizations, whether governments or schools and school systems, find, share and use research."The analysis presented in this paper attempts to address some of these blank spots by addressing three interconnected research questions: (1) what are the factors and influences that drive policy decision-making in Canadian education ministries as compared with school boards, (2) how important is each category of evidence in relation to the others, and ( 3) to what extent is education research valued as 'authentic' policy evidence?

Research Design The 2006 Study
The 2006 study was a two-phase mixed-methods design with two participant groups: Canadian provincial deputy ministers and equivalents (survey/questionnaires) and current or former provincial ministers of education (in-depth, semi-structured interviews).

Survey method.
A population of 95 senior education public servants was identified using the Canadian Education Association Handbook and contact lists provided by the Council of Ministers of Education, Canada (CMEC) and provincial departments of education.A questionnaire developed for this research was field tested and administered to deputy ministers and assistant deputy ministers of education and their equivalents working in Canadian ministries of education using direct and electronic mail.A total of 51 valid questionnaires were returned for a response rate of 53 percent (Quebec was not represented).Demographic information such as age, gender, educational background and years of experience in public administration was also collected.In Phase II, in-depth, semi-structured interviews were conducted with eight (8) participants -five (5) current and three (3) former provincial ministers of education.Participants were selected and contacted based on language (English-speaking), probable access, and proximity to the researcher.
Participants were asked to recall three instances of education policy decisions or policy advice they had provided to ministers during the previous 12 months (N=148).They were then asked to rate, on a seven point Likert scale, the extent to which each of a series of factors influenced the corresponding decision or recommendation.They were then asked to recall examples of research or indicators they had recently used in their work.
Data analysis of numeric and categorical data involved the generation of means and standard deviations for Likert scale items and frequency distributions for categorical data, in accordance with generally established principles and consistent with those procedures outlined by Dillman (1999), Fowler (1998), and Henerson, Morris and Fitz-Gibbon (1987).
Interview method.Interview questions focused on participant representations of research and evidence in education decision-making, in particular (1) how they conceptualize education research; (2) the extent to which research and other forms of evidence are utilized in decision-making; and, (3) perceptions of accessibility, relevance, utilization and value of education research.The interview protocol followed the procedures described by Bok van Kammen and Stouthamer-Loeber (1998), Mangione (1998), Marshall and Rossman (1989), and Maxwell (1998).All, except one telephone interview, were face-to-face, private interviews and all were recorded and transcribed.Interviews ranged in length from approximately 60 minutes to more than two hours with an average interview length of approximately 90 minutes.
The analysis of interview data was modeled after the principles and strategies for interview data described by Creswell (1998) andQuinn Patton (1987).First, there was a general review of the printed transcripts of each of the interviews and the combined interview data was grouped according to question.The transcripts were then analyzed for identifying themes, findings and relationships among the data, and notes are recorded in the margins of these texts.The second step of the analysis process involved examining the transcripts, and over a period of time, thinking about the data sets, and identifying coherent and important examples, themes and patterns in the data, a procedure described by Quinn Patton (1987) as content analysis.These ideas and relationships were further developed in the form of longer notes and memos that were sorted into categories and themes, forming the preliminary findings.As the organization of findings and themes matured, the third step of the process involved selecting samples of text from the interviews that were illustrative and confirmatory of the finding or theme.In cases where the same idea was expressed repeatedly, participant representations were summarized and paraphrased or used as exemplars of the consensus of the group.The final data reduction step involved creating displays of information using diagrams showing interconnections among ideas and findings while identifying larger global themes running through the representations of the participant groups.Finally the interview transcriptions were reread carefully to validate or discard these preliminary findings.Following from Silverman's (2001) discussion on avoiding anecdotalism, the data were searched for negative exemplars in the data that could be argued to falsify or refute the findings and the categories, themes and findings were revised accordingly.

The 2012 Study
For the 2012 study we collected data through the use of a survey/questionnaire and a series of focus group interviews.There were three participant groups representing Anglophone school boards in all Canadian provinces and territories: (1) school board trustees, (2) school district superintendents, and (3) executive directors of each provincial school boards association who were also members of the Canadian School Boards Association executive board.
Participants.The selection of participants for both the survey/questionnaire and interviews was conducted with sensitivity to gender, experience, ethnicity and regional geography.A total of 369 individuals from across Canada (331 school board trustees and 38 district superintendents) completed questionnaires.As well, we conducted 21 focus group sessions with school board trustees, school district superintendents, and members of the Canadian School Boards Association executive board.
Survey method.The survey/questionnaire was modelled after the one used in the 2006 study, with some variations.Prior to the distribution of the questionnaire, it was pilot tested with a convenience sample of six individuals who had extensive experience as either a superintendent of education or trustee.The research protocol was also presented a t a session with interested members of the Canadian Association for the Study of Educational Administration (CASEA) at its 2010 Congress.Minor adjustments to the research instruments were made on the basis of the resulting feedback from these validation exercises.Each participant was asked to identify specific policy-relevant decisions or recommendations that s/he made during the past year.They were then asked to rate, on a seven point Likert scale, the extent to which each of a series of factors influenced the corresponding decision or recommendation.Our analysis of the survey data is limited to descriptive measures such as frequency counts and the calculation of mean scores and standard deviations.For the survey analysis, because of low return rates among school district superintendents, it was not possible to report separately for school board trustees and superintendents.Data from school board trustees and district superintendents are pooled.
Focus group method.The focus group protocol, and analysis of the focus group data, followed the procedures described by Meijer, Verloop, and Beijaard (2002), Merriam (2009), and Miles and Huberman (1994).Focus group sessions were conducted separately for trustees and superintendents.To increase the validity of findings, we took considerable care to collect data from school board members and school district superintendents from all provinces and territories and from those holding office with the CSBA Board.We also triangulated data collection such that each of research team members conducted a number of individual 60-90 minute focus group sessions (5 to 10 participants per session) that occurred at different times and in differing locations over the eighteen-month data collection period.Each focus group session was recorded and transcribed.To ensure anonymity of each participant we assigned pseudonyms as required.We analyzed the transcribed focus group data through the application of a qualitative data analysis program (QDA Miner, 2011).Using this program, we coded separate data segments for school board trustees, executive directors of provincial school board associations and school district superintendents of education.Subsequently, we retrieved these coded segments and saved them as separate reports that facilitated the development of specific themes.Also, we were able to develop frequency counts and the percentage of respondents who provided particular responses.It is important to note, however, that all focus group participants did not respond to each question.As a consequence, our reporting that a specific percentage of the focus group participants held a particular view, for instance, should not be interpreted that a remaining percentage held a contrary view; if no percentage is reported for the contrary view, it means the others were silent on the issue.
Limitations.Although we employed a Likert scale instrument in addition to focus group interviews to collect data, this work is primarily dependent on naturalistic and interpretive methodologies (Cohen, Manion & Morrison, 2011) that acknowledge personal involvement of the researchers and anticipates subjectivity.Data from interviews are actively constituted by the intervention of the researcher with research participants.They would otherwise not exi st.From a theoretical perspective the responses of research participants in this study are seen more as representations of perspectives and 'moral forms' than simple reports on reality (Silverman, 2001).Given the pan-Canadian scope of this study and holding realistic expectations of how much time we could reasonably expect of study participants at any one point in time, we utilized a questionnaire/survey as a means of collecting individual participant perspectives relating to his or her role as either a trustee or superintendent.Our analysis of the questionnaire/survey data is entirely limited to the determination of frequency counts and mean scores.Accordingly, we consider this work to be solely naturalistic and interpretive.
Ethical review.The research protocol for both studies was approved in advance by a university interdisciplinary committee on ethics in human research and followed the principles outlined in the Tri-Council Policy Statement on Ethical Conduct for Research involving Humans (TCPS2).

Research Findings
The 2006 Study Profile of participants: Senior bureaucrats.The data show that senior executive positions in Canadian education ministries are reserved for highly educated, mature individuals with extensive experience in education or public administration, but not necessarily within schools, school boards or other educational institutions outside of government.Deputy and assistant deputy minister positions tend to be male-dominated -females accounted for less than 40% of education executives -and are occupied by mid to late career bureaucrats, who have completed more than one advanced degree.As a group, senior bureaucrats have a broad range of administrative experience both within and outside of government, yet are generally new to their current positions as education executives.More than half the senior officials surveyed were in their current jobs for three or fewer years (Table 1).Categories of policy issues.We first sought to understand the general categories of education policy issues addressed by policy actors at the government (ministry of education) level.Senior education executives were asked to describe three policy-relevant decisions or recommendations they recall having made during the previous 12 months1 and to rate the significance of each policy decision or recommendation on a seven point Likert scale.Data describing 'perceived significance' are means and standard deviations where +3 represented 'very significant,' 0 was neutral, and -3 represented 'not at all significant.Participants reported a total of 148 policy decisions or recommendations that could be assigned to 16 broad policy categories2 .Results showing the frequency cited and perceived significance, by policy category, are presented in Table 2.
Senior education bureaucrats reported their primary focus was on corporate matterseducational finance and resourcing policy, accountability policy and strategic planning.These three areas accounted for almost half of the policy decisions/recommendations recalled.The areas least cited were: school bussing, professional development, special education, social development and equity and education research, together comprising less than seven percent of all policy decision and/or recommendations reported.Program and policy issues, such as literacy and post-secondary education, aboriginal education, as well as finance and resource issues were judged to be the most significant.Also rated as highly significant were a series of related issues -programming, curriculum, instruction and assessment, as well as educational planning, governance and accountability.Evidence and policy practice: Survey data.Data presented in Table 3 were generated from participant representations of their recent policymaking practice.Again, senior public servants were asked to describe three policy-relevant decisions or recommendations that they recall having made during the previous 12 months, 3 and in the context of a given list of factors/evidence, they were asked to indicate the relative influence of each of the factors on each decision or recommendation.Data were pooled and, for each factor, expressed as the mean plus or minus the standard deviation where -3 represented "did not at all influence", 0 was neutral and 3 represented "influenced to a great extent".
Findings reveal that greatest importance was placed on internal advice provided by staff and colleagues, followed by past experience, personal and professional beliefs, corporate/pragmatic evidence4 , and political-democratic considerations5 .Within the broad category of political considerations the data show that the representations of special interest groups was a moderately important consideration while claims about the influence of public opinion and the representations of the mass media were less definitive.Among the factors that were rejected by senior bureaucrats as having any substantial impact on policy development practice were anecdotal stories, the representations of individuals, and externally produced research.Research generally, and particularly, external research studies were considered to be relatively minor considerations as evidence for policy decisions or recommendations, although it is highly probable that at least some reference to research would be incorporated in some forms of staff advice.
Participants were then asked to list the factors or evidence that should be considered in developing education policy.The results generated a list of 151 suggested factors covering a wide range of potential forms of evidence; these were assigned to one of 20 response categories we developed during the course of data analysis.Senior bureaucrats placed highest value on corporate/pragmatic evidence -including financial considerations, efficiency and policy implementation factors, and the direction of government's strategic plan -together comprising more than 42 percent of the factors cited.In order of descending frequency, the following categories of evidence were listed as important to policy decision-making: (a) financial considerations/ efficiency/capacity to implement (21%); (b) government/department's strategic goals/direction (21%); (c) education research or indicators (15%); (d) impact on students/clients (13%); (e) potential for social and economic development of the province (9%), and; (f) representations/pressure from special interest/stakeholder groups (5%).Among the categories of evidence least cited were (a) professional and personal knowledge, values and beliefs; (b) government/minister's reputation; (c) equity and fairness considerations; (e) linguistic and cultural considerations; (f) avoidance of negative media attention; (g) jurisdictional considerations; and, (h) the minister's direction, together comprising about eight percent (8%) of factors/evidence cited.Profile of participants: Ministers of Education.The participant group of ministers of education comprised two females and six males, all over 45 years of age and all veteran politicians, each having served more than two terms in office.In general, however, they had a relatively short tenure in their education portfolios -four or fewer years.
Evidence and policy practice: Interview data.Ministers of education were asked a series of questions relating to their policy development practice and the role of research in decision-making.The findings suggest that political decision makers vary considerably in their perspectives, but generally have very limited direct knowledge of education research.Some ministers responded to questions that probed their understanding of research-based evidence by referring to the importance of experimental studies, using terms like 'double -blinded' and 'solid data'.Others associated valid research with program evaluations.Most participants expressed the view that valid research should be objective and involve "many cases" so it can be generalizable to the population.Ministers' responses to questions that probed their understanding of the relationship of current educational research to policy decisions were divergent and confused, suggesting that the political decision makers in this study have a very superficial awareness of current research and its potential use for educational decision-making.Ministers were clearer in their representations of the value of external or university-based research and expressed negative views with respect to accessibility.The consensus view of the participants interviewed was that university research is obscure and inaccessible to policymakers.The following exchange illustrates this point: M2: Once the research is being done it's just sitting there; it's staying in academia.It's not moving out into government, into communities; into business, as it should be.Now I've seen some instances where it's been successfully done, but I have to admit, that from the year and a half that I've been here I'm struggling very hard to find an instance where it's been useful to me in my work.R: So, in order for it to be more useful you'd say it would have to be more relevant to the issues that are being dealt with by government?M2: You see it could be that the research they are doing is relevant, but we don't know about it.They're just not letting us see the results of their research; they're not saying to us, "This is what we have done.Could it be of use to you?" We're not involved, I guess.That's what [the problem] would be.
The knowledge generated by university research was also seen to be inconsistent with the information needs of government.Participants questioned the relevance of research to the day-today policy questions that are important to education ministers.As one minister noted, education research is 'not a main reason for change'.The following excerpts are typical of these representations: M7: Well, I haven't had a chance to look at anything [university researchers] are doing so I really don't know, but I do find that sometimes, just from past experiences, that the research coming out of (names university) is not always; well not the research, but some of the conclusions they draw from their research is not always applicable….M2: Well, I don't really know about it, to be truthful.As I was saying earlier that's one of the things that is most surprising to me; that with the amount of research that is done at the university, how little discussion or impact it's had on my work as the Minister in this portfolio.
Participants also perceived a wide separation between the education research community and government departments where education policies are conceived.Participants observed that education researchers operate in different operational environment with a vastly different set of goals and orientations.Some ministers charged researchers with criticizing the education policy directions of government instead of offering productive suggestions or engaging in constructive debate.Several ministers expressed the view that university researchers tend to either ignore important policy problems or simplify those problems by remaining focused at a theoretical or ideological level, while failing to account for real-life policy pressures: M1: I guess that what I find with university research is that those who are doing it appear to be somewhat removed from the immediacy of it all; it's not engaged, as it were, and [university researchers are] somewhat reluctant to engage those who are involved in a particular issue on a daily basis.That's being pretty general; that's not the case with all university research.M3: University research?Well, I don't know… we had all the mathematics people look at the mathematics curriculum in (province).They did that in 1998, 1999, 2000, as a result of our kids doing poorly in mathematics and so we had all these pundits.They're all experts.They put together this curriculum.The curriculum came out and…some of the very professors who were involved in the generation of that [curriculum] came out and criticized it in the papers.So I don't understand where they're coming from.
In each of the interviews, ministers were then asked to identify from a list of possible influences or evidence, the five factors they consider to be most influential when making education policy decisions.Collectively, participants identified a total of 42 factors 8 represented in Table 5.These data support the findings from the in-depth interviews.While ministers claim to value a broad range of policy evidence, there was a collective preference for policy evidence that falls within five principal categories of evidence: (1) political-democratic factors, which we frame as inclusive of public opinion, the representations of individuals, special interests/advocacy groups and the representations of the media; (2) pragmatic or practical considerations, such as financial concerns and governance ideology; (3) global (macro-level) competitive influences and pressuresin particular, evidence such as educational rankings or standings based on large-scale national and/or international assessments; (4) staff advice; and (5) personal and/or professional beliefs, values and experiences.Ministers compared with senior education bureaucrats.There are striking similarities in how politicians and bureaucrats represent the importance of different categories of policy evidence, but there are differences in the relative standing or 'ranking' of each category.In Table 6 the key policy drivers for politicians and senior public servants are grouped into five main categories, ranked, in order of influence.As noted, for politicians the most significant influence on policy development seems to be based largely on broadly defined 8 In two cases participants identified six (6) factors 9 The following categories were collapsed: Cabinet/Government's overall direction and the Department's strategic plan 10 The following categories were collapsed: representations of individual students, parents or constituents; representations of business groups and the private sector; representations of individual teachers/instructors/administrator, and; Representations of special interest stakeholder groups. 11The following categories were collapsed: past experiences with related issues and personal or professional beliefs and values political-democratic influences: public opinion, advocacy, the mass media and other political and pragmatic pressures, including political ideology.While senior bureaucrats also consider political-democratic considerations to be important, these factors are ranked lower.Deputies and assistant deputies place highest value on the internal advice provided by staff and colleagues, followed by past experience, personal and professional beliefs and values, corporate and pragmatic evidence and, political considerations.While politicians deem staff advice to be important, it was considered to be less important than corporate and pragmatic considerations (such as financial exigencies and governance ideology), and global (macro-level) competitive pressures.The 2012 Study Profile of participants: School trustees and superintendents.Most survey respondents (62%) were from Western Canada, with 22 percent from the Central region (Ontario and Quebec), and 13% from the Atlantic Provinces.Three percent of respondents did not identify their location.The balance of male and female respondents was similar with slightly more females (53%) than males.The age of respondents varied from under 30 years (1.5%) to over 60 (33%) with the majority (67%) over the age of 50.About two thirds of respondents reported that they had served in the role of trustee for more than five years.School trustee occupations were quite varied (e.g.stay-at-home parents, skilled tradespersons, farmers, fishers, educators, office workers, business owners, health care professionals, clerks, civil servants, managers/executives and retirees).
Evidence and policy practice: Survey data.We first sought to understand the general categories of school board decisions made by trustees in the fulfillment of their governance roles.Their responses, summarized in Table 7 provide a glimpse of the decision -making priorities of school boards across the country.It appears that the primary focus of school board policy decision-making relates to programs, school development, and assessment.The second most commonly identified category was actual engagement in policy development and governance.Other common foci included the provision of safe caring schools, transport ation, school restructuring, human resources, technological innovation, strategic planning, engaging families and community, budgeting, and resource allocation.
Participants were also asked to indicate their views on a series of statements relating to governance and decision-making by using a seven-point Likert scale (strongly disagree to strongly agree).Most trustees/superintendents (56%) either agreed or strongly agreed that "major decisions in this school district are based on data," while 33 percent were less definitive in their responses indicating that they" somewhat agree" with the statement.These responses suggest that although 3.9 1.6 1.6 1.3 1.0 0.7 100 reliance on data may not be an established approach to all decision-making in school boards, a majority of participants feel that it has become a meaningful aspect of their decision-making practice.To obtain a more specific understanding of the factors that influence school board decision-making, we presented trustees with 20 potential factors/influences and asked them to indicate on a seven-point scale, the extent to which each has had an effect on specific decisions or recommendations that they have made as trustees (Table 8).The principal six factors are as follows: (1) personal or professional beliefs and values, (2) potential to directly influence student outcomes/student learning, (3) advice of district staff and/or colleagues, (4) the school board's strategic plan and (5) past experience.Trustees and superintendents rated the following six factors as having the least influence on decision-making: (1) representations of business/private sector, (2) pressure from special interest or lobby groups, (3) a situation or event someone told you about, (4) pressure from government (ministry/department of education), ( 5) public opinion/avoidance of negative media attention, and (6) university-based research.Analysis of the data by province showed very little variance in the factors and evidence that influence school board decision-making across jurisdictions.
Other factors and evidence with a moderate level of influence include (1) the potential to advance the reputation of the school district or board, (2) research or indicators prepared in-house by district staff, and (3) budget considerations while factors such as the representations of teacher and administrators, business/private sector interests, ministry of education assessments and other external research were seen as having only marginal influence on school board decisions.Evidence and policy practice: Focus groups.The focus-group protocol was designed as an in-depth probe of the factors influencing policy decision-making.Accordingly, we were able to assess this measure separately for school board trustees and superintendents .In each focus group session we presented each participant with a list of potential factors and influences on decision-making (similar, but not identical to the list of factors provided on the questionnaire).The list was a slightly modified form of the instrument used in Galway's (2006) study.We asked trustees and superintendents to identify and rank, in order of importance, the top five factors/influences on their own policy practice.Although the two lists were not identical, findings from the focus group data with trustees and superintendents generally paralleled the findings based on the survey data with some minor variations.
Trustees tended to place more value on the "representations of students, parents and constituents" than did superintendents while superintendents ranked "past experience" and "external research" somewhat higher than did trustees.Otherwise there was close alignment between the rankings of trustees and superintendents with the following factors all ranked high: (1) advice of the CEO (superintendent) and senior staff, (2) school board's own research and indicators, (3) budget considerations, and (4) government/department of education's overall direction.The data derived from the focus group responses were somewhat similar to the questionnaire data, with some notable exceptions.For example, the questionnaire data showed the influence of the ministries of education to be a relatively minor influence while the focus group participants considered it to be much more significant.The lowest ranked factors and evidence for both groups were (1) representations of individual teachers and administrators, (2) representations of special interest groups and (3) factors such as "anecdotal reports" and "economic goals for the region." With regard to research utilization, our focus group sessions with school board members in some regions revealed a somewhat different perspective -not entirely consistent with the pan-Canadian survey data.In Quebec, commissionaires (trustees) noted that frequently they work closely with university researchers in determining policies and programs.The following response to our question, "What kind of research, if any, do you value when making policy decisions?"illustrates: It depends on the policy.Our board has a policy on policy development.We don't research that very much.But when we have our policy for dealing with children with special needs then we use a lot of research of our university partners, McGill and Concordia.Doing our policy on digital citizenship…we took a lot of research from the Internet [and] we went out to people who implemented things before.Our policy on fieldtrips …is based on the law and insurance requirements, so it really depends on the nature of the policy, but we will do a fair amount of research, as appropriate.
Another focus group of school board trustees in Saskatchewan echoed these views; their perspective suggesting that they place higher value on research evidence for decision-making than would be indicated by the data derived from the questionnaire: I think that's what validates.I mean that's where…you can separate the politics and [decision-making] becomes evidence-based….When you do that then you've got substance based on evidence and people can stand behind it.It's not so much public opinion and private opinion; it's more a matter of some form of evidence-base.
In one of the Atlantic Provinces, one participant responded, "[i]t depends on the issue.If it is something to do with student achievement then I think we look at university research, scholarly work and evidence more closely than we ever have."These responses suggest that school board trustees, like education ministers and senior education bureaucrats place high value on political considerations in decision-making, such as the representations of students, parents and constituents.Their responses also correspond to the representations of ministry of education policy-makers in that they place high priority on staff advice and past experience.The responses of the participants from the two studies are also similar in that they say they place substantial importance on personal and professional beliefs and values, corporate and pragmatic influences, such as budget considerations and the government's "overall direction." The value placed on external research as policy evidence was mostly consistent among the participants in the two studies.Generally, the questionnaire data indicates a low level of research utilization among Canadian education policy elites.The consensus view of the political decisionmakers interviewed in the 2006 study is that external research, and particularly university research is somewhat obscure and inaccessible to policymakers -a scenario where education researchers make a very marginal contribution to education policy decisions.The representations of school board trustees and superintendents were generally consistent with this perspective, although less definitive.
The mean scale score13 representing the influence of university research on policy decision-making within ministries and school boards was -1.36 and -1.08, respectively.Based on focus group data, there was some variation in these findings for certain regions of the Country where school board trustees and superintendents said they relied on universities to undertake policy research and were likely to utilize research evidence if they were aware of its existence and deemed it relevant to their circumstances.It also appears that for policy decision-making, not all sources of research evidence are given equal sway.In both studies, internal research and indicators prepared either by ministry staff or school district staff was ranked the same -in the middle of the positive range of responses (mean scale scores of 0.86 and 0.86, respectively).The data show that external (university-based) research, however, did not rank highly in either study.

Discussion
Theorizing Low Research Impact Several educational researchers have described the educational landscape in Canada over the past quarter century as turbulent and uncertain as policymakers have tried to negotiate the problems of fewer students, unstable budgets, and new expectations for schools (Levin, 2003b;Levin & Riffel, 1997;Sheppard, Galway, Brown & Wiens, 2013;Ungerleider, 2003).There is widespread agreement that decision-making in education now occurs under a microscope; it has become highly politicized and subject to intense media attention (Levin, 2004a;Lingard & Rawolle, 2004;Thomson, 2004;Ungerleider, 2004).As Beck (1997) and Giddens (1990;1994;2003) argue, cultural changes and the growing complexity of society creates greater levels of uncertainty, resulting in people becoming more restless, taking less for granted, and initiating action to reduce their perceptions of risk.Canadians are passionate about education, and are unabashed about raising the bar for what they expect from schools.But public expectations for education are constantly shifting, creating policy challenges and dilemmas for decision makers.More than in the past, citizens and education lobby groups are better positioned with knowledge and expertise; they are better equipped to challenge institutional experts within school boards and ministries of education and have a greater capacity to lobby governments and force change.
As Guppy and Davies (1999, p. 278) point out, parents and educational interest groups are "less deferential towards the educational establishment and are more likely to feel entitled to challenge experts."Based on a review of two decades of polling evidence, Guppy and Davies (1999) showed that public confidence in Canadian schooling had declined in all population subgroups.A more recent survey, undertaken by the Canadian Education Association, showed that between 1984 and 2007 the percentage of Canadians who reported a great deal or fair amount of confidence in schools dropped by 31 percentage points from 76 to 45 percent (Canadian Education Association, 2007).This dissatisfaction has been expressed in myriad ways -from a startling increase in private tutoring (Davis, Aurini & Quirke, 2002) -particularly among wealthy Canadians (Cain, 2013) -to a rise in the number of legal cases against school boards (Sheppard, 2012).A 2012 Ipsos-Reid poll revealed that Canadians gave a poor rating to the "overall state" of the public elementary school system.More than half of those polled (58%) held the view that a private school education is better than a public school education and about two thirds (63%) said they would choose a private school education for their children, if money were no concern.
At the same time, virtually all regions of the country have been experiencing major demographic changes leading to shifts in student populations, profound enrolment decline and greater diversity in the school-aged population.Census data show that Canada recorded the highest population growth among G8 nations in 2013-14; however, net international migration accounted for almost two-thirds (66.5%) of the total increase.In the last several decades there have also been profound changes in proportion of Canadians leaving smaller towns and settlements for urban areas -from 62% in 1951to 81% in 2011(Statistics Canada, 2011).Within the country the population share of some regions, notably Quebec and the Atlantic Provinces has decreased over the past three decades while Ontario and the Western Provinces has increased (Statistics Canada, 2014).For, example, over the twenty-year period, 1993-94 to 2013-14, the number of school-aged children in the Atlantic region fell by 30 percent or about 1.6 percent per year and enrolment losses between one and two percent per year are expected for at least the next decade (Galway, 2015).
Schools are among the first public institutions to be impacted by socio-cultural and demographic changes.Addressing sustained enrolment changes (influx or decline) and responding to more diverse student populations require considered policy responses.Other social tensions and technological changes are similarly first felt in schools and colleges, leaving policy actors to negotiate everything from school closures and realignment to safe school protocols and diversity policies, often with a restless and impatient public.Ministries of education and school boards are expected to base their policy decisions on the best evidence available to them.The findings from these studies, however, portray the policy development process as ambiguous and political; driven by informal evidence that is only loosely connected to the educational research community.
These findings are consistent with the theoretical orientations of Beck (1994;1997), Majone (1989), Kingdon (1995), Levin (2003b;2004b) and Stone (2002), all of whom conceptualize public policy-making as divergent, uncertain and risky -driven by personal, political, and experiential influences.In a politicized environment, independent university research, especially that which might be tacitly critical of existing policy, may either represent additional political risk or be perceived as inaccessible or irrelevant to settling urgent policy questions.Under such conditions, the values, reward systems and accountabilities against which social scientists operate are likely to differ substantively from those of system-level decision makers, thereby creating a policy environment that might be antagonistic to external research.These conditions may well force policymakers to set aside research-informed policy (which is normally a longer-term proposition) in order to attend to less risky and more immediate pragmatic and democratic/political considerations.

Making Research Matter More
These studies raise questions about whether the existing decision-making approaches are useful in advancing a sustainable policy agenda for Canadian schools.They also underscore the importance of undertaking policy work in consideration of the best available research.Decision makers need ready access to research evidence to guide their practice in the public sphere, but perhaps more pointedly, to refute incorrect information and hearsay that can so often form the substance of unqualified demands for educational change.Following Edwards et al. (2007), Jbilou et al. (2007) Landry et al. (2001), Nutley et al. (2008) and others, we propose that to attenuate the risk associated with greater reliance on education research stronger connections must be fostered between the research community and several levels of the bureaucracy within school districts and ministries of education.In short, to become a more significant consideration in the policy advice senior advisors provide to politicians, research must become more familiar and trustworthy.
The importance of trust and confidence has been discussed elsewhere in the literature (e.g., Nutley et al., 2009;Tseng, 2012).One of the key questions raised by researchers is how do policymakers and other information consumers sort through competing claims in guiding their practice (Lubinenski, Scott & Debray, 2014).While the studies compared here have focused on the standing of research and other factors as policy evidence, several researchers have begun to conceptualize research utilization as a social process: "one that unfolds within a complex ecology of relationships, organization and political and policy contexts" (Yohalem & Tseng, 2015).One recent US study of school board decision makers found that the value of research could not be decoupled from trustworthiness of its sources and the motives of the person conveying the evidence (Asen, Gurke, Donners, Solomon, & Gumm, 2012).Rosenblatt and Tseng (2010, p. 201) have argued for the need to understand the importance of the demand side of research-policy equation, including "user settings and perspectives, political, economic and social contexts and the various uses of research." Based on large-sample studies in the Canadian health care sector several strategies have been suggested to improve reciprocal understandings between the research and policy communities.Landry et al. (2001) identified several determinants of research utilization including the mechanisms linking researchers to users and the user's operational context.Jbilou et al. (2007) added that communication networks, partnerships and other links between researchers and decision-makers could help effect a cultural change in governmental organizations -whereby there is greater support and value for research into practice.Programs such as the former TLRP in the UK and the former Community-University Research Alliance (CURA) program in Canada represented efforts to support alliances between community organizations and universities to foster innovative research, training and the creation of new knowledge (James & Pollard, 2011;Social Sciences and Humanities Research Council, 2013).More to the issue of engendering trust and confidence, Edwards et al. (2007) describes such programs in terms of a reciprocity or co-construction of research-based knowledge that serves to connect organizational boundaries and strengthen the warrant for research studies.In the following we draw upon the findings from these studies and the work of others researching this area to suggest some modest proposals for how such evidence could become a more integral aspect of education policy and practice.
Engaging ministry and school board advisors.Following on Jbilou et al. (2007), we see research collaborations, research forums and regular formal (or informal) meetings with mid-and senior-level school board and ministry officials as important mechanisms for interchanges between knowledge producers and the education policy community.There are few opportunities for external research to reach policymakers, except through the bureaucracy and, to lesser degree, through the efforts of special interest groups and the mass media.Reimer and McGinn (1997) wrote about the difficulty they experienced in communicating research findings to government decision makers.As a dissemination strategy, direct access to school board trustees and/or ministers of education is problematic for several reasons.Access to policy elites is difficult at the best of times and focusing the attention of elected officials on complex teaching and learning issues is a longer-term proposition.Even in instances where researchers are successful in engaging elected decision makers, due to their crowded schedules, such meetings are likely to be short, infrequent and limited in their long-term impact.A more effective strategy would involve the active engagement of mid to senior policy professionals who prepare the information -informal e-mails or briefings, decision notes and policy papersthat is circulated 'up the line.' Bah Diallo (2001), a former minister of education in Guinea, says that politicians can only effectively initiate policy change with the cooperation of the bureaucracy.Administrators identify for ministers not only what is needed to effect change, but also the appropriate pathways to implementation: Administration constitutes the passage oblige, the gateway, the bridging link between policy and practice, between political intent and the hard reality of day-to-day business.As such it plays the difficult role of reconciling the need to maintain the system and the need to lead the changes and the reforms.
[…] administrators have a tremendous influence on the system processes and the policy-makers' capacity to decide because administrators can influence action where it counts the most, that is, in the field, where things happen.(p.22) The policy actors who participated in the 2006 and the 2012 study indicated they place high value on knowledge produced or validated by insiders -ministry and/or school district policy staff, a finding consistent with Majone's (1989) suggestion that the policy analyst plays a pivotal role in determining what evidence is considered by policy elites and the manner in which policy advice is constructed.While mid-level bureaucrats and district program specialists represent the entry point for external knowledge, senior bureaucrats mediate the cascade of information to elected decision makers by distilling and validating evidence and bringing together research with other forms of evidence to generate 'trustworthy' staff advice.

Intersecting communities of practice.
There are several small-scale efforts that could be undertaken to create intersections between the individual communities of practice of researchers and policy advisors/practitioners.Formal and informal professional exchanges cou ld create reciprocal space for discussion of policy issues and problems and the kinds of research that could contribute to public discourse and/or policy deliberations.In our earlier work on researcher perceptions of evidence used in policy making (Galway, 2006), participants referenced the challenges in sustaining regular professional exchanges between academics, senior public servants and ministry advisors, but also their effectiveness in connecting the three communities of practice.
Another point of interchange could be the establishment of rotating research chairs within education ministries and faculties of education -whereby mid-to-senior level public servants could become immersed in the research culture of the university while academics could experience the policy environment of government.There is already some precedent for this kind of exchange.In at least one Canadian province there is a standing research fellowship that enables senior executives of the government to work in a university setting and to teach and research in their professional discipline.Through the fellowship, there have been similar moves in the opposite direction, whereby faculty members take up residence in government departments.For example, the former Canadian Council on Learning had conceived a plan centered on the establishment of 'researchers in residence' as a means of infusing research into policy discussions at the level of education ministries 14 .Under such an arrangement the ministry and/or the university would underwrite or partially underwrite the associated costs of the exchange; the participants would then focus on one or more policy questions occupying the district/ministry agenda.This research would imply that these and similar opportunities for researcher-bureaucrat exchange should be encouraged as a means of creating common understandings among school districts, government departments and education faculties.
Another approach formalizes the involvement of educational leaders in research through the promotion of advanced professional degrees aimed at administrators, bureaucrats and senior educators.Now reasonably well established in many faculties, the professional doctorate requires that candidates undertake research relating to their professional practice.While relatively few professional doctorates are awarded each year in relation to the number of professional staff working in education, the connection of policy analysts and program specialists to the research community through professional learning opportunities would appear to be an effective, albeit small-scale means of infusing research into policy practice.School boards, post-secondary institutions and ministries of education could provide incentives to encourage professional staff to undertake research-based professional degrees as a means of professional development.Financial incentives such as grants to assist with tuition and research expenses, sabbatical leaves for senior public servants and/or promotion incentives could entice more senior education officials to begin doctoral or professional masters programs.Moreover, the design of many professional doctorate programs affords the student considerable flexibility.The completion of degrees, partially through Web-based study, the ability to study part-time over extended periods, and the offering of compressed didactic components using itinerant professorial instruction all contribute to the professional degree option as a viable means of making research more familiar to the policy community.
Moderating the critical stance of the academy.We have argued that research will become more entrenched in policy development if it can become more familiar and less risky for policy makers.The data from the 2006 study show that ministers of education repres ented researchers as sometimes adversarial -either indifferent to the pressing educational problems on the government's policy agenda, or too often, critical of education policies and practices.Recognizing the cautions of Pirrie, Adamson and Humes (2010) regarding the spread of bureaucratic rationality and the responsibilities of academics to speak truth to power and preserve freedom in academic life, we believe some consideration should be given to the ways in which academics represent their research in the public sphere, and how they frame policy critiques.While certain traditions of inquiry lend themselves to critical commentary on social institutions and processes (Creswell, 1998), the findings from this study suggest that sustained critique of policy decisions may serve to position academic research as foreign and threatening to policymakers.When academic researchers publicly argue for a strict (research) evidenceinformed approach to policy, they may inadvertently overlook the political dynamic of a policy situation, and situate themselves as adversaries.As Caplan (1979, p. 461) writes , [t]he need for reciprocal relationships between knowledge producers and knowledge users in policy -making is clear, but the problem of achieving effective interaction of this sort necessarily involves value and ideological dimensions as well as technical ones."The studies discussed here show that decision-makers make choices in the context of an overwhelming and widely diverse set of public expectations.Politicians, in some instances, seem to identify those in the academy with risk and uncertainty.For some researchers -particularly younger scholars -the contextual background to the policy process may represent uncharted territory.In order for education research to begin to make a difference in the policy decisions of policymakers, academics may wish to evaluate how far critical approaches have advanced the influence of research on the public policy agenda and consider different ways and means of drawing attention t o their policy concerns.Researchers may better situate themselves to influence policy decisions by accounting for the ambiguities and political pressures of decision-making in their critique, while advocating for reasoned, research-informed policy.
Balancing academic interests with research impact.The findings from these studies also suggest that attempts to stimulate more policy-relevant research will also come with the need to re-evaluate traditional ideas about the academic utility of research in relation to its practical value to consumers and policy makers.From the perspective of academic progression, researchers have good reason to doubt the value of undertaking commissioned or government/district-sponsored research.Levin (2004b, p. 9) has remarked that, "universitybased research is driven by the university culture and reward system; it is primarily aimed at communication with other scholars and the rewards are related to research grants and peer recognition."In her case study of a large Canadian community-university research collaboration, Tapp-Neville (2015) reported that researchers were concerned that certain forms of collaborative and commissioned research have a lower standing than independent research as evidence of contribution to scholarly inquiry.If university communities, even passively assign lower status to commissioned policy work, they may, unintentionally, restrict the likelihood that collaborative research programs will attract young, able scholars.Moreover, if commissioned education research has limited standing within the academic community it might only be conducted in the 'sidelines' -something done to sustain a stream of research funding or otherwise left to researchers who have advanced in their careers and have already esta blished themselves as credible academics.
From the perspective of enhancing research impact, there is reason to be concerned about a culture that draws a distinction between the importance of research that is needed and valued by ministries of education and school districts and other 'legitimate' research that is valued within the academy.For education research to become a more visible presence in day-to-day policy discussions, and for the quality and amount of commissioned research to improve, consumer-driven research needs be recognized for the purposes of academic reputation and tenure-track promotion.Finally, if commissioned research is assigned a lower status with respect to promotion and tenure, it may unintentionally restrict the likelihood that such research opportunities will attract young, able scholars.This topic will not be further addressed here, although it is important area for future research.

Conclusions
On balance, the findings from the 2012 study of school board trustees and superintendents are generally consistent with those of the 2006 study of education ministers and senior education bureaucrats.We found that elected officials and their advisors in both contexts use a variety of forms of evidence in their decision-making, including non-formal sources of evidence such as local (internal) knowledge and experience.Policy elites, particularly at the ministry level, tend to place considerable reliance on political-democratic forms of policy evidence, followed by staff advice, corporate and pragmatic considerations, personal and professional beliefs and values and past experience.Both participant groups also confirm the influence of formal sources of information, such as research evidence; however the form of such research matters, and appears to be largely limited to internally generated research and indicators.
Findings from the 2006 study show that ministry policymakers express largely negative perspectives about the availability and value of certain forms of research-based knowledge.The ministers interviewed were either oblivious to the research undertaken in universities or deprecating in their appraisal of its availability and utility.Both political actors and senior bureaucrats say that they place greatest value on research-based evidence that originates or has been vetted by staff and policy advisors within the ministries themselves.Trustees and superintendents expressed a similar perspective placing greater value on the school board's own research and indicators, while downplaying university research as less influential to school board decision-making -this form of evidence ranking 15th of 20 possible decision-making factors.Still, school board trustees and superintendents were more predisposed to the notion of "data-driven" decision-making and some focus group participants mentioned collaboration with university researchers; however, these references were few and limited in scope.
We conceptualize use of low research impact partly in terms of trust vs. risk avoidance.In both policy contexts -provincial departments of education and school boards -we conclude that decision-making is ambiguous and risky.Policies and practices are developed and enacted in an uncertain policy context, often with diverse and impatient constituencies.Following Beck (1994Beck ( , 1997)), in uncertain times, decision-makers value knowledge that is familiar and emerges from their own community.Such knowledge, while frequently originating in external studies, is contextualized against other policy evidence, and judgments are made regarding its relative value within the larger policy context.This information -now repackaged in the form of staff advice -becomes privileged and trusted by ministers and senior bureaucrats as authentic knowledge.The problem inherent in a decision-making paradigm that is overly dependent on informal, populist and experiential knowledge is that it is haphazard and fundamentally reactive, thereby inviting continuous reform and readjustment.
The evidence and arguments presented herein support calls from others for a more open, collaborative, familiar and trusted relationship between education practitioners and researchers.Such a change could begin to immunize schools against frequent changes in practices and policies until they have been referred for evaluation against the best information available.Following on other recent work in this area, several tangible strategies are suggested that might be helpful in effecting such a change.These representations also raise policy issues for administrators in some Canadian universities, both from the perspective of enhancing the formal and informal linkages between the research and policy communities, and in re-imagining the standing of user-defined (commissioned) research within the academy.If education research is to become a staple at the table of policy decision makers, those in authority within the academy would do well to examine and debate these issues and, where necessary, make appropriate adjustments.

Table 1
Senior Bureaucrats' Years of Experience in Present Position and in any AdministrativePosition (N=51)

Table 2
Categories, Frequency and Perceived Significance of Policy Decisions Recalled by Senior Bureaucrats (N=148)

Table 3
Senior Bureaucrats' Perceptions of the Relative Influence of Different Factors/Evidence on their Recent Policy-relevant Decisions/Recommendations (N=148)Finally, participants were asked to provide some examples of research or indicators that they remembered or had used in their policy development practice.The findings presented in Table4suggest that senior public servants value survey research and large-scale assessments over qualitative research.The vast majority of examples of research and indicators that they remember or have used in policy development are large-scale quantitative studies such as provincial, national and international assessments, educational finance studies, demographic research and labour market outcome studies.

Table 5
Education Ministers' Perceptions of Most Important Influences or Evidence on which to base Policy Decisions (N=8)

Table 6
Comparison and Ranking of Most Important Categories of Evidence for Education Policy Decisions, Education Ministers and Senior Education Bureaucrats

Table 7
Categories and Frequency of Policy Decisions Recalled by School Trustees and Superintendents

Table 8 :
School Board Trustee and Superintendent Perceptions of the Relative Influence of Different Factors/Evidence on their Recent Policy-relevant Decisions/Recommendations 12