Assessing Organizational Efforts to Mobilize Research Knowledge in Education

This paper presents a framework for evaluating efforts by organizations to share research as exemplified on their websites, then reports the result of an evaluation of these efforts in 100 organizations. The result shows that the overall research sharing efforts of these organizations are modest and not well aligned to evidence on effective practice in this area. Organizations tend to devote more efforts to making products available while interpersonal strategies, though more effective, are less used. Most efforts involve one way communication to potential users. The paper concludes with suggestions for practice and for further research.


Introduction
Improved communication1 of research findings beyond the scholarly community is an important and growing focus of concern in all disciplines.The growing interest in this work around the world is motivated in part by the call for greater accountability in research investments (e.g.Shavelson & Towne, 2002;British Academy, 2008), but even more by growing appreciation of the importance of research for enhancing the quality of public services such as health, justice, and education (Levin, 2004(Levin, , 2010;;Nutley, Walter & Davies, 2007).
Although many different terms2 have been used to refer to the various processes that connect research to policy and practice, we refer to this enterprise as 'knowledge mobilization' [KM] because that phrase well reflects the intentional and active elements required to make knowledge relevant to policy and practice (Cooper, Levin & Campbell, 2009).We also recognize, as described well by Nutley et al. (2007), that there are different meanings ascribed to all the key terms involved.For example, people differ on what counts as 'research', while most analysts also recognize that the 'use' of research also has multiple dimensions, from changing what people know to affecting how people think to influencing how they work (Weiss, 1979, Knott & Wildavsky, 1980, Nutley, Walter & Davies, 2007).Ideas of mobilizing or using research knowledge are also contested politically, with some worrying that this movement is really intended to narrow and control professional work (e.g.Ball, 2001).Yet despite these debates, virtually everyone would be in favor of stronger connections between research and practice in some ways, even if there is disagreement about how those connections should work.
Our conceptual model for research knowledge mobilization is drawn from Levin (2011) and is shown in Figure 1.It draws attention to 3 connected 'contexts' -the production of research, the organizations or settings where it may be used, and the various bodies or processes that mediate between research production and use.These 'contexts' are not necessarily synonymous with particular organizations, since many organizations are involved in two or even all three of these functions.

Figure 1. Research Knowledge Mobilization
Many different kinds of organizations are involved in attempting to mobilize research knowledge.These include organizations with primary interests in each of the three areas, or in all three -from universities to school systems to policy bodies to a whole range of intermediary organizations such as think tanks, unions, and lobby groups (Cooper, 2012).Active involvement in doing research is not a requirement for active engagement in mobilizing research knowledge.
In practice, a wide range of knowledge mobilization efforts can be found in these various organizations.Nutley, Walter and Davis (2009) define five kinds of research communication activities: tailored dissemination, interaction, social influence, facilitation, and reminders and incentives.Each of these categories can include a range of specific actions, any or all of which can be employed in different ways by various organizations.There are many case studies of these efforts and occasional evaluations of them (the journal Evidence & Policy is one source of many examples).
Yet at an aggregate level little is known about the ways that different kinds of organizations attempt to mobilize research knowledge.
Decades ago, Klein & Gwaltney (1991) had already described three types of research dissemination.The first type they call spread, meaning one-way information distribution.Most research publications and other audio and visual materials belong to this type.Within the category, syntheses and interpretations of research knowledge can be viewed as preferable to the reporting of individual studies.The second category they call choice, which focuses on helping knowledge users acquire different sources of information by providing more options.If the first type is passive in nature, the second type is more reactive and responsive.The third type of dissemination is exchange, which centers on interactions between and among people and organizations and emphasizes the "multidirectional flow of information" (Klein & Gwaltney, 1991, p.246) through networks, feedback system, and so on.Many professional and governmental organizations in education have used this approach to build and maintain networks to better support use of research: "It is viewed as interactive dissemination and is congruent with recent education trends such as restructuring schools to empower teachers and students and accountability goals to identify and share what works" (Klein & Gwaltney, 1991, p.246).
As an illustration of how ideas recycle across time, the Klein & Gwaltney typology is very similar to a more recent typology of producer push, user pull and exchange (Lavis et al., 2003) or a very similar proposed by Amara et al. (2004).Moreover, all these conceptualizations fit well with the basic model as shown in Figure 1.

The World Wide Web and Research Communication
It is no exaggeration to say that approaches to research communication have been transformed by the development of the World Wide Web.The web makes vast amounts of information available to every user and has become the primary vehicle for information sharing in almost every field of activity (Chavkin & Chavkin, 2008;Lederbogen & Trebbe, 2003).Whereas twenty or thirty years ago the challenge was to find research material, much of which was not readily accessible, the situation is now reversed, and the main challenge is to sort out what information is valid and reliable from the masses available.For example, a study by our team (suppressed for blind review) has found that in almost every category of education there are literally thousands of resources readily available through the internet claiming to be based on or informed by research.
Internet technologies "have opened the door to an additional and much broader range of dissemination possibilities and have generated entirely new forms of content that must be shared" (Association of American Universities et al., 2009, p. 2).These include not only new forms of product dissemination, but also much more interactive strategies (Lee & Garvin, 2003;Nutley, et al., 2007).In recent years social media are further changing research communication in important ways, though that development is not a focus of this paper.
For most people, the web is the now the first and most important way to look for research knowledge.Moreover, some web embedded features such as electronic storage and search engines create opportunities for organizations to document, arrange and present research knowledge with "extraordinary flexibility, depth, and power" (Association of American Universities, et al., 2009, p.3) and potentially reach more audiences at relatively low cost.
For purpose of research dissemination, websites are essential components of the information architecture in organizations (Baker, 2005).Institutions of all kinds are putting considerable effort into creating, maintaining, and promoting websites.In the developed world virtually every organization involved in research communication now has its own website, often quite extensive, and many organizations in middle and lower income countries also have a web presence.In the case of universities, there may be many sites as sub-units such as faculties and centers, and even individual faculty members maintain their own sites.
Websites offer many features for organizations to share their knowledge such as inventories of research that can be structured in multiple ways.All kinds of materials including print, video, audio or presentation materials can be organized and posted on the web.Researchers may offer tutorials about their own work.Institutions sponsor web-casts of academic conferences, archives of seminars, lectures and so on.Moreover, search engines are powerful tools for users to locate information.A website is also a platform both to support and supplement various other dissemination practices (Supyuenyong, et al., 2009).Specifically, the web allows many new communication vehicles such as bulletin boards, audio and video conferencing, listservs, wikis, social media, feeds and others which collectively are permanently changing the nature of information dissemination.
However, so far, little empirical work has been done concerning research communication on institutional websites.This project is intended to address this gap in the literature by focusing on the research sharing practices of organizations based on the activities shown on their websites.The work explores the way websites are and could be used for the purpose of research communication, as well as finding patterns of research dissemination strategies across institutions.Because there was no instrument available to analyze these practices, we developed a system of indicators through which institutional websites can be understood, measured, compared and improved.
Of course, websites do not necessarily contain or reflect all of an organization's dissemination practices; other research dissemination activities could be going on that are not reported on the web.Yet given their centrality to this work, websites do provide much relevant information (Selvanathan, 2007).It therefore seems reasonable to analyze organizational websites as a way of understanding their strategies for research dissemination.It is also important to note that this project was not aimed at assessing the quality of the websites themselves, such as their aesthetics or functionality; its focus is solely on what they tell us about overall knowledge mobilization and research dissemination activities of institutions.

Method
We began by looking at research dissemination activities on the websites of a large number of organizations.Our search was deliberately broad, starting with educational institutions in the various parts of our conceptual framework such as universities, education ministries, school districts, and other education stakeholders.For comparative purposes we also looked at organizations beyond the education sector.Because the health sector has done the most work on mobilizing research knowledge, we looked at a number of health organizations, and because our model draws attention to the importance of intermediary organizations in the dissemination of research, we also looked at several organizations of that type.As a Canadian research team we focused on Canadian organizations but also looked at organizations in other countries for comparative purposes.Our choice of organizations included systematic elements, such as looking at most or all education ministries in Canada, or a selection of the largest school districts, but we also considered organizations that we had heard about in various ways, or that were linked to some of the sites we initially reviewed.
The analyses were done by a team of graduate students and faculty.We began with an open and subjective team discussion of what we regarded as good dissemination activity on various sites.Several rounds of such discussion, coupled with the reading we were doing in the literature on knowledge mobilization and research dissemination, led us to develop a common set of criteria for evaluating the sites, through which each site was awarded points for various features.

Evaluation Framework
Over time we added elements to our evaluative frame.As each version of the scoring system was developed, members of the team re-evaluated sites using the new criteria.In each round, some sites were evaluated by at least two team members or at team meetings so we could judge the reliability of the ratings.At team meetings we would discuss whether the new ratings captured all the important features, or whether the criteria for ratings were sufficiently clear.We also continued reading the literature on research mobilization so that our analysis could be informed by current evidence.Over several months of such evaluation, the system was steadily refined.Some categories were clarified, other subdivided, and in other cases the criteria within the categories were made more specific.Our current formulation is in Figure 2 below.

Figure 2. Conceptual Framework for Evaluating Knowledge Mobilization on Organization Websites
The three boxes on the left represent the three major approaches to research mobilization, inductively derived (but also, though at the time unknown to us, very similar to Klein & Gwaltney, 1991), while the categories inside the right hand box are quality indicators for each approach.We explain each of these categories in more detail.
Products are attempts to communicate knowledge in written or audiovisual forms, such as journal articles, reports, books, syntheses, bulletins, research newsletters, or video or audio clips.Events are activities such as lectures, conferences, seminars, workshops, symposia, or exhibitions where the aim is to share research with practitioners and users.As such, this category excludes events that are primarily focused on the exchange of research findings within the research community, such as academic conferences.Events in this category can be real or virtual.Networks refer to efforts to build lasting relationships among and between knowledge producers and potential users.The primary interest is in connections that actively share knowledge between researchers and target audiences in a way that is sustained over time and highly interactive (reference removed for blind review).Again, the focus is on interaction that extends outside the research community itself.
The first category is relatively self-explanatory.Often, research communication is equated with products of different kinds such as articles or reports.The range of such products can be enormous, from larger reports to executive summaries, to syntheses, research briefs, and a variety of products written for particular audiences.More recently, in conjunction with the capacities of the

Multiple types
Ease of use

Events
Networks web, research products have included audio and video materials, blogs, podcasts and even more recently the use of social media.The use of events to communicate research findings is also longstanding.Many professional development activities, in many disciplines, could be considered as research communication through events.
While both these categories of activity have merit, empirical work has made it increasingly clear that the active take-up of research findings requires more than either products or events.From a variety of perspectives, ongoing interaction has been shown to be the most powerful factor in shaping people's acceptance and use of research evidence (Nutley et al., 2007;Hemsley-Brown & Sharp, 2003;Wenger, 1998;Masuda et al., 2009).For example, Lavis et al. (2005) highlight the interaction between researchers and research users as follows: Research on the transfer of research knowledge to and its uptake by managerial and policy audiences has demonstrated that interaction between researchers and these audiences (or representative members of these audiences) appears to be important to explaining why some types of research knowledge are used and not others (p.226).However interaction need not just be between researchers and others.Since the strongest influence on professional practice is the views of peers (Mitton et al., 2007), interaction among professionals is also a powerful vehicle for mobilizing research knowledge.[reference suppressed] emphasize that "collaborative projects and networking are vital to KM [knowledge mobilization] as system improvement depends on the multiple contributions among different stakeholder groups" (p.355).Mathiassen and Vogelsang (2005) distinguish between networks and networking.The former stresses the structure while the latter term emphasizes relationship building in terms of trust and collaboration.To assess relationships, we could consider their content, form and intensity.Content here includes services, information, or even emotions that are generated through interpersonal contact.Form refers to how long these relationships last and how close these relationships are.Intensity considers how frequently the information exchange happens (Chen & Mohamed, 2007).
Within these three categories of KM work, as our analysis proceeded, we began to identify particular elements that might be predictive of greater effectiveness.Those eventually became the four categories in the right hand box in Figure 2.

Different Types of Products, Events and Networks
Empirical research on knowledge mobilization (Cordingley, 2008;Landry et al., 2001) suggests that the use of multiple types of products, events and networks can effectively promote research utilization.Thus, the use of multiple types of KM strategies by organizations is one indicator for assessing the overall KM performance.
Although networks are important, they are not automatically valuable.Williams & Bailey (2002) stress that it is essential to differentiate different types of networks to study their dynamics and provide appropriate support, as the concept itself is used indiscriminately in the current literature.Different types of networks are emerging such as knowledge networks (Clark, 1998), innovation networks (Pittaway et al, 2004), research networks (Gunn, 2002, Williams & Bailey, 2002), and so on.Some networks are intended to disseminate knowledge to the wider public, while some networks are limited to the communication among their members.
Assessing the value of networks requires knowledge about their characteristics, functions and purposes as well as the degree of interactivity and frequency of communication.In this project, we considered three types: internal networks, external vertical networks, and external horizontal networks.Internal networks (Creech, 2005) refer to informal or structured network of people in organizations with the purpose of sharing information to achieve KM objectives.External vertical networks (Fliaster, 2008) indicate that the organization has a clear KM-focused mandate in working with other institutions.External horizontal networks (Fliaster, 2008) describes networks made up of people within and across fields.

Focus of Audience
Research utilization literature (e.g.Nutley et al., 2007;Mitton et al., 2007) tends to divide potential users into three categories: policy makers, or those who must make decisions about resource allocations, program support, or new legislation and regulations; citizens, or those who are consumers of the services or who may otherwise be affected by government policies; and service providers, or those who are involved in the operation of actual services-e.g., schools, police services, health facilities, and social service programs.(Yin, et al., 1981, p.557)In any strategy it is important to think about approaches tailored to different audiences, and about giving easy access to the main findings or conclusions (Ruzek & Rosen, 2009).Each of these groups may need different kinds of products, events and networks.Simply listing all research reports, or giving a schedule of activities will not suffice because users usually do not have the ability to evaluate the applicability of the information (Davenport & Volpel, 2001).A clear indication of the intended audience is an important measurement to predict the uptake of research.

Accessibility
Accessibility has long been recognized as a major impediment to effective research knowledge mobilization (Hemsley-Brown & Sharp, 2004;Nutley, Walter & Davies, 2007).As noted, the internet provides enormous new possibilities for making research more accessible to users.Features such as links, search engines, and menus bars embedded in websites enable web users to be more active in obtaining information.On-line interaction makes possible much more synergy between information senders and receivers (Tremayne & Dunwoody, 2001).Knowledge mobilization requires that resources be available to users, hence organizations which freely provide products, events and network score points across this indicator, while those requiring pay or registration get fewer points.We also evaluate the readability (language level) of the relevant information on the website, using the Flesch-Kincaid reading level assessment of the mission statement of each organization, on the grounds that esoteric language can also be a barrier to effective communication.The readability test produces a score based on the number of words per sentence and the number of syllables per word.

Ease of Use
Ease of use refers to how much effort is required to find and use research information.Assessing this criterion varies based on the nature of products, events and networks.Feedback mechanisms to allow input from potential users, especially if interactive, are one essential element to improve KM activities.Such mechanisms include the opportunity to respond to materials, to post queries or comments, or to participate in an going discussion forum or listserv.However simply having a "contact us" feature, such seen on almost every institutional website, is not included here because this does not provide real interaction.
Frequency of interaction assesses how frequently those involved in a network interact with each other, which is one indicator of the efforts by an organization to reach out to its audiences rather than passively waiting for them to seek out what they need.Features of this criterion include follow ups after certain KM events, posting a summary or a visual record of the events, setting up various forms of archives for products and information from previous events, searchable directories for networks, events calendars, and so on.All these features help users get the information they need in a relatively short time.

Additional Elements
It is also worth noting that the three strategies of products, events and networks require a balance.Multiple and varied strategies seem to be essential toed reach different audiences and increase the impact of research.For example, organizations can use various types of products (various forms of print, audio, video), or of events (seminars, workshops, longer and shorter) and networks (short-term or task focused vs long term or more general).We therefore award additional points for balance within and across strategies.Finally, we identified another element that we considered important but was not part of the three main strategies; this was a clear and prominent statement on the site of the organization's intentions and focus around research mobilization as a clear statement to visitors of the value of this work.
The system that we developed on this basis and used to rate sites has a total potential score of 72 points (Figure 3).In keeping with the research on active vs passive dissemination measures, more points (20 points each) are allocated for events and network, with fewer points (12 points) for products.

Results
Over 100 institutional websites have been scored using this system.This sample covers different types of organizations (intermediaries, governments, universities, faculties of education, etc.) in different sectors (education, health, etc.).The single largest category is faculties of education in universities, but there are also significant numbers of sites of school districts, ministries of education, universities as a whole, and various other organizations involved in the sharing of research.While most are in Canada, the full set of organizations is international, though almost entirely from English-speaking and richer countries.The full list of the organizations scored is in Appendix A. As noted earlier, the set of organizations was determined by starting with Canadian education organizations involved in the 3 contexts of our conceptual framework and then adding more organizations of similar types from other sectors or countries.However the sample for this study has not been drawn in a way that can be claimed to be representative of a broader universe of institutional websites, so the findings should be regarded as preliminary and indicative.
Scoring consisted of reviewing multiple pages on each website to determine which of these features was present and to what degree, a procedure that typically took 15-20 minutes per site.Each rater had to look for each feature on each site.For example, rating 'different types' of KM vehicles often required checking in several parts of a website since products, events and networks could well be located in different places.
Several procedures were employed to maximize the reliability of the scoring process and results.All sites were scored by project team members who participated in developing the instrument and are familiar with its aims and elements.Team members, who were all doctoral students at the Ontario Institute for Studies in Education, also participated in several sessions to assess the reliability of the instrument and make improvements through comparing scores for the same sites and resolving disagreements.Although eight or so students (and the faculty supervisor) were involved in rating at least some sites, most of the rating was done by 3 students so that they could acquire significant competence.A random selection of sites from the sample were rated by at least two team members and the average scores of these sites are used for the final analysis.A formal interrater reliability analysis showed an IRR of .799.
The histogram in Figure 4 shows the distribution of the total scores of all the websites in our sample.The highest possible score is 72.A more detailed understanding comes from looking at the scores on some of the sub scales.Table 1 shows the average score and % of possible points in each category.Several conclusions arise from the analysis.First, the overall research sharing efforts of these organizations based on this assessment are fairly low, although virtually all organizations are embodying some KM features.The highest scores awarded are from 51 to 60, or from 70-85%; these scores were mostly for websites of faculties of education or universities.The average score is 28, or well under 50%.This reflects our observational understanding from looking at all these sites that on the whole KM practices, at least as revealed on websites, are not very strong in most organizations.This finding is also consistent with a review of KM practices of 18 major research universities based on interviews with university leaders (Sá et al., 2011).Of course we did see some outstanding practices here and there but rarely consistently either in a particular category or for a particular organization.

Distribution of Scores
To be sure, the institutions in our sample vary significantly in nature and purpose.One would not expect to see the same KM activities in a research producing organization such as a university as one would in a user organization such as a school district or in an intermediary body such as a think tank.However each of these organizations could have a knowledge mobilization strategy and even with their different missions these strategies could usefully involve the range of criteria in our evaluative system.The products or events or networks may well differ given different purposes, but any kind of educational organization with a genuine desire to do a better job of sharing and using research evidence should use some combination of products, events and networks.Instead, we find that based on our data 54% of the organizations have no space for feedbacks or comments on the site while only 6% have space for comments related to specific research products; only 38% of sites offer all their research products or services on line free of charge; 29% have no research-related events and of those that do fewer than half show any follow-up for their events; about 65% of sites have some form of network related to research but of most do not indicate who is in the network, nor do they provide ready online access to the network's past activities; about ¼ of the sites have a clear statement of intent or purpose around their research mobilization work.
Out of the three major research sharing strategies (products, events and networks), organizations tend to devote more efforts to products, although even here, the scores are not particularly high on average.Few organizations seem to move beyond disseminating research reports or journal articles to, say, tailoring clear messages for potential knowledge users.Even though many institutions offer a wide variety of research products on their websites with easy access, they quite often distribute them in a simple table format or in a long list, without distinguishing clearly among different content for the sake of users.On most sites it is hard to know what products are available, or what they contain, without investigating them one at a time.Sites with large numbers of products often do not do much to help visitors sort them out by focus or approach.
Events and networks strategies are less used overall and when used are weaker in terms of ease of use, accessibility and multiple types.Events are used more than networks, and average scores for events are higher than for networks, reflecting, perhaps, the greater complexity of organizing networks.Even where these network strategies have clear instructions about purpose and content, they usually have conditions attached for public participation.Follow-ups such as posting material after advertised events are rare.
Overall, the strategies evidenced on the websites tend to be quite passive in terms of research sharing, relying on visitors coming to find material rather than institutions actively working to mobilize knowledge.In making this claim, it should be noted that there may be further substantive events and networks that are not shown on websites.However from the data in this study, institutions generally do not display clear strategies for engaging their target audience.Where websites do contain vehicles such as listservs these are often not geared towards discussion of research.For networks that are research related, there is often little or no introduction as to what the network is, how it operates or who should participate.There are few opportunities offered on sites for users to provide feedback of any kind, and even fewer for feedback on specific research products, events or networks.This reduces the likelihood of visitors engaging actively with the resources on the site.
In summary, our analysis, consistent with other work cited earlier, suggests that most organizations are not very deliberate in their knowledge mobilization strategies.They have activities but not an overall strategy.As in many areas of organizational life, people tend to do what is easy, consistent with current practice, or being done by others rather than thinking through how goals can be achieved most effectively.Our future analysis will include disaggregating scores according to sector or geographically to see how KM strategies and indicators vary according to context.We also plan to score more organizations and to investigate those organizations with the highest scores to understand their practices more fully.

Discussion and Further Research
The importance of institutional websites for knowledge mobilization is increasingly recognized in the research literature.Websites play a vital role in well designed KM strategies (Levin, 2011).We conclude this paper with suggestions for further research and for policy.
From a research perspective, our work points to the need to develop new tools and measures related to KM in order to get better data on effects.Research sharing has many properties which make it difficult to study (Levin, 2011), one of which is the extensive use of self-report measures.The measurement tool we have developed is certainly not perfect, but it offers a starting point for others to examine KM work in a more objective way, and the process we used to build it also provides an example of how other kinds of analytic tools can be created to assess the KM work of various organizations.
This research is clearly only a beginning.Different audiences will use research information in varying ways.Our system includes the focus of audience as an important indicator but does not address the varying interests and approaches of different audiences.Much detailed descriptive work is needed to understand how different kinds of visitors -for example teachers as compared with parents -use the resources provided by institutional websites and their preferences in terms of research format, type of language, search patterns, and so on, as well as how these different strategies impact learning and ultimate knowledge use.Our team is conducting other research in these areas and on the use of web-based research materials using web analytics and longitudinal survey data to assess the actual use of these materials (see www.oise.utoronto.ca/rspefor details).
The measurement system and data presented in this paper contribute to thinking about how KM strategies can be more effective and how organizations can build knowledge sharing infrastructure that is more responsive to knowledge users.From a policy perspective, this work shows the importance of greater attention to research communication.If research is to have its full effect in helping improve education, more active measures will have to be taken to link research knowledge with the mechanisms that lead to changes in practice.While that discussion requires a paper in its own right (see Levin, 2010Levin, , 2011)), the data here show clearly that current efforts are not optimal.
Further investigation is needed as to whether these various strategies increase not only access to research but also its use in various policy and practice settings.This is itself a challenging task given the complexities around the meaning of 'research use' (Nutley et al., 2007).That being said, we already know enough to make some recommendations as to how organizations could change their KM work to promote knowledge use.Our major recommendations in this regard would include organizations should be more strategic about KM work.The different components of KM should be integrated so that they reflect an overall strategy rather than a series of discrete actions, and so that the various elements are more consistent with the growing knowledge base on what is effective.Interpersonal and organizational links are more important than the passive provision of research reports or other products, since take-up is more strongly affected by social factors.This means more attention to building connections rather than just supplying information.Websites need to be linked to more active or 'push' strategies that make use of existing organizational mechanisms and processes.Organizations should track the impact of their efforts through such vehicles as web analytics and surveys of visitors.
Fortunately, there is growing interest internationally in empirical research into research utilization, suggesting that our understanding of useful practices around knowledge mobilization is likely to grow, leading to more benefit from research in education and other fields.