This article has been retrieved times since December 22, 1995.

Education Policy Analysis Archives
Volume 3 Number 20	December 22, 1995	ISSN 1068-2341

---

A peer-reviewed scholarly electronic journal. Editor: Gene V Glass   Glass@ASU.EDU College of Education Arizona State University,Tempe AZ 85287-2411 Copyright 1995, the EDUCATION POLICY ANALYSIS ARCHIVES.Permission is hereby granted to copy any article provided that EDUCATION POLICY ANALYSIS ARCHIVES is credited and copies are not sold.

---

Possible Indicators of Research Quality
for Colleges and Universities

Ronald H. Nowaczyk
Clemson University

David G. Underwood
Clemson University

udavid@CLEMSON.EDU

Abstract:

The move toward more public accountability of institutions of higher education has focused primarily on undergraduate education. Yet, many institutions view research as an important component of their mission. Much of the literature on assessing research quality has relied on quantitative measures such as level of outside funding and number of publications generated. Focus groups consisting of research faculty were conducted at a landgrant university. Faculty were asked to evaluate current indicators of research quality as well as to suggest additional measures. While faculty recognized the need for the traditional measures, they cautioned against over-reliance on these indicators. Additional indicators focusing on graduate education as well as external peer reviews were recommended. Developing indicators that provide evidence of long-term impact on social and scientific advancement was suggested.

The public desire for accountability of programs at institutions of higher education has contributed greatly to the assessment movement in higher education. Regional accrediting associations (e.g., Middle States, New England, North Central, Northwest, Southern and Western) expect colleges and universities to demonstrate their institutional effectiveness through an ongoing program of self-examination.

Considerable work in the area of assessing the undergraduate teaching mission has led to a growing body of literature (e.g., Astin, 1985. 1991; Bogue & Saunders, 1992; Erwin, 1991; Ewell, 1983). Less attention has been directed toward other aspects of an institution's mission including research. Adding to the body of knowledge by engaging in research and related scholarly activities is an important component of the mission for many institutions especially those with graduate programs or a substantial level of external research funding. The development of effective assessment programs for the research mission is in its early stages. A good assessment program should include quality indicators that are acceptable to the academic community.

Review of the Literature on Research Productivity

The role of research in a university environment must not be underestimated. As Dill (1986) points out, the vast majority of discoveries are made in a higher education environment. In recent years the emphasis for research universities seems to be focused on productivity. The system used by the Carnegie Foundation (1987) to classify colleges and universities, uses a measure of dollars generated by research and number of Ph.D. degrees granted as the major method of classifying institutions into categories. Additionally, the National Science Foundation annually ranks institutions based on Total and Federally Financed Research and Development Expenditures. In each of these cases, "more" is interpreted to mean "better." The research produced by those ranked higher is considered to be of better quality than that produced by those who are ranked lower.

The issue of "quality" in research has been a topic of great debate on university campuses, but it has produced very little in the way of literature which identifies the meaning of the term as it applies specifically to research. Several indicators have been identified as being useful in determining the quality of research programs. Among those are the number of dollars generated, number of publications, number of citations, and peer review (Kogan, 1989).

Productivity Dollars. Snyder et al. (1991) focused on aspects of strategic management to determine factors which were equated with research excellence. They selected a sample of institutions from the top 100 in the previously mentioned National Science Foundation rankings. The implication is that if an institution is ranked highly then excellence in research is the reason. Their finding that the number of dollars generated by research was the most often cited measurement of success should come as no surprise. They also found that while most of those surveyed could identify factors, such as dollars, which were used to measure the success of the research program, "it was not clear whether or not these factors were selected consciously as factors necessary for the attainment of objectives, or because they were the easiest factors to measure" (Snyder, et al., 1991, p. 52). Directly related is their finding that "Those universities that are ranked higher, have faculty that are adept at obtaining research grants" (p. 55). The emphasis on dollars is further illustrated by Archambault (1989) who identifies the need for quality in research but points out that, in addition, it must be "profitable."

Productivity Publications. With the emphasis on productivity, the number of publications is frequently used as an indicator of quality in research. The fact that the research is published is taken as an indication of its quality. This indicator is often further categorized and weighted by identifying the type of publication (book or research article) and if it is an article, the type of journal (refereed or non-refereed). Although it is a good indicator of how prolific the researcher or the department is in producing acceptable articles it does not address the impact of those articles. A study conducted by Moed, et al. (1989) attempted to make a distinction between what they saw as "output" (the number of publications) and the "impact" of those publications. The impact was determined by checking citations of the articles over a period of years. Their determination was that one should use caution in adopting such indicators because "Citation practices appear to differ significantly from field to field" and "Citation practices within fields can also change during the decade" (p. 190).

Peer Evaluation. Peer evaluations of research and research programs are often cited as a method to insure the quality of the research efforts. The objective of such a review is to assemble a group of peers to review the research efforts and make a determination of the quality of those efforts. Studies conducted by Henkel (1989) list several of the concerns related to the peer review process which are often heard on a university campus. One of the concerns focuses on the idea that "group judgements of scientific quality were thought to be insufficiently acknowledged" (p. 179). The implication here is that the visiting group does not fully understand the work of the unit or individual being reviewed. These concerns are even greater when the work is multidisciplinary or covers a wide span of interest.

An additional concern also noted by Henkel (1989) is that "Scientists do not feel they belong to a republic of equal citizens" (p. 179). This perception does not readily lend itself to being reviewed. The more diverse the span of focus, the greater this concern becomes.

These concerns associated with the normal methods of determining quality in research, coupled with the pressures from our regional accrediting body, the Southern Association of Colleges and Schools (SACS), and our state Commission on Higher Education (CHE) require a closer look at how quality of research is determined. Since most of the concerns associated with how research quality is determined originate with faculty, our intent is to use those faculty, through focus groups, to identify their concerns and develop other indicators which might be more appropriate.

Assessment of Research Programs at Clemson University

Clemson University is a state-assisted, coeducational land grant university located in the northwest corner of South Carolina. The Fall 1992 enrollment included 13,197 undergraduates and 4,263 graduates. The University has nine colleges and awards bachelor's degrees in 72 programs, master's degrees in 68 programs and doctor's degrees in 34 programs. Historically, its mission has emphasized the agricultural and engineering programs.

In response to SACS requirements and state requirements on assessment, Clemson requires each of its academic departments with a research mission to have an action plan in place to evaluate its progress in the area of research and related scholarly activities. The action plan is a two- to three-page document outlining the departmental mission and expected results in the area of research. Each department is expected to include outcome measures that would demonstrate the success of the department in meeting its research goals. Narrative reports based on the plan are expected every three years.

These action plans were submitted during the 1990-91 academic year. It was clear at that time that faculty were struggling with the process of developing indicators to assess the progress of the department regarding its research mission. A variety of indicators were proposed to assess research. These indicators were grouped into one of four categories, productivity, departmental commitment to research, faculty participation, and awards and expenditures. Table 1 lists the percentage of departments listing indicators from these categories.

Table 1
Percent of Clemson University Departments Listing Indicators from Four Categories of Research Activities

Category or Indicator	Percent of Departments
Productivity Measures	100
(Journal publications	100)
(Conference presentations	78)
Authored or edited books	27
Faculty Participation in Research	41
Departmental Commitment to Research	30
Awards & Expenditures	35

Productivity indicators include journal publications, conference presentations, artistic exhibits and performances, and authored or edited books. All departments included one or more indicators from this category. Fewer departments listed indicators in the other three categories. Departmental Commitment focused on the level of departmental resources devoted to research. These indicators included the level of faculty release time for research, the level of financial resources devoted to research (e.g., equipment expenditures, space allocation, computer time), number of graduate students in the program, and level of faculty involvement in supervising graduate research. These indicators are clearly more process than outcome oriented. Faculty Participation which was listed by more than a third of the departments included the number of proposals submitted by faculty, level of faculty involvement in professional organizations, and number of faculty on sabbatical or involved in an exchange program. The category of Awards and Expenditures included level of sponsored research from grants and contracts, level of funding for graduate students, and the number of faculty receiving professional awards.

While a number of these indicators are consistent with those cited in the literature on academic research, we sensed a frustration on the part of our faculty with the adequacy of these indicators to represent completely the research activities at Clemson. At the same time, there was a concern voiced by the faculty that the University might opt for the easiest method of reporting research activity by solely providing quantitative measures that do not necessarily reflect research quality. The listings of the number of research publications and dollar level of external funding were most often mentioned. In fact, the first report to the state focused on quantitative measures that were readily available. In the most recent report the University included a measure of student participation in research, both sponsored and unsponsored.

Given the uneasiness faculty were expressing with these action plans in research, the Office of Assessment and the University Assessment Committee endorsed this project which was designed to elicit faculty input at a broader level regarding assessment of research. Rather than developing and distributing a campus-wide questionnaire, we decided to bring groups of faculty together for discussion on assessment of research. Focus groups were conducted to answer questions faculty may have about the assessment program itself as well as to have the faculty brainstorm about additional indicators that might be developed and used to assess research at Clemson.

Method

Focus Groups

Four focus groups were conducted during the Spring and Fall of 1992. Groups consisted of 6 to 10 faculty members along with the two authors who served as facilitators. Faculty elected to the University Faculty Senate who served on the Senate's Research Committee participated in one focus group. Collegiate Deans nominated two individuals from their colleges, who were respected by their peers as researchers, to serve in the other focus groups. Eighteen of the 20 faculty nominated attended one of the groups. An additional faculty member who serves on the University Assessment Committee also participated.

Procedure

Each focus group lasted approximately one hour. Following an introduction of all participants, the authors described the purpose of the focus groups which was to solicit their opinions regarding evaluation of research at a departmental level. The focus on departmental productivity rather than individual productivity was stressed. Then a series of four questions were presented, one at a time. The questions were:

1. Currently, how would your department head report the overall quality of research in your department?

2. Where do you feel faculty within your department would feel dissatisfied with that type of report?

3. What could be offered as indicators to improve that type of report?

4. What do you see as the distinguishing characteristics of quality research within your discipline?

Each question was shown on an overhead projector and comments were recorded on the overhead as the discussion proceeded. One of the authors also took notes during the sessions. The results are based on a compilation of both the comments on the overheads and the notes that were recorded.

Results

The first two questions asked the faculty to describe how quality of research would be reported in their department currently, and with what aspects of that report they feel faculty would express dissatisfaction. There was considerable consistency among the groups in terms of their responses. Table 2 lists the primary indicators of quality research that faculty feel their departments are currently using. As one would expect at a research university, publications and grantsmanship were the two main areas of focus. Within each area, however, the faculty made a number of distinctions and expressed reservations they held concerning the misinterpretation or over-reliance on various indicators.

Table 2
Frequently Mentioned Quality Indicators Currently Used by Departments for Reporting Research Activities

---

Research Publications:

Number of publications (in some cases, ratio per faculty member)
Types of publications (journal articles, monographs, chapters, books)
Quality of publication

- Reputation of publication in discipline
- Distribution of publication (e.g., regional, national,international)
- Refereed vs. non-refereed journals
- Invited chapters or papers

Citation statistics of research publications (number of citations as well as who is citing the work and the frequency of citation)

Research Grants & External Funding:

Number of grants submitted and funded
Competitiveness of grant process
Reputation of granting agency (e.g, NSF)
Total dollars generated
Granting agency response to grant reports (i.e., satisfaction with work)
Success rate of grant renewals

Other Indicators:

Papers presented at conferences and professional meetings
- Number of papers presented
- Quality of conference

External research awards, fellowships, and recognitions of faculty
External and peer reviews of research programs
Creative and scientific research "products" (e.g., art exhibitions, patents, new research applications and methods)
Amount and quality of interdisciplinary research including collaboration as well as consultative support
Proportion of graduate students who complete terminal degree as well as level of graduate funding for research
Number of completed research projects
Customer satisfaction with research product