This article has been retrieved  times since August 19, 1998.

Education Policy Analysis Archives
Volume 6 Number 15	August 19, 1998	ISSN 1068-2341

---

A peer-reviewed scholarly electronic journal.  Editor:  Gene V Glass   Glass@ASU.EDU.  College of Education  Arizona State University,Tempe AZ 85287-2411  Copyright 1998, the EDUCATION POLICY ANALYSIS ARCHIVES.Permission is hereby granted to copy any article provided that EDUCATION POLICY ANALYSIS ARCHIVES is credited and copies are not sold.

---

A Note on the Empirical Futility of Labor-Intensive Scoring Permutations for Assessing Scholarly Productivity: Implications for Research, Promotion/Tenure, and Mentoring Christine Hanish John J. Horan Bethanne Keen Ginger Clark Arizona State University Abstract The measurement of scholarly productivity is embroiled in a controversy concerning the differential crediting of coauthors. Some researchers assign equivalent shares to each coauthor; others employ weighting systems based on authorship order. Horan and his colleagues use simple publication totals, arguing that the psychometric properties of labor-intensive alternatives are unknown, and relevant ethical guidelines for including coauthors are neither widely understood nor consistently followed. The PsycLIT and SSCI data bases provided exhaustive publication and citation frequencies for 323 counseling psychology faculty. All PsycLIT scoring permutations yielded essentially identical information; inter-correlations ranged from .96 to unity. Moreover, all PsycLIT methods correlated highly with SSCI within a very narrow band. Since attention to the number and/or ordinal position of coauthors yields no useful information, productivity should be defined parsimoniously in terms of simple publication counts. Implications for research, promotion/tenure, and the mentoring of graduate students are discussed.             Publishing behavior is perhaps the most revered and reviled variable in education and psychology. The bipolar affect it generates undoubtedly derives from the fact that although the act of publishing is inextricably entwined with status and the reward system of a scientific discipline (e.g., promotion, tenure, merit pay, and the like), the criteria for evaluating what an individual publishes are much less clear (Merton, 1973). The concepts of productivity, impact, and quality are often used interchangeably as descriptors, yet there are important methodological and psychometric differences.             Productivity refers to the quantity of publications attributable to a given scholar, expressed as a lifetime total or a yearly rate when divided by the scholar's professional age. Impact generally means how frequently that individual's work is cited by other authors, which likewise can be expressed as a lifetime total or a yearly rate. Quality is almost never assessed directly; productivity and impact, though, frequently pose in its place (see Keen, Horan, Hanish, Copperstone, & Tribbensee, 1998).             Since vita entries provide no assurance that a document really exists, the assessment of productivity is usually confined to the number of publications by an individual that appear in large data-bases such as ERIC or PsycLIT (Horan & Erickson, 1991). The gate-keeper functions in these data-bases, however, infuse raw counts of productivity with elements of quality. For example, PsycLIT only lists articles that appear in refereed journals recognized by APA as relevant to the discipline of psychology.             The assessment of impact is likewise usually restricted to full citation histories contained in large holdings such as SSCI, though smaller segments of that data base and/or fewer numbers of outlets have been used (albeit, unreliably, see Horan, Hanish, & Beasley, 1995). SSCI is more often associated with quality than is PsycLIT, but that kudo may not be warranted. Hanish, Horan, Keen, St. Peter, Ceperich, and Beasley (1995) reported high relationships between PsycLIT and SSCI; moreover, other limitations of SSCI are less well known and understood. For example, SSCI scores may be inflated by hidden self-citations, citations by prolific colleagues, advisees, or significant others, the notoriety of a study rather than its importance, and so forth (see Horan, Hanish, Keen, Saberi, & Hird, 1993).             The measurement of productivity has become embroiled in a controversy concerning the differential crediting of coauthors. Some researchers (such as Bohn, 1966; Goodstein, 1963; Goodyear, Abadie, & Walsh, 1983; Katz & Brophy, 1975; Tinsley & Tinsley, 1979; Walsh, Feeney, & Resnick, 1969) give each coauthor equal partial credit (e.g., a third of a point to three coauthors of a given article); others (such as Delgado & Howard, 1994; Ellis, Haase, Skowron, & Kaminsky, 1993; Howard, 1983; Howard, Cole, & Maxwell, 1987; Osipow, 1985; Skovholt, Stone, & Hill, 1984) apply various weighting formulas based on the ordinal positions of coauthors (e.g., first author receives half of the credit, the second author 30% of the credit, and the last author the final 20%).             In contrast, Horan and his colleagues (e.g., Hanish, et al.,1995; Horan & Erickson, 1991; Horan, Weber, Fitzsimmons, Maglio, & Hanish, 1993b) have always used simple raw PsycLIT totals for each author, arguing that the psychometric properties of the foregoing schema are unknown, and APA's ethical guidelines for assigning authorship are neither widely understood nor consistently followed (e.g., see Fine & Kurdek, 1993; Goodyear, Crego, & Johnston, 1992).             The present study, therefore, attempted to clarify the relationships between the various scoring permutations of PsycLIT with each other and with SSCI. Although the same scoring controversy could apply to coauthorships listed in ERIC or in other data bases, we chose PsycLIT because its refereed holdings are obtained independent of author consent, and thus provide a more meaningful basis for comparison with other indices of scholarly merit. Method Subjects             Hanish et al. (1995), identified the entire population of academic counseling psychology faculty (n = 323) who were members of Division 17 and who had governance responsibilities in any active doctoral training program; for each individual, they secured complete PsycLIT data from 1974 to 1991 and SSCI data from 1971 to 1991. In the present study we updated all PsycLIT and SSCI data on these individuals to be current to 1996. Measures             The PsycLIT data base includes all Psychological Abstracts references attributable to individual authors published from 1974 to present. A search by author name yielded a full bibliographical citation list for that author including coauthors and abstracts. These data were scored according to six different methods described as follows: Method 1, used by Horan and his associates (e.g., Horan & Erickson, 1991; Hanish et al., 1995), awards a single point to each author for each publication regardless of the number of coauthors or their ordinal position. If an individual has 13 sole or coauthored publications in the PsycLIT data base his or her score will be 13. Method 2 is relatively popular (e.g., Bohn, 1966; Goodstein, 1963; Goodyear, Abadie, & Walsh, 1983; Katz & Brophy, 1975; Tinsley & Tinsley, 1979; Walsh, Feeney, & Resnick, 1969); coauthors receive equal partial credit (e.g., a third of a point to three coauthors of a given article). First and last authors are treated alike. Method 2 and all methods that follow are increasingly labor intensive in that they require the computation and summing of various amounts of credit for each bibliographic entry on a given author's publication record. Method 3 (Delgado & Howard, 1994; Howard, 1983) awards one point to sole authors. The first and second authors of a coauthored publication receive .67 and .33 points, respectively. If three coauthors are involved, the differential credit allocations are .50, .30, and .20. Additional coauthors result in decreasing credit for all. Method 4 (Howard, Cole, & Maxwell, 1987) uses a very complex formula to compute the differential allocation of credit. As with Method 3, authors and coauthors receive declining amounts of credit as their numbers increase and their ordinal positions descend. Method 5 (Osipow, 1985; Skovolt, Stone, & Hill, 1984) awards sole authors and first authors 5 points, second authors 4, third authors 3, and fourth authors 2; all subsequent coauthors receive a score of 1. Points are thus constant across ordinal position. Method 6 was devised by Ellis, Haase, Skowron, and Kaminsky (1993). Weights depend on the number of authors, the order of authorship, and the value of the article using the method of Skovolt, Stone, and Hill (1984). For example, an article with three coauthors has a value of 12 which is derived by adding five points for the first author, four points for the second author, and three points for the third author. The first author's credit then is 5/12 or .417; the second author's credit is 4/12 or .333 and so on. For articles with more than four coauthors, the fifth and subsequent authors receive equal shares of .067 such that, for example, the fifth and sixth authors would each receive .034.             The credit consequences of the six different productivity scoring methods on the coauthors of a given article can be seen in Table 1. Table 1 Template for Productivity Scoring Methods Indicating Comparative Credit by Number and Ordinal Position of Coauthors. Author/ Coauthors Method 1 Horan Method 2 Walsh Method 3 Howard 1 Method 4 Howard 2 Method 5 Skovholt Method 6 Ellis 1/1 1.000 1.000 1.000 1.00 5.000 1.000 1/2 1.000 .500 .670 .600 5.000 .556 2/2 1.000 .500 .330 .400 4.000 .444 1/3 1.000 .333 .500 .474 5.000 .417 2/3 1.000 .333 .300 .316 4.000 .333 3/3 1.000 .333 .200 .210 3.000 .250 1/4 1.000 .250 .400 .415 5.000 .357 2/4 1.000 .250 .300 .277 4.000 .286 3/4 1.000 .250 .200 .185 3.000 .214 4/4 1.000 .250 .100 .123 2.000 .143 1/5 1.000 .200 .330 .384 5.000 .333 2/5 1.000 .200 .270 .256 4.000 .267 3/5 1.000 .200 .200 .171 3.000 .200 4/5 1.000 .200 .130 .114 2.000 .133 5/5 1.000 .200 .070 .076 1.000 .067 1/6 1.000 .167 .286 .365 5.000 .333 2/6 1.000 .167 .238 .244 4.000 .267 3/6 1.000 .167 .190 .162 3.000 .200 4/6 1.000 .167 .143 .108 2.000 .133 5/6 1.000 .167 .095 .072 1.000 .035 6/6 1.000 .167 .048 .048 1.000 .035 1/7 1.000 .143 .250 .354 5.000 .333 2/7 1.000 .143 .214 .236 4.000 .267 3/7 1.000 .143 .179 .157 3.000 .200 4/7 1.000 .143 .143 .105 2.000 .133 5/7 1.000 .143 .107 .070 1.000 .023 6/7 1.000 .143 .071 .047 1.000 .023 7/7 1.000 .143 .036 .031 1.000 .023 1/8 1.000 .125 .222 .347 5.000 .333 2/8 1.000 .125 .194 .231 4.000 .267 3/8 1.000 .125 .167 .154 3.000 .200 4/8 1.000 .125 .139 .103 2.000 .133 5/8 1.000 .125 .111 .069 1.000 .017 6/8 1.000 .125 .083 .046 1.000 .017 7/8 1.000 .125 .056 .030 1.000 .017 8/8 1.000 .125 .028 .020 1.000 .017 Note: The names are those of researchers most closely associated with the various scoring methods. Under Author/Coauthors, 1/1 = sole author, 1/2 = first author of an article by two authors, 2/3 = second author of an article by three authors, etc.             SSCI is a compilation of citations to a given sole or first author by that same author and other scholars from 26 disciplines in the social and behavioral sciences. Cited authors are arranged alphabetically in bound volumes covering the years 1966 to present. Our search was confined to the SSCI volumes paralleling our PsycLIT database. Below each cited author's work in SSCI is a list of individuals who referenced that work along with abbreviated outlet information. We used two SSCI scoring methods, namely, the grand total and the grand total minus obvious self-citations. An obvious self-citation occurred when a first author cited himself or herself in a first-authored reference. SSCI makes no provision for detecting "hidden" self-citations, for example, second authors citing their first-authored works. Procedures             Procedures for faculty identification, biographical information, reliability analyses, and so forth are described in Hanish et al. (1995). The new PsycLIT and SSCI raw data obtained for the present study were secured in the same fashion. Each of the 323 faculty publication histories was then coded according to the methods described above by doctoral students working independently. This, of course, was an extremely time-consuming process. A random sample of 1752 publications was rechecked by additional students; disagreements between coders were trivial (1.9%). To facilitate further work in this area, a priori scoring templates are presented in Table 1. For example, if an individual is listed as third of four authors on a particular publication, the columns contain the precalculated author-position scores for each of the six methods. Results             The actual raw data on which all analyses are based are being made available to the reader. From this point, the data files can be accessed in EXCEL, SPSS or ACII format. Of 323 individual faculty, only 10 had no evidence of publishing history in the PsycLIT and SSCI data bases. A similar number exceeded 65 publications and 650 citations. The median faculty member in our study had 13 publications in PsycLIT and was cited in SSCI 50 times including an average of 3 obvious self-citations. Table 2 depicts the correlations involving PsycLIT scoring permutations with each other and with SSCI. Table 2 Correlations between PsycLIT and SSCI scoring permutations Variable PsycLIT Method 2 Walsh PsycLIT Method 3 Howard1 PsycLIT Method 4 Howard2 PsycLIT Method 5 Skovholt PsycLIT Method 6 Ellis SSCI Total SSCI Minus SelfCites PsycLIT Method 1 Horan .961 .963 .965 .998 .966 .711 .669 PsycLIT Method 2 Walsh .997 .998 .971 .999 .703 .659 PsycLIT Method 3 Howard1 1.00 .975 .999 .701 .654 PsycLIT Method 4 Howard2 .976 1.00 .703 .657 PsycLIT Method 5 Skovholt .976 .712 .669 PsycLIT Method 6 Ellis .704 .659 SSCI Total .995 Note: The names are those of researchers most closely associated with the various scoring methods.             The relationships among the six scoring methods for assessing productivity are remarkably high. No individual pairwise correlation was lower than .96; several r's reached unity. Similarly, the Pearson r between SSCI total and SSCI minus obvious self-citations also approached unity (.995).             More importantly, however, despite the fact that productivity and impact reflect different concepts and derive from disparate assessment methodologies, the relationships between these variables, regardless of scoring method, were strong and consistent. All six PsycLIT scoring permutations correlated with SSCI total inside a very narrow band of .701 to .712; and the band remained high and narrow (.654 to .669) when obvious self-citations were deleted. Discussion             Our data reflect the lifetime publishing behavior of an entire population of academic faculty affiliated with doctoral training programs in counseling psychology. Although we have not established that the foregoing relationships hold true in other sectors of science, there are no a priori reasons to think otherwise. Essentially, the controversy involving the comparative merits of various methods for assessing scholarly productivity has been settled. All PsycLIT scoring permutations yield essentially identical information; inter-correlations range from .96 to unity. Moreover, all of these PsycLIT methods also correlate with SSCI data at a fairly high level and within a very narrow band.             Several implications are apparent. For example, future researchers are now informed that labor-intensive scoring permutations are not cost beneficial in comparison to the use of simple raw scores to assess an individual's scholarly productivity. The law of parsimony demands that a scholar's productivity be defined in terms of the number of articles carrying his or her name; attention to the number and/or ordinal position of coauthors yields no useful information.             It would be interesting to observe if the behavior of promotion and tenure committees will change as a result of increased awareness of the relationships reported in this study. Such committees can exhibit highly variable judgment even within the same institution. Collaborative research, for example, is sometimes valued ("has good collegial relationships"), sometimes denigrated ("needs to demonstrate more independent scholarship"); our findings suggest that the phenomenon of coauthoring is simply a facet of academic life, not a basis for evaluation.             Finally, we hope that our data eliminate a thorny disincentive to the formation of good mentoring relationships. Scoring methods 2 through 6 clearly advantage those in differential power relationships who chose self-interest over propriety while still staying within the letter of relevant ethical codes. Reptilian supervision modes are predictable, though no less abhorrent in the context of promotion, tenure, and merit pay systems that, for example, heavily weight sole authorships. Half of the publications by our institution's counseling psychology faculty in the PsycLIT data base involve students as coauthors, a percentage possibly comparable to that displayed in many other graduate programs. In contrast to labor-intensive, and empirically unwarranted alternatives, the use of simple raw scores to assess productivity contributes to the class-action benefit of everyone at no cost to anyone. References Bohn, M. J. (1966). Institutional sources of articles in this journal of counseling psychology--Four years later. Journal of Counseling Psychology, 13, 489-490. Delgado, E. A., & Howard, G. S. (1994). Changes in research productivity in counseling psychology: Revisiting Howard (1983) a decade later. Journal of Counseling Psychology, 41, 69-73. Ellis, M. V., Haase, R.F., Skowron, E. A., & Kaminsky, L. (1993, August). Institutional affiliations of contributors to scholarly and professional activities in counseling psychology: 1987-1990. Paper presented at the Annual Meeting of the American Psychological Association, Toronto, Canada. Fine, M. A., & Kurdek, L. A. (1993). Reflections on determining authorship credit and authorship order on faculty-student collaborations. American Psychologist, 48, 1141-1147. Goodstein, L. D. (1963). The institutional sources of articles in the Journal of Counseling Psychology. Journal of Counseling Psychology, 10, 94-95. Goodyear, R. K., Abadie, P. D., & Walsh, W. B. (1983). Graduate school origins of Journal of Counseling Psychology authors: Volumes 15-28. Journal of Counseling Psychology, 30, 283-286. Goodyear, R. K., Crego, C. A., Johnston, M. W. (1992). Ethical issues in the supervision of student research: A study of critical incidents. Professional Psychology: Research and Practice, 23, 203-210. Hanish, C., Horan, J. J., Keen, B., St. Peter, C. C., Ceperich, S. D., & Beasley, J. F. (1995). The scientific stature of counseling psychology training programs: A still picture of a shifting scene. The Counseling Psychologist, 23, 82-101. Horan, J. J., & Erickson, C. D. (1991). Fellowship behavior in Division 17 and the MOMM cartel. The Counseling Psychologist, 19, 253-259. Horan, J. J., Hanish, C., & Beasley, J. F. (1995). A methodological reply to a motivational charge. The Counseling Psychologist, 23, 125-128. Horan, J. J., Hanish, C., Keen, B., Saberi, D., & Hird, J. S. (1993a). When examining the cerebral functioning of Division 17, which organ should we dissect? The Counseling Psychologist, 21, 307-315. Horan, J. J., Weber, W. L., Fitzsimmons, P., Maglio, C. J., & Hanish, C. (1993b). Further manifestations of the MOMM phenomenon: Relevant data on editorial board appointments and membership composition. The Counseling Psychologist, 21, 278-287. Howard, G. S. (1983). Research productivity in counseling psychology: An update and generalization study. Journal of Counseling Psychology, 30, 600-602. Howard, G. S., Cole, D.A., & Maxwell, S. E. (1987). Research productivity in psychology based on publication in the journals of the American Psychological Association. American Psychologist, 42, 975-986. Katz, G. M. & Brophy, A. L. (1975). Institutional sources of articles in the Journal of Counseling Psychology, 1962-1973. Journal of Counseling Psychology, 22, 160-163. Keen, B., Horan, J. J., Hanish, C., Copperstone, J., Tribbensee, N. (August, 1998). Publication frequency, citation frequency, and quality of counseling psychology research. Paper presented at the annual meeting of the American Psychological Association, San Francisco. Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. (N. W. Storer, Ed.). Chicago: The University of Chicago Press. Osipow, S. H. (1985). Skovholt, Stone, and Hill's (1984) "Institutional affiliations of contributors to scholarly and professional activities in counseling psychology: 1980-1983" -- A critique. Journal of Counseling Psychology, 32, 466-468. Skovholt, T.M., Stone, G. L., & Hill, C.E. (1984). Institutional affiliations of contributors to scholarly and professional activities in counseling psychology: 1980-1983. Journal of Counseling Psychology, 31, 394-397. Tinsley, D. J. & Tinsley, H. E. A. (1979). Trends in institutional contributions to the Journal of Counseling Psychology. Journal of Counseling Psychology, 26, 152-158. Walsh, W.B., Feeney, D., & Resnick, H. (1969). Graduate school origins of Journal of Counseling Psychology authors. Journal of Counseling Psychology, 16, 375-376. About the Authors Christine Hanish Homepage: http://psy.ed.asu.edu/~hanish/             Christine Hanish is a doctoral student in counseling psychology at Arizona State University. She works for ASU's Preventive Intervention Research Center which specializes in the development and validation of programs for children, adolescents, and family. She is currently immersed in a research project attempting to establish the norms of scholarly behavior for academic counseling psychologists. John J. Horan Email: horan@asu.edu Homepage: http://horan.asu.edu             I am a professor of counseling psychology at Arizona State University. I graduated from Michigan State University and taught at Penn State before moving to ASU in 1985. Most of my writing has focused on the evaluation of cognitive-behavioral intervention strategies.             For more than a decade I have been examining the experimental construct validity of these interventions. For example, do they produce changes on measures of high theoretical relevance while simultaneously failing to effect changes on measures of low theoretical relevance? Lately, I have concentrated on adapting and evaluating computer and Internet interventions for a variety of counseling problems.             For a quick look at how I squandered my youth, click on my web-based vita. My most important accomplishments, however, are not listed there. I have had many extraordinary students in my career, including those who share this masthead. I feel privileged to have contributed to their professional development; they surely have enhanced my own. Bethanne Keen, Ph.D. Email: BethKeen@aol.com Homepage: http://psy.ed.asu.edu/~keen/             Bethanne Keen received a Ph.D. in counseling psychology from Arizona State University in December 1997. She is currently completing a postdoctoral residency in psychology with a large group practice in Phoenix, Arizona. She also serves as chair of the Legislative Affairs Committee for the Arizona Psychological Association. Her dissertation, currently being prepared for publication, explores the relationships between publication frequency, citation frequency and quality of research conducted by counseling psychologists in academe. She is currently involved in a research project designed to illuminate the challenges faced by new Ph.D.s in psychology to achieve employment and licensure in Arizona. Her other research interests include collection and analysis of clinical outcomes data. Ginger Clark ginger.clark@asu.edu Homepage: http://psy.ed.asu.edu/~clark/             Ginger Clark is a doctoral student in counseling psychology at Arizona State University. She has conducted or contributed to studies in human sexual styles, parent education, parent education in career development, health habits, and quality of life for mid-life women. She has also written book reviews in the area of family therapy. Clark received her Bachelor's and Master's degrees in psychology at California State University Long Beach. She is currently in her fourth year of doctoral study, and is working toward an academic position in counseling psychology. Correspondence concerning this article should be addressed to John J. Horan, Division of Psychology in Education, Arizona State University, BOX 870611, Tempe, AZ 85287-0611. Electronic mail may be sent to: horan@asu.edu

Copyright 1998 by the Education Policy Analysis Archives The World Wide Web address for the Education Policy Analysis Archives is http://olam.ed.asu.edu/epaa General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. (602-965-2692). The Book Review Editor is Walter E. Shepherd: shepherd@asu.edu . The Commentary Editor is Casey D. Cobb: casey@olam.ed.asu.edu . EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Andrew Coulson a_coulson@msn.com Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov hmwkhelp@scott.net Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Marshall University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Richard M. Jaeger University of North Carolina--Greensboro Daniel Kallós Umeå University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Rocky Mountain College Dewayne Matthews Western Interstate Commission for Higher Education William McInerney Purdue University Mary P. McKeown Arizona Board of Regents Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton apembert@pen.k12.va.us Hugh G. Petrie SUNY Buffalo Richard C. Richardson Arizona State University Anthony G. Rud Jr. Purdue University Dennis Sayers Ann Leavenworth Center for Accelerated Learning Jay D. Scribner University of Texas at Austin Michael Scriven scriven@aol.com Robert E. Stake University of Illinois--UC Robert Stonehill U.S. Department of Education Robert T. Stout Arizona State University    archives   |   abstracts   |   editors   |   board   |   submit   |   comment   |   subscribe   |   search