This article has been retrieved  times since September 6, 1998.

Education Policy Analysis Archives
Volume 6 Number 17	September 6, 1998	ISSN 1068-2341

---

A peer-reviewed scholarly electronic journal.  Editor:  Gene V Glass   Glass@ASU.EDU.  College of Education  Arizona State University,Tempe AZ 85287-2411  Copyright 1998, the EDUCATION POLICY ANALYSIS ARCHIVES.Permission is hereby granted to copy any article provided that EDUCATION POLICY ANALYSIS ARCHIVES is credited and copies are not sold.

---

Performance Indicators: Information in Search of a Valid and Reliable Use E. Raymond Hackett Auburn University Sarah D. Carrigan University of North Carolina at Greensboro Abstract Measures of overall institutional performance were explored from a decision support perspective with twenty similar Carnegie Classification Baccalaureate II institutions. The study examined the usefulness of performance indicators in campus decision making following both a hypothesis testing and case study approach. Two conclusions were reached: first, that the performance measures most commonly cited in the literature as measures of institutional financial viability are of limited use for institution specific policy development; and second, that performance indicators are most effectively used within an institution specific, whole system framework.

Introduction           State-defined performance indicators for institutions of postsecondary education are rapidly becoming the hallmark of the 1990s. By 1993 over one-third of the states had some form of performance indicator legislation enacted (Bogue, Creech & Folger, 1993) and with each legislative session since the number has increased. Discussion at the state level has begun to shift toward funding the enterprise based on outcomes, effectiveness, and efficiency (Gather, Nedwek, and Neal, 1994). Significant attempts at operationalzing these concepts and weaving them into the fabric of planning, policy and budget development were given license in several states during the 1997 and 1998 legislative sessions.           In the 1994 Education Commission of the States publication Charting Higher Education Accountability (Ruppert, 1994) a case study of ten states indicated that the adoption of state-level performance indicators most often was done rapidly, relied on existing data and usually was driven by legislative initiative. This report implied that few states have accomplished the analysis necessary to define measures appropriate for systemic decision making and public reporting.           With the advent of student right-to-know legislation, federally defined performance indicators for institutions of postsecondary education became a larger part of the institutional reporting cycle. In 1996, the Department of Education proposed a far more explicit use of performance indicators, and this proposal led to a national debate. The belief that a unique equation could provide an indication of institutional financial and programmatic health, and that institutional scores on a specific set of indicators should impact the disbursement of federal funds, was outlined in the Federal Register Volume 61, Number 184 on September 20, 1996. In this Notice of Proposed Rulemaking, the Secretary of Education proposed to amend the Student Assistance General Provision regulations by revising the requirements for compliance audits and adding a new subpart establishing financial responsibility standards. The proposed regulations would require institutions participating in programs authorized by Title IV of the Higher Education Act of 1965, as amended, to meet cutoff scores on certain calculated financial ratios to avoid a compliance audit.           Certainly, institutions of postsecondary education should be held accountable to their constituents, their service area, and the public that provides monetary and other support. However, there is a concomitant reality, that is the reality of the deans, administrators, faculty and staff attempting to manage real institutions in a real world. At this level there is only one question. How do I make good decisions? And that is a powerful question. For it is the sum of the decisions made during the campus year that create the future for an institution. It is the sum of these decisions that lead to outcomes, effectiveness, and efficiency. It is at the decision point where institutional research finds its home and performance indicators have meaning. Offices of institutional research conduct studies and convert data into information for two primary purposes: to support the decision making process by providing analyses that serve to reduce uncertainty prior to making a decision; and to assess how effective the institution has been at meeting the goals and objectives outlined in the campus plan. The former is for internal constituents and the latter for both internal and external constituents.           While there is significant research in postsecondary education on the development of information to support an understanding of the operation and outcomes of the enterprise, further research must be focused on defining decision points. With a taxonomy of decision points, and an understanding of how they interrelate, research can be focused on the amount of uncertainty that is reduced by various performance indicators at given decision points. A clearer understanding of performance indicators and their relationship to decision support must be developed.           This article approaches the use of performance indicators from two perspectives. In the first study eleven frequently cited performance indicators were used to explore the implications of enrollment stability and financial viability with twenty similar Carnegie Classification Baccalaureate II institutions. This study examined issues addressed in the Federal Register Volume 61 proposal to amend the Student Assistance General Provision regulations by revising the requirements for compliance audits and adding a new subpart establishing financial responsibility standards. The implication here was that institutional scores on a specific set of indicators define the financial viability of an institution and should impact the disbursement of federal funds.           The second study used a case study approach to focus on a campus included in the sample of institutions used in the first study. In this particular case study, the institution had decided that challenges on two fronts were threatening the institution. The institution moved to change both the population of students served and the focus of the academic program. The use of information and performance indicators to support decisions related to the repositioning was explored.

Study One Review of the Problem and Literature           Measures of academic programs, staffing, enrollment level, student and faculty characteristics, and revenue and expense can help define an institution's programmatic and financial strengths and weaknesses. At independent institutions, particularly the smaller liberal arts institutions, it is essential that the campus leadership understand the implications of these numeric indicators and their interrelationships. A significant change in the value of key performance indicators at smaller institutions can signify changes that will impact the campus for a given year, or a number of years. With the publishing of the National Association of College and University Business Officers (NACUBO), Financial Self- Assessment: A Workbook for Colleges and Universities, in the early 1980s a move began to understand the campus and campus policy in terms of performance indicators. Certainly, the total quality improvement concept of benchmarks falls along the continuum of work that has been conducted on performance indicators.           There has been significant discussion on the development of performance indicators and their use in higher education. Among the extant models are: the National Association of College and University's Financial Self-Assessment Workbook (1987); Performance Measurement Systems for Higher Education (Kidwell and Long, 1995); Strategic Indicators for Higher Education (Taylor, Myerson and Massy, 1993); and Measuring Up: The Promises and Pitfalls of Performance Indicators (Gather, Nedwek and Neal, 1994). The Joint Commission on Accountability Reporting (JCAR), a project of the American Association of State Colleges and Universities, the American Association of Community Colleges, and the National Association of State Universities and Land-Grant Colleges has produced a framework for accountability reporting recently summarized in the 1996 publication JCAR Technical Conventions Manual. Currently in progress is the NACUBO Benchmarking Project, which is developing quantitative measures to set as a point of reference and standard for basic operations. However, most of the analysis and literature on the development of state defined institution-level performance indicators describes a pattern of implementation with little prior conceptual development and a focus on interinstitutional comparison (Bogue, Creech & Folger, 1993). A 1994 Education Commission of the States study found that performance indicator initiatives in the various states contain many of the same measures (Ruppert, 1994). Most of the states studied used 20 or so indicators that were collected by a governing board and reported in a tabular form. The indicators most commonly used reflected some measure of: instructional inputs; instructional process and use of resources; instructional outcomes; efficiency and productivity; diversity and access; articulation; and relation to state needs.           In the 1987 revision of Financial Self- Assessment: A Workbook for Colleges and Universities (Dickmeyer & Hughes), the concept of an overall institutional equation, defined in terms of key performance indicators, was again emphasized. It was strongly implied in this volume that there were ranges within the various indicators presented that indicated good, moderate or poor performance on a given indicator. It was also implied, in this major work of a standing NACUBO committee, that a certain equation could be inferred for an institution from a combination of these indicators. It was further implied that this unique equation could provide an indication of institutional health, and areas of institutional strength and weakness. Since 1987 a number of institutions have adopted the self-assessment strategy put forth in this volume and a modest research literature has developed. A noticeable addition to this strategy was put forth by Mary Sapp (1994) in the AIR Professional File document, Setting a Key Success Index Report: A How to Manual. A recasting of standard financial ratios to accommodate the Financial Accounting Standards Board's Statement of Financial Accounting Standards No. 116 and 117 was accomplished by Prager, McCarthy and Sealy (1995). These new ratios were cited in the proposal to amend the Student Assistance General Provision regulations.           Given the intense interest in performance indicators, there is a surprising lack of literature examining the policy relevance of indicators of institutional performance. There are few articles assessing the construct validity of the widely used Dickmeyer and Hughes (1987) NACUBO publication. The first study explores a portion of the financial viability framework outlined in Financial Self-Assessment: A Workbook for Colleges and Universities using 20 Baccalaureate II institutions that are members of the College Information Systems Association.           The College Information Systems Association is an association of over 30 liberal arts institutions, each with less than 2,500 FTE students, that share a common data set and share a research office staffed by faculty and graduate students in a higher education doctoral program at a land-grant university. These institutions provide an excellent laboratory for testing performance indicators. These liberal arts colleges and universities are similar in size, age of institution, and academic program offerings. Financial and enrollment data from these institutions offer few confounds. The students are overwhelmingly full-time and seeking the bachelors degree. Revenues come in the form of tuition and fees, gifts, endowment earnings, and auxiliary enterprise charges. Expenses are primarily for faculty and staff, academic support, student life, and physical plant. The primary units of production are the student credit hour and headcount student.           The purpose of this study was to examine the concept of key performance indicators with institutional viability defined in terms of enrollment stability and the ability to meet financial obligations.

Methods           Data were collected from twenty institutions of the College Information Systems Association for a five year period from FY 1992-93 through FY 1996-97 and included 265 measures. Most of these measures were data already being supplied by the colleges to National Center for Educational Statistics and other national organizations such as the Council for the Advancement and Support of Education. Data were collected on revenues, private support, expenditures and transfers, balance sheet items, plant, personnel, faculty development, instruction, student characteristics, financial aid, library holdings, and data processing equipment. A data element dictionary recapitulating and refining national definitions was prepared and taught to the institutions through a series of workshops. Institution level performance indicators were developed from primary data and took the form of primary data; totals of primary data; percentages of total; ratios; and appropriate algorithmic transformations. At the time of the study only twenty of a possible 26 had reported and verified data for the five years under study. The association staff have discovered through this project the difficulty of collecting accurate, timely and comparable data from a number of institutions even with national data standards. Each campus has a number of primary data providers and that will confound any study over multiple campuses.           For the purposes of this preliminary investigation, performance indicators carry the maximum of information when they provide the decision making process with insight into whether an institution is maintaining a steady level of viability; losing viability; or gaining viability. Institutions were defined as viable if they maintain enrollment and maintain financial viability. Of course an essential element of institutional viability is whether an institution is meeting the goals and measurable objectives outlined in the campus plan. Assessing institutional outcomes in terms of consistency with institutional goals was outside the scope of this study.           The first step in this investigation was to develop a set of core indicators that could provide an indication of institutional viability. In Measuring Up: The Promises and Pitfalls of Performance Indicators (Gaither, Nedwek and Neal, 1994) ten core indicators that are found in use and cited most frequently as measures of institutional viability are listed. In the present study that list was modified slightly to focus on the institutional viability construct outlined in Dickmeyer and Hughes (1987). This construct focused on enrollment stability and flexibility in managing available revenues, funds, and expenditures. An eleventh indicator, percent change in fall fulltime equivalent students (fte), was included with the core indicators in order to explore the concept of institutional viability as defined in this study. The eleven indicators used are defined in Table 1.

Table 1 Definition of the eleven performance indicators used in study one. Covered Expenditures - Excess (deficit) of current fund revenues over (under) current fund. expenditures FTE - Fall full-time equivalent students. Percent Change FTE - Percent change in fall full-time equivalent students over previous year Constant Dollar Net Student Revenue - Total tuition and fee revenues minus unrestricted current fund scholarships and fellowships adjusted by the HEPI. Constant Dollar Net Expenditures per Student - Total current fund expenditures and transfers adjusted by the HEPI index divided by fall FTE. Tuition Discount Percentage - Defined as: ((tuition & fee revenues minus unrestricted current fund scholarships an & fellowships) divided by full-time tuition and fee rate)) divided by fall FTE students. Available Funds Ratio - Defined as: (sum of the unrestricted current fund balance, quasi-endowment at market value, and unexpended plant fund balance) divided by unrestricted education and general expenditures plus mandatory transfers. Liquidity of the Current Fund Balance - Defined as: cash in the unrestricted current fund plus investments in the unrestricted current fund divided by liabilities in the unrestricted current fund. Average Faculty Salary - Average salary for all full-time faculty. Acceptance Ratio - Number accepted divided by number applied. Matriculant Ratio - Number matriculated divided by number accepted.           The identified indicators were first examined using descriptive statistics and analysis of variance across all the institutions for five years. The institutions were then divided into two groups defined in terms of their viability based on the stability of the student population and the institution's financial position. For the student population, stability was defined in terms of number of enrolled students and change in number of enrolled students. Financial viability was defined in terms of the institution's ability to meet its financial obligations without significantly changing fund balances and by the NACUBO ratio level definition for liquidity of the current fund balance and availability of fund balances to meet current obligations. The performance indicators defined were then compared within the new groupings of institutions and financial viability and enrollment stability examined using descriptive statistics and multivariate statistics. Results           The twenty institutions for which valid and reliable data were available were all Carnegie Classification Baccalaureate II and similar in academic program offerings. A Pearson Product Moment Correlation Coefficient was calculated for each of the chosen core indicators in relationship with each other for all the institutions for all years. That matrix, found in Figure 1, indicated only four relationships of any magnitude: (1) enrollment was positively related to net student revenues; (2) average faculty salary was positively related to net student revenues; (3) the matriculant ratio was positively related to the applicant ratio; (4) the available funds ratio was negatively related to expenditures per student. Figure 1. Pearson product moment correlation coefficients for the ten core indicators. The first three of these relationships might have been expected. The implication that institutions with a stronger available funds position were expending less per student, though understandable, certainly warranted further study. The lack of other relationships was considered the strongest indication that further study was warranted.           In terms of an overall profile, Figure 2 details five years of percent change in fall fte data for the institutions. Figure 3 details five years of covered expenditures, or the excess (deficit) of current fund revenues over (under) current fund expenditures. As can be seen in Figure 2, in almost every case the institutions managed to maintain or expand enrollment over the five year period. The financial data presented in Figure 3 suggests that two distinct groups could be developed based on ability to meet expenditure demands with available revenues. Figure 2. Percent change in fall FTE from previous year.   FY 92-93 FY 93-94 FY 94-95 FY 95-96 FY 96-97 College A 0.14% 6.60% 9.35% 0.00% 4.02% College B 1.22% 6.65% 5.29% -9.87% 0.82 % College C -0.98% -1.23% 2.75% 4.87% 1.3 5% College D 7.66% -3.20% 4.50% 2.82% 2.95 % College E -20.21% -8.39% 37.94% 2.45% 2.95% College F 18.63% 3.14% 1.52% -3.75% 4.89% College G -8.47% -9.72% -0.31% -0.56% - 4.77% College H -6.79% 0.13% 6.60% 3.03% 0.75 % College I -0.36% 2.61% -7.18% -3.40% -2 .08% College J 0.95% -0.42% 1.29% 117.53% 29.84% College K 9.58% 1.73% -10.88% 8.94% 2.34% College L N/A N/A N/A N/A N/A College M 6.54% -0.60% 0.35% .62% 1.98% College N N/A N/A N/A N/A N/A College O 11.07% 10.08% 13.73% -1.25% 8.41% College P 26.77% 6.99% 8.56% 16.18% 14.62% College Q 5.04% -2.40% -2.69% -2.40% -0 .61% College R 3.65% -3.44% -3.57% 3.82% 0.1 1% College S -0.59% 4.30% -1.69% -2.17% -0 .04% College T -3.14% -6.78% 3.38% -9.59% -4 .04%   Figure 3. Covered expenditures, FY 1992-93 to FY 1996-97.   FY 92-93 FY 93-94 FY 94-95 FY 95-96 FY 96-97             College A $24,071 ($222,895) $322,656 ($305,060) $1,967,207 College B $487,342 $931,612 $1,075,006 ($93,044) $2,421,350 College C ($191,674) ($83,843) $5,298 $85,799 $1,019,395 College D $15,881 ($295,349) ($109,014) ($301,853) ($22,474) College E ($180,268) ($955,771) $118,237 $135,813 $1,105,218 College F ($376,915) $1,185,143 $128,121 ($2,205) ($118,892) College G ($408,189) $1,654,871 ($359,433) ($1,263,470) ($689,241) College H $1,159,043 $28,428 $1,009,956 $619,496 $678,227 College I $49,512 $10,728 ($300,306) $12,480 ($456,712) College J $1,570,039 $2,090,371 $1,578,264 $1,533,880 $210,909 College K $217 $2,707 $1,068 $3,692 $102,458 College L $808,590 $770,227 $361,301 $452,010 ($398,757) College M $753,339 $372,701 $198,771 $468,484 ($179,087) College N ($1,065,616) ($202,832) ($1,861,550) ($1,699,522) ($1,173,749) College O $434,207 $707,564 $300,052 $1,772,279 $951,086 College P $265,177 $154,010 $136,554 $264,129 ($74,804) College Q ($56,006) ($325,000) ($68,577) $33,890 ($115,278) College R ($291,984) ($389,348) ($343,438) ($994,877) ($1,892,805) College S $221,497 $212,235 $799,562 $878,162 $245,226 College T $66,847 ($395,514) ($783,982) $5,523 $68,381           After reviewing the Covered Expenditure data together with the available funds ratio and liquidity of the current fund balance, the institutions were divided into two groups. One group was designated as the strong group and consisted of ten institutions that were able to consistently maintain financial viability as indicated by covered expenditures, available funds ratio and liquidity of the current fund balance. The second group was designated the weak group and consisted of ten institutions that were not able to consistently maintain financial viability as indicated by covered expenditures, available funds ratio and liquidity of the current fund balance. These two groups were used to explore the relationship between the construct institutional viability, defined in terms of the three financial measures and fte, and the information provided by the selected performance indicators.           It was decided to use multiple linear regression to begin to define sets of information that might be related to institutional viability using the two groups identified. The three financial measures and fte were used as dependent variables and each of the ten indicators compared individually as independent variables for all institutions in each group, for all five years. Independent variables with a significant R2 (F-test) and P-value were placed in a multiple linear equation as independent variables with the related dependent variables. The eight independent variables were: Covered expenditures - strong group. Covered expenditures - weak group. Liquidity of the current fund balance - strong group. Liquidity of the current fund balance - weak group Available funds ratio - strong group Available funds ratio - weak group Full-time equivalent students - strong group Full-time equivalent students - weak group           As can be seen in Figure 4, for the institutions that were able to consistently maintain financial viability the dependent variable, covered expenditures, was positively related with the matriculant ratio. This would indicate that the size of the freshman class over the five year period had a significant though small (adjusted R2 = .164) impact on a balanced budget. For the institutions that were not able to consistently maintain financial viability the dependent variable, covered expenditures, was only positively related with the acceptance ratio. The implication here is that becoming less selective over the five year period had a significant though inconsequential (adjusted R2 = .066) impact on decreasing budgetary imbalances. What was interesting in this analysis was not only the minimal impact of the noted effects, but also that none of the other independent variables had an effect for this dependent variable for either group. Figure 4. Significant results for the dependent variable: covered expenditures. Strong group. Regression Statistics Multiple R 0.432 R Square 0.186 Adjusted R Square 0.164 Standard Error 591099.732 Observations 39 ANOVA df SS MS F Significance F Regression 1 2.963E+12 2.963E+12 8.481E+00 6.050E-03 Residual 37 1.293E+13 3.494E+11 Total 38 1.589E+13 Coefficients Standard Error t Stat P-value Intercept -455867.482 376535.084 -1.211 0.234 Matriculant Ratio 2092804.983 718621.433 2.912 0.006   Weak group. Regression Statistics Multiple R 0.298 R Square 0.089 Adjusted R Square 0.067 Standard Error 605017.940 Observations 44 ANOVA df SS MS F Significance F Regression 1 1.496E+12 1.496E+12 4.087E+00 4.961E-02 Residual 42 1.537E+13 3.660E+11 Total 43 1.687E+13 Coefficients Standard Error t Stat P-value Intercept 678212.270 399274.801 1.699 0.097 Acceptance Ratio -2333811.738 1154382.406 -2.022 0.050           For the institutions that were able to consistently maintain financial viability the dependent variable, liquidity of the current fund balance, was positively related to the three independent variables, percent change fte, constant dollar net expenditures per student, and average faculty salary (adjusted R2 = .288), as seen in Figure 5. This would indicate that a consistent growth in the size of the student population is related to financial strength in these institutions. For the institutions that were not able to consistently maintain financial viability, the liquidity of the current fund balance was positively related to the independent variables, covered expenditures and constant dollar net student revenue. The positive relationship evidenced by these two financial variables would be expected. What is interesting is the modest amount of variance that is accounted for (adjusted R2 = .196) by two financial variables that should have a strong relationship with this measure of institutional viability. This could be construed as a fairly explicit indication that other expenditure related pressures must be considered in reviewing the financial viability of these institutions, institutions that have been unable to balance revenue to expense on a consistent basis. As with covered expenditures, what was interesting in this analysis was not only the modest impact of the noted effects, but also that none of the other independent variables had an effect for this dependent variable for either group.   Figure 5. Significant results for the dependent variable: liquidity of the current fund balance. Strong group. Regression Statistics Multiple R 0.598 R Square 0.357 Adjusted R Square 0.288 Standard Error 5.834 Observations 32 ANOVA df SS MS F Significance F Regression 3 529.239 176.413 5.183 0.006 Residual 28 953.016 34.036 Total 31 1482.255 Coefficients Standard Error t Stat P-value Intercept 13.687 8.186 1.672 0.106 Percent Change in FTE 38.001 15.238 2.494 0.019 Expenditures per Student -0.001 0.001 -1.517 0.141 Faculty Salary 0.000 0.000 -0.716 0.480   Weak group. Regression Statistics Multiple R 0.479 R Square 0.230 Adjusted R Square 0.197 Standard Error 1.417 Observations 50 ANOVA df SS MS F Significance F Regression 2 28.131 14.065 7.007 0.002 Residual 47 94.347 2.007 Total 49 122.478 Coefficients Standard Error t Stat P-value Intercept 1.789 0.478 3.740 0.000 Covered Expenditures 0.000 0.000 -2.866 0.006 Student Revenues 0.000 0.000 -1.861 0.069           As can be seen in Figure 6 below, for the institutions that were able to consistently maintain financial viability the dependent variable, available funds ratio, was positively related with full- time equivalent students and tuition discount percentage. This seems to imply that size of the student population, maintained by leveraging tuition, is related to overall institutional financial strength. This effect was one of the larger effects seen in this study (adjusted R2 = .349). For the institutions that were not able to consistently maintain financial viability the dependent variable constant dollar net expenditures per Student was the only independent variable positively related with the available funds ratio (adjusted R2 = .197).   Figure 6. Significant results for the dependent variable: available funds ratio. Strong group. Regression Statistics Multiple R 0.614 R Square 0.376 Adjusted R Square 0.349 Standard Error 1.686 Observations 49.000 ANOVA df SS MS F Significance F Regression 2 78.886 39.443 13.884 0.000 Residual 46 130.682 2.841 Total 48 209.568 Coefficients Standard Error t Stat P-value Lower 95% Intercept 4.513 0.682 6.623 0.000 3.142 FTE -0.002 0.000 -3.582 0.001 -0.002 TDP -5.120 1.236 -4.143 0.000 -7.607 Weak group. Regression Statistics Multiple R 0.439 R Square 0.193 Adjusted R Square 0.173 Standard Error 72.699 Observations 42 ANOVA df SS MS F Significance F Regression 1 50506.737 50506.737 9.556 0.004 Residual 40 211405.735 5285.143 Total 41 261912.472 Coefficients Standard Error t Stat P-value Intercept 81.084 24.070 3.369 0.002 Expenditures per Student -0.014 0.004 -3.091 0.004 The implication of the above results is that, within this group, those institutions that have a higher expenditure level are also financially more viable. The available funds ratio is perhaps the single best measure of an institutions financial viability in that it accounts for all funds that could be marshaled to meet institutional financial obligations. What stands out, using this most inclusive of financial measures, is that the noted effects are due to so few independent variables.           For the institutions that were able to consistently maintain financial viability the dependent variable, full-time equivalent student, was positively related to the two independent variables, constant dollar net expenditures per student and acceptance ratio (adjusted R2 = .439), as can be seen in Figure 7. This result implied that less selective entrance requirements led to a larger student population and so to a larger expenditure base. An alternative explanation for the acceptance ratio effect would be that the market niche of each of these institutions is clearly understood by potential students. The lack of a relationship with tuition discount percentage and constant dollar net student revenues might also suggest that, in this group, the larger institutions scholarship with restricted funds as opposed to leveraging with unrestricted current funds. For the institutions that were not able to consistently maintain financial viability the independent variables, tuition discount percentage and acceptance ratio, were positively related to the dependent variable full-time equivalent student ratio (adjusted R2 = .323).   Figure 7. Significant results for the dependent variable: full-time equivalent student. Strong group. Regression Statistics Multiple R 0.687 R Square 0.473 Adjusted R Square 0.439 Standard Error 453.587 Observations 34 ANOVA df SS MS F Significance F Regression 2 5716405.062 2858202.53 13.892 0.000 Residual 31 6377963.398 205740.755 Total 33 12094368.46 Coefficients Standard Error t Stat P-value Intercept 2884.694 495.518 5.822 0.000 Expenditures per Student -0.348 0.067 -5.163 0.000 Acceptance Ratio -869.179 777.033 -1.119 0.272 Weak group. Regression Statistics Multiple R 0.599 R Square 0.359 Adjusted R Square 0.323 Standard Error 420.206 Observations 39 ANOVA df SS MS F Significance F Regression 2 3556039.554 1778019.78 10.070 0.000 Residual 36 6356641.902 176573.386 Total 38 9912681.456 Coefficients Standard Error t Stat P-value Intercept -131.099 314.892 -0.416 0.680 Tuition Discount Percentage 1427.638 642.304 2.223 0.033 Acceptance Ratio 2458.980 1026.606 2.395 0.022   For this group, the implication is that the institutions with a larger student population accept more potential matriculants and leverage the cost to attend with unrestricted current fund dollars. Taken together, these two results clearly suggest that some combination of less selectivity or identification to market niche combined with a higher level of financial aid, or leveraged tuition, was related to a larger student population in both groups. Discussion           Taken together, the results related to these performance indicators suggest that the recruitment and retention program is an important source of institutional financial viability. The results indicate that leveraging the cost to attend is integral to maintaining and expanding the student population for these institutions. The implication was that all the institutions discount tuition, though the financially more viable institutions were seen to rely less on discounting and more on funded scholarships. The performance indicators used for this study are among the most frequently cited as measures of institutional viability and the results did provide information related to institutional financial viability.           This study did demonstrate that there are unique groupings of liberal arts institutions and that unique financial equations for these groups might be defined in terms of several performance indicators. However, what does stand out is that there are few policy-related implications. These institutions, most of which have been in existence for over a century, are maintaining enrollment and graduating students. Some have more financial flexibility than others and that can be traced to size of enrollment and cost to attend. These standard financial ratios were being considered in a number of states as triggers for audits during deliberations related to the Statewide Postsecondary Review Entities (SPRE). Equations involving these financial viability indicators are being considered in the proposal to amend the Student Assistance General Provision regulations by revising the requirements for compliance audits, as detailed in the Federal Register Volume 61. Certainly these results did not imply that these frequently cited performance indicators should trigger federal policy and institutional sanctions.           The results of Study One did suggest that serious consideration should be given to questions of institutional viability, unique institutional profiles and the use of performance indicators in institutional management. Study Two explored a whole system approach developed around the concept of decision support as suggested by Kaufman in Educational System Planning (1972). One of the institutions included in the first study was used for the case study approach employed in Study Two. Study Two Review of the Problem and Literature           Institutions of higher education are, by any standard, complex entities. Even the least complex of institutions, the small liberal arts college, provides an enormous number of pedagogical, social, behavioral and economic phenomena to study. As campus decision-makers begin to understand these phenomena they become more effective at defining and creating the information needed to support decision making. The campus year might be envisioned as multiple threads woven together. Among these threads would be the recruitment and retention thread, an academic programs thread, a student life thread, a staffing thread, a physical plant thread, and a fiscal thread. Along each of the threads lie decision points. The sum of the decisions at these points are instrumental in creating the fabric and design of an institution's future.           It is a fairly straightforward task to list some of the critical decision points in the campus year and the questions they raise. What decision rule will we use for admitting students? How will financial aid be apportioned? Will there be unfunded financial aid, and if so how much? Will there be a raise? Can maintenance be deferred? What programs will be targeted for excellence and at what expense?           The answers to these questions, and a myriad more that confront the campus administrative and planning team will be cast in terms of decisions. At the very least, the leadership of every campus must ask the following two questions at the beginning of each academic and planning year. First, will we be intentional in making decisions for this campus? And, will we use the best possible information to reduce uncertainty before we make decisions? Assuming that decisions are to be intentional, our primary concern then is the need to reduce uncertainty before the decision is made. It is the role of institutional research to provide the information that reduces uncertainty prior to making decisions.           There are a number of decision points during the campus year encompassing a number of dimensions from departmental decisions to decisions with campus-wide implications. From a temporal perspective, there are decisions that are made daily, weekly, each academic term, and yearly. Almost all of the literature related to decision support focuses on the for-profit business and industry sector. This literature began to call for, and then examine, integrated decision support systems (DSS) starting in the early 1970s (Van Gundy, 1988). These analytical software engines were intended to provide the necessary decision support information at the appropriate desk for everything from daily to annual decisions throughout the firm (Alavim and Joachimsthaler, 1992). Implementation of completely integrated decision support systems in the for-profit sector has been marked by mixed results and the implementation of such systems remains a complex issue (Lucas, Ginzberg, and Schultz, 1990). The control of operations and support for marketing have seen a wide spread acceptance and use of decision support tools, primarily for daily, weekly and quarterly decisions (Alavim and Joachimsthaler, 1992). The literature on the use of decision support systems for major policy and direction related issues has shown that there is far less consensus on the use of DSS by top-level management (Reagan-Cirincione et al., 1991).           The acceptance of decision support systems in postsecondary education is similar to the experience of the for-profit sector, though the literature is not as rich. Most of the administrative software systems in use by the institutions provide adequate support for daily, weekly, and academic term decisions. The marketing function, embodied in the admissions and development programs, have become quite sophisticated. However, the use of information to support decisions related to the major policy, performance, and direction related issues faced by institutions leaves much to be desired (Gaither, Nedwek and Neal, 1994; Kidwell and Long, 1995).           For the purposes of this research, those decisions will be defined as decision points. Information developed to support those decisions is defined as a performance indicator (PI). For example, the decision to admit or not admit a student is, in fact, a daily or weekly decision. However, setting a decision rule that some measure, such a s school class rank, will be used as an admission criteria is probably done only once a year. This is a key decision point. The information used to make that decision, probably developed from a retention study and related descriptive statistics, would be defined as key performance indicators.           Though there is a large body of institutional research literature, that literature should be strengthened in three areas: 1. There is a need to develop a taxonomy of key decision points within the campus year; 2. There is a need to understand what key performance indicators reduce uncertainty prior to making a given decision and the impact of the information on decision making; 3. There is a need to understand the campus as a system defined by decision points and sets of decision points that are interrelated.           From a practical perspective, decisions are approached, and usually made, within the context of the institution's program structure. A framework for program structure was established nationally in the 1960s and has evolved into the current national program classification structure defined in NACUBO's Administrative Service and implicit in the National Center for Educational Statistics Integrated Postsecondary Education System. Specific decisions will be made relative to the goals or budgetable objectives of a specific program or the cost centers defined at the sub-program or sub-sub-program level of the Program Classification Structure. Decisions are also within a temporal plane and related to specific times within the academic or fiscal year. A decision point is defined here as related to a specific program at a specific time in the academic or fiscal year. A decision model of the campus could be made that resembles a PERT chart with each line representing a program and the action points representing decision points.           Decision points can also be characterized in terms of the type of decisions that are made. Most will be regular and identifiable, located within the aegis of a program and at a specified time within the year or academic term. Other decisions will be unexpected and will encompass either new opportunities or decisions that need to be revisited. Decisions that need to be revisited are inevitable, even the best plans will require mid-course corrections.           Specific decisions are made and there are discreet decision points. However, decisions are rarely made in a vacuum. Specific decision points group together within decision sets. The information that is developed for the reduction of uncertainty at each decision point within a decision set is often reviewed together. Specific decisions are made within the context of the decision set.           If the most appropriate framework for decisions is the decision set, the nature of decision sets can best be described as a cascade. Even single decisions can lead to a cascade of additional discrete decisions. Multiple measures of outcome can be impacted in the same way. Perhaps the most important skill in policy analysis is being able to understand and predict the cascade effect.           This case study focuses on a campus that was included in the sample of institutions used above to explore the use of performance indicators as measures or predictors of institutional viability. In this particular case study the institution decided, in FY 1991- 92, that challenges on two fronts were threatening the institution. The first challenge was in the retention of students, with only 40% of entering freshmen returning for the second year. The institution was convinced that this was unacceptable in terms of cost to the institution to recruit a large freshman class, and in relation to the mission of the institution. The second challenge was in the construction of two new state supported branch community college campuses serving nearby counties. These counties had traditionally been a source of students for the institution, though many of these local students required remediation. The institution had a number of medial courses included within the academic program. Decision Sets: A Case Study Retention Challange           The first step was to collect information from the student record files and an entering freshman survey the institution had been administering and conduct a probability regression analysis to determine factors that correlate with retention into the second year. The results of that study are outlined in Figure 8. Figure 8. Factors related to retention the second year. Dependent Variable: First-time freshmen returning for second year. Independent Variables Investigated with probit analysis ACT English Amount of Loans ACT Math Amount of Workstudy Hours ACT Social Studies or Reading Distance from Home ACT Science Dorm Student ACT Composite College Grade Point Average Graduation Quartile Gender High School Grade Point Average Married Amount Non-institutional Aid Elected Major Amount Institutional Aid Religious Preference Groupings of variables found to significantly increase the probability of returning for the second year with probit analysis Group One Group Two College Grade Point Average Dorm Student Institutional Aid None Institutional Aid ACT Composite College Grade Point Average Elected Major High School Grade Point Average Dorm Student Loans Given the results of the retention study, four policy related decisions were made. All freshmen were required to live in a college dorm except those living with a relative. Admissions standards were refined and evaluation of all applicants was moved to a faculty committee using a multiple criteria best-fit model. Policy for awarding financial aid was changed to focus on students most likely to be retained. Faculty began to work with students on electing a major before arriving on campus. Figure 9. Percent of freshman returning for the second year.             As can be seen in Figure 9 above the percent of freshmen returning for the second year rose dramatically from Fall 1992 to Fall 1996. Also, an intentional decision was made to increase the use of financial aid to recruit students who were more likely to be retained. Figure 10 shows the increase in scholarship and fellowship aid per full-time student from FY 1992-93 to FY 1996-97. As significant as these changes are, they should be explored within the context of a related, yet separate, decision set that was being addressed at the same time. Figure 10. Scholarship and fellowship aid per full-time student.     Community College Challenge           A significant segment of the institution's overall enrollment profile was from the counties surrounding the campus. Two community college branch campuses were opening in counties adjacent to the campus. Further examination of the student data set revealed that many of the students in the institution's freshman year remedial program were, in fact, students that would be candidates for these branch campuses and their open door policies. The institution made four key decisions: to focus on students that have the highest chance of retention; to withdraw from remedial programs and leave those students to community colleges; to reduce the size of the student body concomitant with the withdrawal from remedial programs; to reduce the size of the faculty relative to the size of the reduction in the student population.           As can be seen in Figure 11, the student population was significantly reduced between Fall 1991 and Fall 1996. There was an equivalent reduction in the faculty that occurred during that period of time. Faculty positions were reduced by 10% over the five year period with reductions related to remedial courses that were dropped. These faculty were in several disciplines and attrition and early retirement were the chief reduction in force strategies followed. Figure 11. Fall enrollment 1991-1996.           A series of other decisions and results cascaded from these initial decisions. Some of these are outlined below. To attract and retain the caliber of student that had been identified as most likely to persist, the retention studies suggested some increase in financial aid. Though reduction in faculty offset most of the loss in student tuition and fee revenues, substantial increases in tuition and fees were necessary. Analysis had indicated that the institution was underselling its product. The ACT scores of new freshmen increased dramatically as well as related measures of previous academic success.           Through intentional analysis and decision making the institution had changed the profile of its student body and reduced the size of the faculty. Though only one measure of the entering freshman class, the changing ACT profile, as seen in Figure 12, is indicative of the new more rigorous decision model for admitting students applied by the new admissions procedures. Figure 12. Fall 1991-1996 new freshman ACT composite scores.           Perhaps the cascading nature of decision sets is also seen in the impact on the current funds. Figure 13 below shows the change in current fund expenditures as a percent of total from FY 1992-93 to FY 1996-97. The most significant feature of this period is the shift in expenditures from instruction to scholarships and fellowships. An additional impact is seen in reviewing tuition increases and the behavior of tuition and fee revenues at this institution during the five years being studied. Figure 14 indicates that tuition and fee revenue per student, net of scholarship, rose over the five-year period being studied. This was due to significant increases in tuition and fees and a decrease in number of students recruited. The institution was successful at recruiting a more academically prepared and affluent student population. Figure 13. Current fund expenditures as a percent of total from FY 1992-93 to FY 1996-97. Expenditures by program FY 92-93 FY 93-94 FY 94-95 FY 95-96 FY 96-97 Instruction $4,357,598 $4,452,393 $4,267,199 $4,343,119 $4,393,923 Academic Support $734,794 $751,267 $757,292 $822,828 $887,878 Student Services $3,209,702 $3,042,999 $3,686,927 $4,258,749 $3,648,876 Institutional Support $2,742,729 $2,613,368 $2,581,843 $2,743,601 $2,477,247 Operation & Maintenance of Plant $1,295,546 $1,445,542 $1,336,703 $1,556,446 $1,579,062 Scholarships & Fellowships $3,512,193 $3,498,727 $3,278,937 $3,702,188 $5,512,980 Mandatory Transfers $10,603 $11,971 $11,971 $19,031 $35,741 Auxiliary Services $2,526,940 $1,909,342 $1,908,835 $2,297,643 $3,528,274 Total Expenditures $18,390,105 $17,725,609 $17,829,707 $19,743,605 $22,063,981 Expenditures by program as a percent of total expenditures Instruction 23.7% 25.1% 23.9% 22.0% 19.9% Academic Support 4.0% 4.2% 4.2% 4.2% 4.0% Student Services 17.5% 17.2% 20.7% 21.6% 16.5% Institutional Support 14.9% 14.7% 14.5% 13.9% 11.2% Operation & Maintenance of Plant 7.0% 8.2% 7.5% 7.9% 7.2% Scholarships & Fellowships 19.1% 19.7% 18.4% 18.8% 25.0% Mandatory Transfers 0.1% 0.1% 0.1% 0.1% 0.2% Auxiliary Services 13.7% 10.8% 10.7% 11.6% 16.0%   Figure 14. Net tuition and fee revenue per student. Discussion           This article approached the use of performance indicators from two perspectives. In the first study eleven frequently cited performance indicators were used to explore the implications of enrollment stability and financial viability with twenty similar Carnegie Classification Baccalaureate II institutions. This study examined issues addressed in the Federal Register Volume 61 proposal to amend the Student Assistance General Provision regulations by revising the requirements for compliance audits and adding a new subpart establishing financial responsibility standards. The implication here was that institutional scores on a specific set of indicators define the programmatic and financial viability of an institution and should impact the disbursement of federal funds. These results did not imply that these frequently cited performance indicators should trigger federal policy and institutional sanctions. What did stand out is that there are few policy related implications that can be drawn from these internationally accepted institutional viability measures. These institutions, most of which have been in existence for over a century, are maintaining enrollment and graduating students. Some have more financial flexibility than others and that can be traced to size of enrollment and cost to attend. What does stand out is that there are few policy-related implications.           The second study used a case study approach to focus on a campus included in the sample of institutions used in the first study. In this particular case study, the institution had decided that challenges on two fronts were threatening the institution. The institution moved to change both the population of students served and the focus of the academic program. The institution was successful over a five-year period in changing both the character of the student body and the academic program mix while improving its overall financial position. The institution used performance indicators within a whole system context, as suggested by Kaufman in Educational System Planning (1972), to reduce uncertainty before changing institutional policy and to measure the outcomes of those changes.           This case study was seen by the authors to reinforce their belief that specific decision points group together within decision sets. Information that was developed for the reduction of uncertainty at each decision point was reviewed, and decisions made, within the context of the decision set. The belief that decision sets exhibit cascade effects was also reinforced. In this case study single decisions led to a cascade of additional discrete decisions. As well, multiple measures of outcome were impacted in the same way.           Three overall conclusions were reached as a result of these two studies. First, the performance measures most commonly cited in the literature as measures of institutional financial viability are of limited use for institution specific policy development. Second, performance indicators are most effectively used within an institution specific, whole system framework. Third, being able to understand and predict the cascade effect in the use of performance indicators is essential for effective policy analysis. References Alavim, M. & Joachimsthaler, E.A.(1992). Revisiting DSS implementation research: a meta-analysis of the literature and suggestions for researchers. MIS Quarterly, 16 (10) 95- 116. Bogue, G. Creech, J. & Folger, J. (1993). Assessing quality in higher education: policy actions in SREB States. Atlanta, GA: Southern Regional Education Board Dickmeyer, N.& Hughes, K. S. (1987). Financial self- assessment workbook: a workbook for colleges and universities. Washington, DC: National Association of College and University Business Officers. Gaither, G., Nedwek, B.P. & Neal J.E. (1994). Measuring up: the promises and pitfalls of performance indicators in higher education. (ASHE-ERIC Higher Education Report; No. 5.) Washington, D.C.: The George Washington University, Graduate School of Education and Human Development. Joint Commission on Accountability Reporting. A need answered. Washington , DC: American Association of State Colleges and Universities. Lucas, H.C., Ginzberg, M.J. & Schultz, R.L. (1990). Information systems implementation: testing a structural model. Ablex Publishing Corporation, Norwood, NJ. Kaufman, R. A. (1972). Educational Systems Planning. Englewood Cliffs, NJ: Prentice Hall Kidwell, J.J. & Long, L. (1995). Performance measurement systems for higher education. (NACUBO's Effective Managment Series) Washington, DC: National Association of College and University Business Officers. Notice of Proposed Rule Making, Number 184, Volume 61, Federal Register (1996) Reagan-Cirincione, P, Shuman, S., Richardson, G.P. & Dorf, S.A. (1991). Decision modeling: tools for strategic thinking. Interfaces, 21:6, 52-65. Ruppert, S. (ed) (1994). Charting higher education accountability. Boulder, CO: Education Commission of the States. Prager, McCarthy & Sealy, Inc. (1995). Ratio analysis in higher edeucation. New York, NY: Prager, McCarthy & Sealy. Sapp, M.M.(1994). Setting up a key success index report: a how-to manual. (AIR Professional File Number 51, Winter, 1994) Tallahassee, FL: Association for Institutional Research. Taylor, B.E., Myerson, J.W. & Massy,W.F. (1993). Strategic indicators in higher education: improving performance. Princeton, NJ: Peterson's Guides. Van Gundy, A.B., Jr.(1988). Techniques for Structured Problem Solving, second edition, Van Nostrand Reinhold, New York. About the Authors E. Raymond Hackett is Executive Director of the College Information Systems Association, a data sharing consortium of liberals arts colleges and universities. He is also an Assistant Professor in the Educational Leadership program at Auburn University. Dr. Hackett has served as a state system-level policy analyst and as a campus vice president for administration and finance. Email: hackera@mail.auburn.edu Sarah D. Carrigan is Coordinator of Institutional Studies at the University of North Carolina at Greensboro and formerly served as the Research Director for the College Information Systems Association. Dr. Carrigan has also served as a campus residence life director at a liberal arts college.

Copyright 1998 by the Education Policy Analysis Archives The World Wide Web address for the Education Policy Analysis Archives is http://olam.ed.asu.edu/epaa General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. (602-965-2692). The Book Review Editor is Walter E. Shepherd: shepherd@asu.edu . The Commentary Editor is Casey D. Cobb of the University of New Hampshire: casey.cobb@unh.edu . EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Andrew Coulson a_coulson@msn.com Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov hmwkhelp@scott.net Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Marshall University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Richard M. Jaeger University of North Carolina--Greensboro Daniel Kallós Umeå University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Rocky Mountain College Dewayne Matthews Western Interstate Commission for Higher Education William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton apembert@pen.k12.va.us Hugh G. Petrie SUNY Buffalo Richard C. Richardson Arizona State University Anthony G. Rud Jr. Purdue University Dennis Sayers Ann Leavenworth Center for Accelerated Learning Jay D. Scribner University of Texas at Austin Michael Scriven scriven@aol.com Robert E. Stake University of Illinois--UC Robert Stonehill U.S. Department of Education Robert T. Stout Arizona State University    archives   |   abstracts   |   editors   |   board   |   submit   |   comment   |   subscribe   |   search