The Effects of Performance Budgeting and Funding Programs on Graduation Rate in Public Four-Year Colleges and Universities

This study was conducted to determine whether states with performance budgeting and funding (PBF) programs had improved institutional performance of higher education over the five years (1997 through 2001) considered in this study. First Time in College (FTIC) graduation rate was used as the measure of institutional performance. In this study, the unit of analysis is institution level and the study population is all public four-or-moreyear institutions in the United States. To test PBF program effectiveness, Hierarchical Linear Modeling (HLM) growth analysis was applied. According to the HLM analysis, the growth of Page 1 of 27 EPAA Vol. 12 No. 22 Shin & Milton: The Effects of Performance Budgeting and Funding Pr... 5/25/2004 http://epaa.asu.edu/epaa/v12n22/


Introduction
The purpose of this study was to examine the effects of performance budgeting and funding (PBF) programs on the performance of public higher education institutions in the United States using longitudinal performance data.As of 2001, thirty-six states in the U.S. utilized either performance budgeting or performance funding programs, or both, in institutions of higher education (Burke & Minassians, 2001).This trend began during the 1990s, as state policy-makers sought to enhance accountability in higher education, find more effective budget allocation methods, and in some cases, trim budgets.Although performance budgeting (PB) and performance funding (PF) are different in focus, both create a link, to different degrees, between budget allocation and institutional performance.In addition, both types of programs are based on the underlying premise that higher education institutions are motivated to improve their performance when performance is linked to budget allocation.
The trend toward increased use of performance budgeting and funding programs is not limited to the U.S. The United Kingdom led other European countries in initiating a market-driven higher education accountability program, first introduced by the Thatcher government.Influenced by trends in the U.S and Europe, the Canadian provinces of Alberta and Ontario have also adopted performance-based funding mechanisms to assess university efficiency and quality (Barnetson 1999).In Australia, 40 public research universities and two private institutions are subject to an accountability framework that links their funding to performance requirements (Atkinson-Grosjean & Grosjean, 2000).In spite of vigorous protests from students and faculty, New Zealand established a new accountability system in 1998.In the early 1990s, South Korea adopted a performance-based funding program consisting of nine independent sub-programs, each with a particular purpose, to which funding was tied (Kim, 2001).

Current Status Of Pbf Programs Major Stakeholders: State Politics and Business Involvement
The major stakeholders in higher education reform of the 1990s have been state legislatures, state chancellors and state agencies, governors, and higher education institutions (Blackwell & Ciston, 1998).Among these stakeholders, governors and state legislatures play the primary role in the adoption of PBF programs (Burke & Minassians, 2001).In contrast, a relatively small number of initiatives (in 24 states) have been conducted within state agencies or university systems themselves.
A more complete understanding of the widespread adoption of PBF programs since the 1990s must encompass more than the vested interests of state legislatures and governors.In the 1990s, higher education reform has been guided by economic values-competition, productivity, and efficiency-more than ever before.This shift in priorities might be explained in part by the growing interest of business leaders in state higher education policy-making (Usdan, 2002;Zernike, 2002;Zumeta, 2000).
Business interests have changed the picture of higher education.As Levin (2001) argues, higher education is becoming like a "globally competitive business" (p.237).In some cases, business leaders are actively involved in PBF program development.For example, in South Carolina, which links 100% of higher education funding to institutional performance, two of the 12 members of the Higher Education Joint Legislative Study Committee represented the business community.[Not one of the 12 members of the committee came from higher education (China, 1998)].
However, other major stakeholders in higher education-institutional administrators and faculty-have always been more concerned about institutional improvement than accountability, unlike politicians who emphasize external accountability through the implementation of PBF programs (Burke, 1998).Because of these perceptual and value differences, tensions almost invariably exist between both groups around the issue of PBF programs.

Type of Program: Performance Budgeting or Funding?
Performance budgeting and funding are similar in that both programs link institutional performance with budget allocation, but the methods differ in the way each ties institutional performance to state funding.While performance funding (PF) programs link budget to institutional performance in a direct, automatic, and formulaic manner, the link in performance budgeting (PB) programs is loose, indirect, and uncertain (Burke et al, 2000).
Because of the loose link between performance and budget allocation, PB programs may be less effective in enhancing institutional performance.Notwithstanding this limitation, state policy-makers typically prefer performance budgeting to performance funding because of the flexibility in implementation (Burke, et al., 2000).
Although more states adopted PB programs than PF programs, recent trends indicate, "both programs are borrowing elements from the other approach to gain its benefits while evading their own problems" (Burke et al, 2000, p.3).Further, in order to minimize the limitations and maximize the strengths of each program, an increasing number of states have adopted both programs at the same time (Burke et al, 2000).Ruppert (1995) conducted an in-depth case study on performance indicators (PIs) and performance reporting in ten states that have been among the leaders in designing and using PIs in higher education.(Note 1) He found that most states share a common core of performance indicators.Graduation outcome data, for instance, is one measure that responds to policy concerns -rising college costs and the economic return to the state of college-educated citizens.

Commonly-Used Indicators for measuring institutional performance
A survey by the State Higher Education Executive Officers (SHEEO) conducted between 1996 and 1997 identifies some indicators used frequently in the U.S. at that time: graduation rate (32 states), transfer rate (25 states), faculty workload/productivity (24 states), follow-up satisfaction (23 states), and externally sponsored research funds (23 states) (Christal, 1998).Another study (Burke, 1998) of 10 states conducted in 1997 found less commonality among PIs in different PBF programs.Among the states surveyed, the following performance indicators were in use: graduation and retention rates (10 states), two-to-four-year transfers (6 states), faculty workload (5 states), institutional choice (5 states), graduation credits and time to degree (4 states), licensure test score (4 states), transfer graduation rates (4 states), workforce training and development (4 states), and external research fund ratio (3 states).Overall, the common performance indicators across these studies were graduation rate, retention rate, transfer rate, faculty workload, and sponsored research funds.

Previous Evaluations of the Impacts of Pbf Programs
Most evaluative studies of the impacts of PBF programs focus on a specific state or set of institutions within a state.As many studies assert, the effectiveness of PBF programs is a critical concern of policy-makers.Nevertheless, few academic evaluation studies of PBF program impacts have been conducted, save the occasional evaluation project or dissertation.State-level evaluations for budget allocation purposes mainly use quantitative data on specific performance indicators.Florida and South Carolina, for example, annually evaluate each institution's performance in order to guide budget allocation.
In a book on PBF programs, Bogue (2002) explores the impact and effectiveness of the PBF program in Tennessee.From the actual 1994-1998 performance data, he found that the universities have consistently scored above the national norm on the College BASE.One interesting outcome of the Tennessee PBF program is the clear increase in the proportion of academic program accreditations, which improved from 65 percent to 100 percent (Bogue, 2002).He found, however, that institutions generally failed to significantly raise persistence-to-graduation rates and overall job-placement rates over the same period.
Florida's Office of Program Policy Analysis and Government Accountability (OPPAGA) (2001) evaluated the performance budgeting program of the State University System in Florida as part of an evaluation of performance budgeting programs at each state agency.In the evaluation, OPPAGA reported that graduation and retention rates have increased since the state adopted a performance budgeting program, and externally-funded financial support for the research program has significantly increased as well.Additional confirmation of these findings exists in Florida's relatively high score on completion indicators versus other evaluation indicators in the 'Measuring Up 2000' study.OPPAGA claims this result occurred because the Florida Legislature decided to make raising the completion or graduation rates a priority in its indicator system.
The South Carolina Commission on Higher Education (CHE) (2002) has been studying the impacts of performance funding program in South Carolina as a three-year project for the Fund for Improvement of Postsecondary Education (FIPSE) grant.In this evaluation, the South Carolina CHE used diverse methods (journal analysis, surveys, interviews, and historical data) to review its PF program.According to preliminary findings, institutional policies have changed to accommodate the PF program and some outcomes are changing as well.Interestingly, the impacts of the PF program seem to vary depending on which evaluation method is used in the evaluation process.For example, respondents to the survey answered that the PF program has had no impact on institutional quality or efficiency.Yet historical data on institutional performance shows that performance on SAT scores, minority enrollment, and externally funded research have all increased, and that graduation rates are slowly rising as well.These results suggest that the actual performance of South Carolina higher education institutions has been increasing since South Carolina adopted the PF program.
In a review of performance data from Missouri higher education institutions, Marcus et al. (2000) found that student assessment scores, licensure exam pass rates, first time in college (FTIC) graduation rates, minority graduation rates, and student job placement rates have all increased since Missouri adopted a performance funding program.While the causal relationship between PF and increased performance was not conclusive, the PF program was determined to be "clearly responsible for the identification of priorities for funding, for the establishment of assessment measures, and for helping institutions to accept that part of their state allocation is linked to results" (p.215).
The studies mentioned above indicate that PBF programs have had a positive impact on institutional performance.However, these results are based on limited time periods, and the interpretation of the performance data can differ from one researcher to another.For example, Dallet, et al. (2002) analyzed the same performance data used in an earlier study by OPPAGA (2001) to discuss the impact of Florida's PBF program.While OPPAGA reported that the PBF program had had a positive impact on Florida's higher education institutions, the conclusion of Dallet and his colleagues was quite different.They argued that, although the graduation rate and retention rate in Florida's higher education institutions increased somewhat after the inception of the PBF program in 1994, this growth actually occurred at a level similar to that of the 1990-91 academic year, when no such policy was introduced.In Colorado, which adopted a PF program from 1994/95 to 1996/97, the effect of performance level on budget reward was found in a regression analysis to be lower than expected (Bridges, 1999).Instead, there was a higher correlation between institution size and performance funding than between performance funding and performance level.

Research Questions
The overarching purpose of this study is to explore the impacts that PBF programs have had on institutional performance.Thus, there are two basic research questions addressed in this study: Has institutional performance in states with PBF programs (PBF states) improved more than in states without PBF programs (non-PBF states)? 1.
Has institutional performance in states with both programs (PB and PF) improved more than in states with only a PB or PF program? 2.

Unit of Analysis and Population
The unit of analysis for this study is higher education institution, rather than the state, because the target of PBF programs is institution-level change, and state-level performance is simply the aggregation of each state's institution-level performance.The study will be limited to public higher education institutions because private institutions are not the main objects of state policy.Further, although some states have the same PBF programs for public four-year institutions and community colleges, many states (e.g., Florida, California, New York) have different PBF programs for public four-year institutions and community college systems (Burke, 1998).In addition, as Hoyt (2001) addresses, the missions of community colleges are different from those of four-year colleges.Therefore, for this study the object of interest is limited to public four-year institutions, and excludes public community colleges.Also, because PBF programs focus mainly on undergraduate education, graduate-only institutions will be excluded from the analysis.Thus, the target population of this study is all public four-year higher education institutions in the United States.Based on the criteria above, 456 institutions were selected for the study population.(Note 2) The information on the annual PBF program status of each state is available from surveys conducted each year from 1997 through 2001 (Burke & Minassians, 2001;Burke et al., 2000;Burke & Modarresi, 1999;Burke & Serban;1998;Burke & Serban, 1997).

Dependent Variable
The dependent variable used in this study is the First Time in College (FTIC) graduation rate of each institution during the years 1997 through 2001.This dependent variable was selected based on three criteria: 1) it is the most commonly-used performance indicator nationwide; 2) nationwide data are readily available; (Note 3) and 3) there is internal validity in using graduation rate as a measure of performance in higher education.The term is also in accordance with the Student Right-To-Know Act (SRTK), which mandated institutions to report the graduation rate of only "first-time" and "full-time" students who spent "up to 150 percent of normal time to complete their degrees" (9PL 101-542;Federal Register, 1992).
In conducting a broad policy studies such as this, one must consider the question of whether a given policy is accomplishing its goals (Affholter, 1994).In PBF programs, the attainment of program goals is measured using performance indicators that are chosen by state legislatures, coordinating boards, or in some cases, higher education institutions themselves.According to an analysis of performance indicators nationwide (Christal, 1998), graduation rate is the most commonly-used indicator in U.S. higher education institutions with PBF programs.Of the 33 states with PBF programs in 1997, 32 states used graduation rate as one of their performance indicators.
In addition, when an inappropriate or weak indicator is chosen as a dependent variable, it can invalidate the results of a policy study.Each institution admits applicants who are deemed qualified to study at the institution.These students study approximately four years to satisfy graduation requirements that are set forth by the department, college, university, or state.If many students do not graduate in a timely manner, these students have either not yet satisfied the graduation requirements, have chosen to leave, or have transferred to another institution.Any of these cases poses a problem for the institution, which, due to the very nature of its mission, seeks to maintain a high graduation rate.As a result, low graduation rates may imply that the institutions are falling short in performing some of their functions.Therefore, graduation rate is a persuasive and valid indicator to represent how well the institution performs in at least one of its many goals.(Note 4) FTIC graduation rate is thus calculated using the completions within 150% of normal time to degree divided by the total number in a particular year's cohort.The Integrated Postsecondary Education Data System (IPEDS) provides total cohort information, completion within 150% of normal time to degree, and the final FTIC graduation rate.Therefore, for the purpose of this study, the final graduation rate provided by the IPEDS will be used without any adjustment.

Independent Variables
Independent variables in the statistical model will be of two types: program (or treatment) variables, and control variables.Institution-level control variables are variables that have been shown to affect graduation rate at the institution level.The average tuition has been associated with graduation rates (Heller, 1997;Hsing & Chang, 1996;Dayhoff, 1991).The availability of financial aid (grants, loans, work study, etc.) has a positive impact on enrollment (John et al., 2001;Braunstein et al, 1999;Sheridan et al., 1994;Cabrera et al., 1992).The opportunity to live in a campus residence hall, especially during the freshman year, has also been shown to influence retention rates and time to completion (Astin, 1997;Sheridan et al., 1994).
Student Dormitory Ratio is defined as (space availability in dormitory facilities)/(full-time students). b.
Financial aid student ratio is the percentage of full-time, first-time degree and certification seeking students who receive any financial aid (grants, loans, assistantships, fellowships, tuition wavier, tuition discounts, veterans benefits, employer aids and other monies). c.
State-level control variables are state level characteristics which have been shown to influence graduation rate, and which must be included in the causal model in order to capture accurately the effects of PBF programs.Astin (1997), Schmitz (1993), and Kahn and Nauta (2001) found that college preparation factors (mean entrance test score on ACT or SAT, and freshmen's high school grades) are positively associated with graduation rate.Also, unemployment rate has been shown to be negatively associated with graduation rate (Heller, 1999;Hsing & Chang, 1996).Statewide family ability to pay college costs is the variable that reflects financial support from the family to the students (Braunstein et al., 1999;John, 1993) as well as the economic conditions of each state.
College preparation is defined as "Overall score" of college preparation (Note 6) as described in "Measuring Up 2000" (NCPPHE, 2001). a.
Family ability to pay college costs is defined as the student's family's ability to pay at public four-year colleges, as described in "Measuring Up 2000" (NCPPHE, 2001). b.
Unemployment rate is defined as the number of unemployed as a percent of the labor force (U.S.Department of Labor, 2002). c.

Data Sources
Study data are available directly or indirectly from diverse sources.Data on PBF program variables have been collected in annual surveys conducted by Burke and associates in the years 1997 through 2001 (Burke & Minassians, 2001;Burke et al., 2000;Burke & Modarresi, 1999;Burke & Serban, 1998;Burke & Serban, 1997).Nationwide data on state-level control variables were included in the nationwide performance evaluation study, "Measuring Up 2000" (NCPPHE, 2001).The data on unemployment rate are available from the U.S. Department of Labor (U.S.Department of Labor, 2002).All the other control variables, and the dependent variable, graduation rate, are available from the IPEDS database.

Method and Analysis Procedures
Identifying the data structure of a dependent variable and independent variable is the first step in choosing the most appropriate statistical method.In this study, the dependent variable, FTIC graduation rate, is a continuous variable, while the independent variables consist of continuous variables and dichotomous variables.The data to be analyzed have two important characteristics.First, the data related to the dependent variable are longitudinal in nature, and will allow for an analysis of changes in the FTIC graduation rate in the years 1997 through 2001.Second, as discussed in a previous section, the data related to the independent variables have two hierarchical structures: institution-level variables and state-level variables.
Considering the data structure and research questions, recently developed Hierarchical Linear Modeling (HLM) growth model will be applied.HLM growth analysis enables to describe changes over time in longitudinal data (unlike a pre-test and post-test design), and analyzes program impacts within a particular period of time specified by the researcher (unlike time-series analysis).What is more, when the data structure is hierarchical, the HLM growth model can better estimate the contribution of variables at each level (Arnold, 1992;Bryk & Raudenbush, 1992).Thus, under a hierarchical data structure, the HLM growth model is more relevant and useful than other methods such as pre-and post-test design or interrupted time series design.
One strength of the HLM growth model is that HLM considers each different variable at each different level, and each different level is formally represented by its own sub-model-institution-level and state-level in this study.These sub-models express relationships between variables at each level, and specify how variables at one level influence relations occurring at another (Bryk & Raudenbush, 1992).When the change in FTIC graduation rate (growth trajectory) is included in the analysis, this study has three different hierarchical structures.Accordingly, three different types of sub-models will be generated.The first is the institutional growth model, which includes each individual institution's growth trajectory for FTIC graduation rate.The second is the within-state model, which reflects institutional characteristics.The third is the between-states model, which analyzes the effects of state-level policy variables.
The first step in model building is to identify the growth trajectory of each institution between linear and polynomial models.To identify the growth trajectory, a visual inspection of all the institutions' growth trajectories was conducted, and the average growth trajectory was generated and visually inspected as Bryk and Raudenbush (1992) recommend.Based on the identified growth trajectory, a linear within-institution model (Level-1 model) was generated.
Prior to specifying institution-level and state-level models, it is useful to fit the unconditional model which does not include explanatory variables at the institution and state levels.The major purpose of the unconditional model is to collect information about the growth trajectory and point of origin (i.e., the estimated FTIC graduate rate in 1997).Based on the information obtained through unconditional models, the first step is to determine if sufficient variability among institution-and state-level variables exists.For example, if the growth rate within a given state is the same for every institution, it means that the parameter is constant across all the institutions within a state (Level-2).In this case, the parameter is retained at Level-2 and the corresponding random effect term is set to zero, but no Level-2 level predictors are included in the conditional model.
Based on the results of the analysis of the unconditional model, the conditional model, which includes explanatory variables at Level-2 and Level-3, is considered.When the coefficients of the unconditional model are significant at each level, the variables are included in the conditional model.By introducing explanatory variables at each level, the total variation of FTIC graduation rate can be apportioned by levels.At Level-2, institution-level control variables are included, and state-level control variables and program variables of interest are included at Level-3.

Results of the Unconditional Model
Fitting the unconditional model at Level-2 and Level-3 exclusive of any Level-2 or Level-3 predictors provided useful information about the general pattern of growth and the difference in the growth rates between institutions and states.

Fixed Effects
The state-level fixed effect of graduation rate at the beginning of the five-year period indicated that, averaged across all institutions in the 41 states included in the analysis, the estimated graduation rate in 1997 was 39.92 percent (t = 28.23;p<.001).The estimated state-level fixed effect of slope indicated that, averaged across all institutions in the 41 states, the growth in graduation rates was estimated to be 0.62 percentage points per year.This growth trajectory was significantly different from zero (t = 4.47; p<.001).(See Table 1.)

Random effect
The estimated variance components appear in the lower panel of Table 1.At the institution level (Level-2), the large chi-square values (1545.11 and 585.84) indicate that there were large amounts of random variations (p<.001) among institutions for both initial graduation rate and the growth in graduation rate.Therefore, further conditional analyses at the institution level were warranted.
Across the states (Level-3), a significant amount of random variation also existed in the state mean initial graduation rate and state mean growth rates.The chi-square values for the two parameters were 132.72 (p<.001) and 121.10 (p= <.001), respectively.Again, further conditional analysis for each parameter at the state level was also indicated.

Table 1 Fixed Effects and Random Effects (Unconditional Model) Model Specification at Institutional Level and Sate Level
Based on the results of the unconditional model, the next step was to fit the institution-level model (Level-2 model) in order to account for random variation found in the Level-1 growth parameters.To search for Level-2 predictors with sufficient predictive power, all Level-3 models were temporarily left unconditional.Then, the Level-3 predictors were considered when the Level-2 model was specified.

Institution Level Model (Level-2 Model)
To explain the variability between institutions, institutional characteristics were considered at Level-2.Among the three potential Level-2 variables, average tuition and financial student ratio were excluded in the analysis because both variables were highly correlated with other included variables.
For the institution's initial status, the variance component associated with student dormitory ratio (277.36) was relatively large, and also had statistically significant predictive power (t = 6.598; p< .001).Also, the residual variance for student dormitory ratio was significant (chi-square = 63.70,p = 0.008).For the institution's growth rate, dormitory ratio was the only significant predictor (chi-square = 85.40, p < 0.001).
Therefore, student dormitory ratio was included for institution's initial status and for institution's growth rate.The final Level-2 conditional models were specified as:

State Level Model (Level-3 Model)
To explain the differences between state variations, state characteristics were included in the models.The results of a correlation analysis suggest that family ability to pay college costs was the strongest potential predictor.(College preparation and unemployment rate were highly correlated with family ability to pay, and were thus excluded from the analysis because of multicollinearity.)Therefore, the final state-level models were:

The Effects of PBF Programs
The next step was to test the PBF program effects by including the program-related variables in the model.The first research question was to compare the growth in institutional graduation rates between PBF states and non-PBF states.To test for PBF program effects, the final model was expanded by adding a PBF program variable.This program variable was included only in the state average growth rate parameter -not the initial status parameter-because the research question concerned the effects of the PBF program on the growth in graduation rates.Therefore, the final models to test program effects were: In the model, "PBF" represented a contrast between PBF states and non-PBF states.The effect of PBF programs on FTIC graduation rate was tested in two ways.First, the model fit improvement was tested to know whether significantly more state-level variance in graduation rate was explained by the introduction of PBF program variable.This test was conducted using a model fit test with D-statistics.Second, single parameter tests of the contrast of interest -PBF states vs. non-PBF states -were conducted.
The model fit was not improved significantly by adding the PBF program variable (chi-square = 0.188; df = 2; p>.500).This result suggests that the PBF program variable did not help to reduce the unexplained variance at the state level.(See Table 2.) No matter whether the states were PBF or non-PBF states, the inclusion of the PBF program variable did not influence significantly the growth in graduation rates.In addition, the results for the single parameter tests showed that the contrast for PBF states vs. non-PBF states was not statistically significant (for state mean growth rate t = -0.50;p >.500, and for Dorm Rate t = 0.044; p = >.500).(See Table 3.)

Table 2 Fixed Effects of Final Model (Testing PBF Program Effects) Table 3 Random Effects of Final Model (Testing PBF Program Effects) The Effects of Program Types
Research question 2 concerned the effects of PBF types on growth in graduation rates within the PBF states.That is, did states with both PB and PF programs have higher growth in graduation rates than states with only a PB or PF program?Therefore, a new data file with only PBF states (30 states) was obtained by splitting the original data file.Fitting the model followed the same steps described above.
The variance for the Level-2 model indicates that there were significant amounts of random variations (p<.001) among institutions for both initial graduation rate and growth of graduation rate.Therefore, further conditional analysis at the institution level was warranted.Across states, too, a significant amount of random variation existed in the state mean initial status, and in the state mean growth rate.
Based on the unconditional model, institutional characteristic variables were included at Level-2 models.Each predictor variable for an institution's initial status and growth rate were the same as for the previous model specification.Through the model specification procedures, student dormitory ratio was selected for the institution's initial status predictors and for the institution's growth rate predictor.
Based on the results of the Level-2 conditional model, state-level characteristics were included to construct a conditional Level-3 model.The model specification procedures were similar to those of the previous model, which was based on 41 states.Family ability to pay for college proved significant and was included in the final model.In addition, the performance program type variable was included in the final model to address the research question.Therefore, the final models were: By adding a PBF program type variable in the models, the model fit improved significantly (chi-square = 8.12; df = 2; p = 0.017).In addition, single parameter test results showed that the PBF program type was significant overall.The effects of PBF program type on the state mean growth in graduation rates was statistically significant (t = 2.36; p = 0.026).Also, PBF program type had a significant effect through student dormitory ratio: in states with both PB and PF programs, the ratio of dormitory beds has a greater impact on graduation rate than in states with only one program (t = 2.61; p = 0.015) as shown in Table 4.The test results demonstrate that the states with both PB and PF programs performed better than the states with only a PB or PF program.

Summary And Discussion
The purpose of this study was to determine whether states with performance budgeting and funding (PBF) programs had increased the effectiveness of their public institutions of higher education over the five years considered in this study.To explore this question, two primary research questions were generated.An HLM analysis made it possible to test PBF program effectiveness: 1) between PBF states and non-PBF states; 2) between the states with both PB and PF programs and with only PB or PF program.The following is a summary of the research findings: (1) During the five years under study, the growth of graduation rates in PBF states was not greater than in non-PBF states.
(2) The growth in graduation rates in the states with both PB and PF programs was higher than in the states with only one (PB or PF) program.These results could be disappointing for state policy-makers who support performance-based reforms, or might be good news to those who disapprove of state government-initiated reforms in higher education.The following discussion explores the results and considers their implications.
The first finding to consider is that the growth in graduation rate in PBF states was not found to be different from that in non-PBF states.In interpreting these results, it is important to consider: a) the nature of change in graduation rates at institutions of higher education; b) the amount of funding tied to PBF in a given state; and c) the degree to which PBF programs influence college-and departmental-level decision making.
As many higher education administrators have recognized, the change of the graduation rates among colleges and universities generally is very slow.This tendency is confirmed by a review of a decade of changes in graduation rates in Florida (Florida Board of Education, 2001), where the graduation rate at state colleges and universities varied by only four points from 1992 through 1999 (57.2 % in 1992, 58.7% in 1993, 59.5% in 1994, 59.4% in 1995, 59.5% in 1996, 59.5% in 1997, 61.1 % in 1998, 59.6% in 1999, and 59.9% in 2000).
Despite changes in budgeting and other policies made by the state during this decade, the graduation rate of Florida institutions of higher education remained very stable, a fact that could be attributed to the intrinsic nature of change in higher education institutions and one which could have limited the impact of PBF programs on graduation rate.Such an alternative explanation for the slow growth of the graduation rate in PBF states may account for the lack of difference seen between PBF and non-PBF states, which would otherwise suggest that PBF programs are ineffective.
The tendency of graduation rates to change slowly calls into question the use of FTIC graduation rate as a program indicator, particularly for performance funding (PF) programs, which tightly link budget with institutional performance.When the graduation rate is selected as a performance indicator for a PF program, policy-makers should not expect instant results.Thus, the findings of this study would suggest that PF states which use graduation rate as a performance indicator take into account the caveats described above.Another possible explanation on the ineffectiveness of the PBF programs might be the small amount of funding tied to institutional performance.Although some researchers (Hoyt, 2001;Serban, 1998;MGT of America, INC., 1999) argued that the amount of money may not be crucial in PBF programs, the SHEEO survey results show that the proportion of money linked to performance is too small-from 0.5% to 4% across all PBF states except South Carolina, the only state that allocates 100% of its higher education budget based on institutional performance (Christal, 1998).A considerable amount of program ineffectiveness might be explained by the low monetary stakes tied to institutional performance.
In addition, although many states adopted PBF program, these states still allocate most portion of their budget based on traditional criteria.As Wildavsky (1988) argued, even when state governments adopt new budgeting policies, the tradition of incrementalism remains entrenched in many state agencies and state legislatures.As is well known, the main budget allocation criteria in higher education are traditionally the number of full time equivalent (FTE) students.Therefore, one possible scenario in budgeting allocation is that low-performing institutions with high student enrollment might receive more money than high-performing institutions with lower student enrollments, as Bridges (1999) found in his dissertation on the PF program in Colorado.
From the point of program implementation, one critical issue related to ineffectiveness of the PBF programs is the level at which the PBF program has an impact within the institution.Expectations related to a PBF program might be communicated and tied to funding at the institutional level, departmental level, or faculty level.Considering that faculty plays a large part in the performance of higher education institutions, it is important that they be 'on board' in order to improve institutional performance.As Burke and his associates (2000) found, however, performance funding tends to become invisible on campus below the level of the vice-president.In some extreme cases, Poisel (1998) found that presidents of community colleges do not exactly understand the programs at the beginning of the PBF program implementation.
The second finding to consider is the greater growth of the graduation rate in states with both PB and PF programs than in states with only one of these programs.To complement the strengths and accommodate the weaknesses of the two programs, ten states adopted both programs at the same time, thereby combining the flexibility of performance budgeting and the certainty of performance funding.
The states with both programs might transfer strong political intentions to their colleges and universities by adopting the PF program; and they may allow for needed flexibility by adopting the PB program.Regardless of whether the PBF programs were mandated by state legislature, and whether performance indicators are prescribed or not, the flexibility and participation of institutional administrators and faculty in the design and implementation of PBF programs is crucial.Participants in the program implementation process tend to feel more responsible for the outcomes (Coulson-Clark, 1999).As Van Vaught (1994) argued, some of the failures of government-initiated changes in higher education may be attributed to the independent nature of the academic profession.This characteristic of higher education institutions ensures that government-initiated reforms in higher education systems will fail unless they are flexible in their implementation -and take faculty culture into account.

Limitations
This study has two limitations in generalizing its findings.First, although the most common performance indicator, FTIC graduation rate, is used to compare institutional performance, some states do not use FTIC graduation rate as a performance indicator.However, there is no comprehensive study of performance indicators nationwide, except a State Higher Education Executive Officers (SHEEO) survey which found that 32 of 33 states in the U.S. include graduation rate as one of their performance indicators (Christal, 1998).
Second, FTIC graduation rate is only one of the definitions of graduation rate.The report 'Measuring Up 2000' (NCPPHE, 2000) used two different terms in its calculation of completion.One approach was FTIC graduation rate and the other type involved the proportion of total completion to total enrollment.Only one of these terms, FTIC graduation rate was used for this study because most states use FTIC graduation rate as their graduation rate indicator.

Conclusions
In the 1990s, higher education institutions faced the pressure of externally-imposed reforms designed to link budget with institutional performance.Among these reforms were performance budgeting and performance funding programs.As traditionally autonomous institutions, however, colleges and universities were slow in responding to the demand for change posed by such programs.As the results of this study show, the implementation of PBF programs did not have the immediate or dramatic impact on higher education institutions that policy-makers may have expected.
Institutional performance-FTIC graduation rate in this study-did not improve markedly after the adoption of PBF the programs by states.This outcome might be attributed to the tendency of higher education institutions to change slowly, to the use of graduation rate as a measure of institutional performance or effectiveness, or to problems intrinsic to the PBF programs themselves.
The lack of growth in institutional graduation rates, however, does not mean that PBF programs failed to achieve their goals.More time may have been necessary for changes to become apparent, or changes might have appeared indirectly rather than directly.From a strategic point of view, then, legislatures might do well to encourage institutions to engage in actions which will lead to long-term change.Concluding that PBF programs are not useful based simply on changes in graduation rate over a short period, then, is not advised.
Another issue comes to light when evaluating the effectiveness of PBF programs, that is, the common practice of state legislatures of allocating monies for higher education annually.As long as state governments measure institutional performance annually and allocate budgets based on this annual measurement, institutions will direct more money and effort towards short-term rather than long-term fundamental changes.The pressure of annual budgetary decisions is accentuated when states review and sometimes change their performance indicators annually.In such cases, institutions are not sure if today's performance indicators will be next year's indicators.Under these circumstances, it is not surprising that institutions tend to focus more on short-term than long-term efforts to increase their institutional performance.
In consideration of the findings of this study and the evaluation issues discussed above, policy-makers are advised to sustain PBF programs long enough until such programs bear their fruits or prove ineffective.In addition, a distinction between short-term and long-term performance indicators is essential.Long-term indicators should be considered in the budget allocation process after a reasonable time has passed, or weighted in such a way as to guide institutions to focus more on long-term than short-term changes.If policy-makers attend to such details, PBF programs may prove effective in fostering the sort of institutional change that benefits all involved with higher education.

Notes
The program variables are the PBF program-related factors.The control variables include factors other than program-related factors that influence graduation rate.PBF program variables include: Control variables are used to control for other factors that affect FTIC graduation rate.Based on the literature, several variables were chosen to control for exogenous influences on the graduation rate in order to accurately capture the effects of PBF programs.These control variables are divided into institution-level control variables and state-level control variables.