Impact of Performance-funding on Retention and Graduation Rates

As the architect of the oldest and most stable performance-funding program, Tennessee provides a unique opportunity to analyze the impact of changes in performance-funding policies on changes in institutional retention and six-year graduation rates over time. Utilizing spline linear mixed models, this study examines the impact of changes in Tennessee’s performance-funding policies on retention and six-year graduation rates at public four-year institutions from 1995-2009. The results show tying retention and graduation rates to performance-funding was unrelated to changes in the performance measures over the fifteen year period examined. Additionally, the doubling of the monetary incentive associated with the retention and six-year graduation rate measures in 2005 was not associated with increases in retention rates. These results suggest that at their current funding levels, states’ adoption of performance-funding programs, such as the one in Education Policy Analysis Archives Vol. 19 No. 33 2 Tennessee, may be insufficient to incentivize changes in institutional outcomes as desired by state leaders.


Introduction
Public higher education institutions continue to find themselves involved in a national discourse over concerns of accountability, assessment, and performance.In an attempt to realign institutional goals and state priorities, states are tying a portion of state appropriations to institutional performance measures such as retention and six-year graduation rates.Beginning with Tennessee in the late 1970s, there has been significant growth in the number of states adopting performance-funding programs, reaching a high of 25 states in 2003 (McLendon, Hearn, & Deaton, 2006).Additionally, a 2009 Midwestern Higher Education Compact (MHEC) report indicates several more states (Indiana, Louisiana, and Texas) are considering or have recently adopted performance-funding incentives in an attempt to increase the performance of public institutions.
In addition to the growth in the number of states adopting performance-funding programs, states are also increasing the percentage of state appropriations that are tied to performance measures.Historically, the pool of state appropriations tied to performance-funding has remained around 5% (Petrides, McClelland, & Nodine, 2004); however, by 2014 at least three states, Louisiana, Indiana, and Tennessee, will have over 25% of their appropriations to higher education tied to outcomes.More than three decades after performance-funding was initially adopted by Tennessee, the idea of tying state appropriations to institutional performance remains a "hot topic" in state legislatures across the nation (Burke, Modarresi, & Serban, 1999, p. 17).The question remains: how effective is performance-funding as a policy tool?
To better understand the validity of linking state appropriations to institutional performance, researchers need to examine stable programs.Doyle and Noland (2006) suggest that, "the increased national emphasis on accountability and assessment has placed institutions under the crosshairs of legislative oversight.In order to ensure that both internal and external accountability concerns are addressed by this movement to direct legislative oversight, it is critical that scholars examine 'stable' performance-funding programs" (p. 7).Burke and Modarresi (2001) selected the performance-funding programs in Tennessee and Missouri as representative stable programs based on "age, continuity and acceptance by state and campus leaders" (p.53).Tennessee, which pioneered performance-funding in the 1970s, operates the most stable performance-funding program, since it has maintained a continuous program, while other states programs, including Missouri (MHEC, 2009), have aborted their programs or are too infantile to be effectively evaluated (Burke & Serban, 1999).The growth in states adoption of performance-funding policies in an attempt to improve institutional performance begs the question: Are performance-funding policies working?
This study examines the impact of performance-funding on institutional outcomes.Specifically, the study analyzes the impact of the adoption and evolution of Tennessee's performance-funding program on Tennessee public four-year institutional outcomes (retention and six-year graduation rates) over a fifteen year time period (1995)(1996)(1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008)(2009).The stability of the performance-funding program in Tennessee provides a unique opportunity to analyze the impact of performance-funding on institutional outcomes over an extended time period.

Review of the Literature
Historically, the public has had great confidence and trust in public higher education.However, it is clear that public land-grant universities' commitment to the social compact is eroding (Couturier, 2005), and as a result, Lingenfleter (2008) notes, "many in the public lack the confidence that additional investment will generate the results we need" (p.4).For instance, U.S. colleges and universities are failing to graduate a greater percentage of baccalaureate students than in previous decades.While the number of students enrolling in colleges after high school graduation has increased from 45.9% in 1974 to 71.5% in 2004 (Horn, Berger, & Caroll, 2004), during the same period the six-year graduation rate has lingered around 66% (Adelman, 2006).This means that more students are accessing higher education; however, more of them are also leaving without a degree.Public higher education continues to find itself in an "age of disenchantment" (Barnett, 1992, p. 216), and the public is no longer willing to accept peer-review and accreditation as satisfactory forms of accountability.In an attempt to improve institutional outcomes, states have altered their accountability strategies over time from system efficiency, to educational quality, to organizational productivity, and finally, to external responsiveness to public priorities or market demands (Burke, 2005).More recently, states have also tried using performance-funding, performance-budgeting and performance-reporting as accountability methods to link state funding and priorities to institutional performance.
Performance-funding links state funding to prescribed performance measures agreed upon by the state and institution(s).If the institution meets the measures, it receives a predetermined amount of state appropriations (Burke & Minassians, 2003;McLendon, Hearn, & Deaton, 2006).Performance-funding provides states with the most direct form of influence on institutional outcomes.Performance-budgeting allows state officials complete control to determine if any state funding should be appropriated to an institution based upon performance measures (Burke & Minassians, 2003;McLendon, Hearn, & Deaton, 2006).This approach allows more flexibility in the measures selected and has been more widely implemented; by 2003, 35 states utilized the performance-budgeting method.Finally, the majority of states, 42, have employed some version of performance-reporting.There are no appropriations tied to performance-reporting; however, this method relies on institutions to provide information to the state as publicity (McLendon, Hearn, & Deaton, 2006).Serban and Burke (1998) point out that states have implemented performance-funding to "achieve two major goals: increased accountability and improved institutional performance" (p.157).Additionally, Gaither, Nedwek, andNeal (1994), andLayzell (1999) indicate policymakers have a renewed interest in incentivizing institutions to perform well in areas that more closely align with current and future state needs.Alexander (2000) notes that as a result, "the entire nature of the traditional relationship between government and higher education is in the process of significant change in stretching the public dollar to serve more students in attempting to maximize economic returns" (p.413).
However, these attempts at improving institutional outcomes may have proved ineffective because, the percentage of state appropriations attached to the performance measures is around 5% of the total appropriations to institutions (Petrides, McClelland, & Nodine, 2004).For example, in 2008 the University of Tennessee at Knoxville received $11.8 million from performance-funding (Tennessee Higher Education Commission, 2011), while they received double that amount ($23.6 million) in private gifts for the same year (U.S.Department of Education, 2008).Additionally, in 2008 the University of Tennessee at Knoxville's unrestricted state appropriations represented 54% of the institution's operating revenues (U.S.Department of Education, 2008), while revenue from performance-funding only represented 1.5%.
One exception to the typical performance measure allotment was the state of South Carolina, which attempted to link all state funding to performance measures (Alexander, 2000).The state abandoned its model, however, after it became clear that the model posed major funding deficits for the State's institutions.Three other states (Arkansas, Colorado, and Minnesota) have also abandoned their performance-funding models due to sustainability issues (Burke, Modarresi, & Serban, 1999).
The effectiveness of performance-funding programs continues to receive mixed reviews.On the one hand, research indicates there are three main factors contributing to the challenges facing implementation of performance-funding: performance indicator selection (e.g., course or degree completion, retention, and six-year graduation rates), determination of success (e.g., benchmark) criteria, and the limited amount of funding allocated for the policy initiative (Burke & Serban, 1997).Layzell (1999) comments, "the third factor reflects the need to provide a meaningful incentive for institutions to take performance-funding seriously.If the dollar value is too low, neither institutions nor policymakers will likely find performance-funding worth the effort" (p.244).In other words, the 5% allotment may not provide enough of an incentive to comply with the policy, especially if there are other (e.g., unrestricted and restricted gifts, grants) revenue pipelines.On the other hand, Burke and Modarresi (2001) found performance-funding, in a stable program like Tennessee's, has "improved performance, demonstrated accountability and increased state funding" (p.445).
There also appears to be evidence in Tennessee that the longevity of its performancefunding program and continued support from both state and institutional leaders affirms its success.For example, the periodic policy reviews by stakeholders engenders ownership in the continuance of the program.Additionally, the performance-funding program, as an accountability policy, continues to persist while other states have aborted their programs (Bogue & Johnson, 2010).Furthermore, some experts see performance-funding producing remarkable results despite the small percentage of a state budget allocated to this kind of program (MHEC, 2009).
The empirical literature on the impact of performance-funding remains scarce.Most of the literature focuses on policy adoption and abandonment.McLendon, Hearn, and Denton (2006) examined factors that influenced states to adopt performance-funding policies.The authors found that performance-funding and budgeting policies were adopted primarily due to "legislative party strength and higher education governance arrangements" (p.11).More specifically, the authors note that states with a greater proportion of Republicans in the state legislature, and those with a more decentralized governance structure were more likely to adopt new performance-funding policies.Dougherty and Natow (2009) examined the factors that led to performance-funding demise.Specifically, the authors examined the termination of performance-funding programs in Florida, Illinois, and Washington.Apart from state-specific conditions, the researchers found that: drastic reductions in state funding, policy originating from a budget proviso rather than unique legislation, lack of strong support from the business community, and a loss of supporters for the policy, are common determinants that led to states' discontinuance of performance-funding.
A more recent study by Gorbunov (2010) examined states' adoption, re-adoption, and abandonment of performance-funding policies across 47 states for the time period 1979-2009.Gorbunov found that the annual change in public enrollment was significant in predicting policy adoption.Additionally, the higher the proportion of Republicans in the state legislature, the more likely a state was to adopt a performance policy.Coordinating boards with and without budgeting authority were also found to have a positive relationship with policy adoption.Policies initiated through an appropriation bill or budget proviso were less likely to be readopted than policies mandated by legislative act.
In one of the few studies examining the impact of performance-funding on performance measures, Doyle and Noland (2006) examined the relationship between performance-funding and retention at Tennessee public institutions.The authors chose to examine Tennessee, because of the historical stability of the program.Specifically, the authors' research focused on changes in retention rates within institutions and systems over time.The authors found very few changes in retention rates over the period studied associated with performance-funding.They did find, however, that at a few institutions performance-funding was associated with increased retention rates.Their research suggests mixed results based on the institution; in other words, performance-funding seemed to have had a substantive impact on a few institutions, but not the majority.One of the limitations of the study is that the authors only examined the within institution variation of retention rates.Performance-funding in Tennessee, however, was based on changes in institutional retention and graduation rates compared to an institution's peers (between group variation) as defined in Tennessee's funding formula (Tennessee Higher Education Commission [THEC], 2000a;THEC, 2005).Shin and Milton (2004) examined whether institutions in states with performance budgeting or funding programs experienced improved graduation rates.Utilizing hierarchical linear modeling growth analysis, they found that the institutional graduation rates of states with performancefunding/budgeting did not outperform states without performance-funding programs.The authors caution that the lack of growth in graduation rates may not mean that the performance-funding policy was ineffective, rather that the five year time period (1997)(1998)(1999)(2000)(2001) analyzed may have been insufficient for the implementation of these policies to impact graduation rates, a typically slow changing measure.Due to this limited time examined in the study, Shin and Milton (2004) go on to propose "policy makers are advised to sustain pbf [performance-based funding] programs long enough until such programs bear their fruits or prove ineffective" (p.18).
In summation, much of the literature on performance-funding models has focused on stability and successful practices (Wall, Frost, Smith, & Keeling, 2008;Banta, Rudloph, Dyke, & Fisher, 1996;Burke, Modarresi, & Serban, 1999;Burke & Modarresi, 2001;Bogue & Johnson, 2010) or policy adoption and abandonment (McLendon, Hearn, & Deaton, 2006;Dougherty & Natow, 2009;Gorbunov, 2010).Additionally, two studies examined the relationship between performancefunding and student outcomes.Doyle and Noland's (2006) institutional level study found that performance-funding was related to increased student retention rates at a few institutions.While, Shin and Milton's (2004) study showed that institutions in states with performancefunding/budgeting programs did not outperform institutions in states without performance-based programs, over a five-year period.
A gap in the literature exists in examining the impact these funding models have on student performance at public institutions.A review of the existing research demonstrates that the literature is divided on the success of states' adoption of performance-funding programs and on the measures these programs are created to incentivize institutions to improve.Regardless of the effectiveness of performance-funding, Tennessee continues to implement iterations of its original program and other states plan to implement programs in the coming years (MHEC, 2009) or increase the percentage of state appropriations tied to performance.
The existing research has been limited to examining within institution variation (Doyle & Noland, 2006) and to examining shorter windows of time, which may have been insufficient for measuring the impact of performance-funding policies on institutional outcomes (Shin & Milton, 2004).This study seeks to fill these gaps by examining the relationship between the adoption of performance-funding policies and institutional retention and six-year graduation rates of Tennessee public institutions compared to their peers (between group variation) over a fifteen-year time period.By examining between group variation, the study's analyses reflect how institutions were funded based on performance-funding in Tennessee.Additionally, analyzing changes in retention and graduation rates over a fifteen year period provides policymakers with a longitudinal perspective of how these programs are related to changes in institutional outcomes.

The Evolution of Performance-funding in Tennessee
Tennessee has long been a leader in the higher education accountability movement, starting its performance-funding program in 1979.Since the program's inception, the goal has been to incentivize institutions to improved performance, particularly by enhancing student-learning outcomes (Banta et al., 1996).Initially, in 1979Initially, in -1982 public two-and four-year institutions had the opportunity to garner a two percent budget supplement above and beyond the incremental or formula budgeting model as part of the pilot performance-funding program.The budget supplement was tied to performance in the following activities: (1) gaining accreditation for qualified programs; (2) utilizing standardized examinations to test graduating students in general education and specific majors; (3) surveying institutional stakeholders (e.g.currently enrolled students, recent graduates, community members and employers) regarding satisfaction with academic programs; (4) conducting peer reviews of academic programs; and (5) making campus improvements and academic program revisions based on assessment results (Banta et al., 1996).
While Tennessee is often cited as the most stable performance-funding program, it has undergone eight revisions since the initial pilot cycle in 1979 -in 1980, 1982, 1987, 1992, 1997, 2000, 2005, and 2010.Not only have the criteria (initially internally focused, now allowing external peer comparison) evolved over time, but also the budget supplement has increased from 2 to 5.45%.Despite these revisions, Tennessee's program has experienced mild variability compared to many states, such as South Carolina.
In Tennessee's performance-funding model, retention and six-year graduation rates fell under standard three -Student Persistence.The joint six-year graduation and retention rate measure was added to the performance-funding model as a factor in the fourth cycle (1997)(1998)(1999)(2000).Initially, institutions were compared to a rolling three-year average of their retention and six-year graduation rates.The combined rates, as one measure in the performance-funding criteria, were worth 5% of the 5% of unrestricted state appropriations tied to performance-funding. Beginning in 2000, however, institutions were evaluated on their retention and graduation rate performance relative to their funding formula peer institutions (THEC, 2000a).Tennessee's public institutions' funding formula peers were self-selected by each institution, with the approval of THEC (Russ Deaton, Personal Communication, August 2, 2011).In 2005, Cycle 6, the six-year graduation and retention measures were divided into separate measures, and the State of Tennessee doubled the monetary value of these measures to a combined worth of 10% of the 5.45% allocation.
Beginning in 2010, Cycle 7, Tennessee has revamped its entire funding model moving from an enrollment driven model to an outcomes-based funding model.The new outcomes-based funding model funds institutions based on their performance on weighted measures (10 for 4-year institutions, 11 for community colleges).The measures account for mission differentiation, so that the University of Tennessee at Knoxville's top measure (6-year graduation rates) is different than that of Middle Tennessee State University (number of Bachelors and Associates Degrees).The Funding Formula Review Committee, with extensive input from the institutions and systems, selected and aligned the goals and priorities for each institution type (as classified by Carnegie) with the Complete College Tennessee Act.
For the first time, 75-80% of state institutions' entire unrestricted appropriations will be allocated based upon outcomes, such as student retention and six-year graduation rates, rather than enrollment.The remaining funds are allocated according to the performance-funding program (5.45%) and fixed costs.As a result of this new funding formula, retention and six-year graduation rates were removed from the performance-funding program and added to the new outcome-based funding formula for the appropriate institutions.The performance-funding program has switched its focus primarily to quality assurance measures, such as alumni feedback surveys (THEC, 2010).

Theoretical Framework
Organizational theory is helpful in understanding why institutions receiving performancefunding would outperform their peers in measures tied to funding.Resource dependency theory (Pfeffer & Salancik, 1978) explains that organizations are inescapably bound to the conditions within their environment.Resources can become more or less scarce, and the organization's survival is contingent on its ability to adapt in response to these environmental changes (Pfeffer & Salanick, 2003).
For public institutions, performance-funding represents an opportunity to obtain additional funding outside of and at times in addition to the two primary modes of funding: incremental and formula (Layzell, 1999).In the performance-funding model, institutions agree with the states on the measures tied to their funding.If the performance measures are met, institutions receive the funding.As public higher education becomes more market-oriented and as competition over scarce resources intensifies, resource dependency theory suggests that institutions would strive to meet the measures to ensure their survival and competitiveness.
The study also adopts Swail's (2003) Geometric Model of Student Persistence.Swail's model suggests that there are three primary factors that contribute to student persistence: Cognitive, Social, and Institutional.For Swail, the institutional level factors consist of "the practices, strategies, and culture of a college or university that, in either an intended way or unintended way, impact student persistence" (p.77).Swail's model recognizes the role and responsibility of the institution in helping students to persist and succeed, both academically and socially.In the Geometric model, institutional factors make up the base of the model because, for Swail, the institution forms the basis for student success.The institution can influence student persistence by providing academic, social, and financial support.
The Geometric model accurately reflects the state's belief, as evidenced through their accountability and performance-funding initiatives, that the institution plays a vital role in the success of its student population.Increasingly, states are attempting to incentivize institutions to shift their practices, strategies, and culture levers through performance-funding.Institutions that improve retention and six-year graduation rates have the opportunity to secure scarce resources in an increasingly competitive environment.

Purpose of Study and Research Questions
This study attempts to examine the impact of the adoption of performance-funding policies on retention and six-year graduation rates at public four-year institutions.Specifically, this study is concerned with the impact that the adoption of state performance-funding in Tennessee has had on the state's public four-year institutions' retention and six-year graduation rates compared to their peer institutions.Tennessee's stability provides the researchers with a unique opportunity to examine the impact of the state's adoption of multiple performance-funding policies on retention and sixyear graduation rates over time.This study addresses the following research questions: 1) What is the impact of Tennessee's adoption of retention rates as a performance measure in the state's performance-funding program in 1997 on institutional retention rates at Tennessee public four-year institutions compared to their peers?2) What is the impact of Tennessee's adoption of six-year graduation rates as a performance measure in the state's performance-funding program in 1997 on institutional six-year graduation rates at Tennessee public four-year institutions compared to their peers?
3) What is the impact of Tennessee's doubling the monetary value associated with its retention rate measure in the performance-funding program in 2005 on institutional retention rates at Tennessee public four-year institutions compared to their peers?

Research Design and Methodology
The stability of the performance-funding program in Tennessee provides a unique opportunity to analyze the impact of these policies on changes in institutional retention and six-year graduation rates over time.Operating the oldest and most stable performance-funding program, Tennessee serves as a good quasi-experiment in which the outcomes of Tennessee public four-year institutions can be compared with their out-of-state peer institutions.The peer institutions used in the study were the same peer institutions identified for each institution for comparison in the performance-funding formula and are presented in Table A-1 (THEC, 2000b).As was previously discussed, an institution's funding formula peers were self-selected by the institution with the approval from THEC.
Of the peer institutions included in the analyses, South Carolina was the only state that had institutions engaged in performance-funding during the study's time frame.The researchers chose to keep the three peer institutions from South Carolina in the analysis because they represented only three institutions out of the sixty-seven in the peer group, and graphical and descriptive analyses suggested that they did not behave as outliers, which could bias the peer group.Rather, they behaved very similarly to the rest of the peer institutions.Additionally, evidence suggests that the performance-funding experiment in South Carolina had no significant effect on the outcomes of institutions (Schmidt, 2001).
Quasi-experimental designs allow for effects related to the research question of interest to be analyzed despite the lack of random assignment (Pedhazur & Schmelken, 1991).To discern the response of Tennessee institutions to the performance-funding policies, the analysis utilizes data from several sources, which are presented in Table A-2, from 1995-2009.The fifteen years included in the analysis allow the researcher to examine the association of the initial adoption of six-year graduation and retention rates as a joint measure into the performance-funding formula in 1997 and to assess the impact of the state doubling the financial incentive for institutions to increase their sixyear graduation and retention rate measures in 2005.
To answer the proposed questions, the study utilizes spline linear-mixed models (LMM) to examine the impact of performance-funding policies on institutional six-year graduation rates at Tennessee public four-year institutions and their peer institutions.LMMs are the most appropriate statistical technique for several reasons.First, using LMMs allows the researchers to ask questions about and model within-group change and between-group change simultaneously (Raudenbush & Bryk, 2002).Second, the LMM accounts for the nested nature of the data (dependency among the repeated measures) through the variance-covariance matrix in the model (Fitzmaurice, Laird, & Ware, 2004;Singer & Willett, 2003).Third, LMMs incorporate a random error term that accounts for measurement unreliability.Fourth, LMMs account for between group differences by incorporating fixed and random effects, which allows the researcher to fit the most parsimonious model.Finally, LMMs can also accommodate both dynamic and static predictors.
Spline models are appropriate for modeling non-linear trends.Conceptually, splines divide the time series axis into segments with differing slopes, while still considering a model for the trend over time that is composed of "piecewise linear trends, having different slopes within each segment but joined or tied together at fixed times" (Fitzmaurice, Laird, & Ware, 2004, p. 8).The locations where the segments meet are referred to as knots.Utilizing splines allows the mean response to vary as time proceeds based on the individual regression slopes for each segment.The simplest linear spline model has the following form in Linear Mixed Model (LMM) notation: (1) Where * t is the knot, and In the current study, the use of spline models allows the researchers to compare the mean change in retention and six-year graduation rates before the knot, which denotes the performancefunding policy change event, with the mean change after the knot.To test for this response, the following model was specified and is presented in LLM notation: (2)

X
Where t represents the time metric (year), performance-funding group represents whether an institution was in Tennessee (engaged in performance-funding) or a peer institution, and X represents a matrix of the control variables.In terms of performance-funding group comparison, the null hypothesis of no group differences in retention or six-year graduation rates after the performing funding policy was implemented is: Organizational theory informs us that organizations adapt to change incrementally and would not be able to adjust to these changes in their strategy instantaneously (Perdu & Levis, 1998).To capture the effect of the policy changes and allow institutions incremental time to adjust, three separate analyses are conducted.Model A corresponds to research question 1, Model B is the fitted model addressing research question 2, and Model C corresponds to research question 3.Each analysis allowed four years of observation both before and after the policies were introduced.As a result, the spline knot for each analysis is modeled two years after the policy was introduced, with Model C being the exception.
The first analysis examines the initial policy adoption, which added retention rates to the performance-funding model in 1997 (Model A).The analysis compares the retention rates at Tennessee public four-year institutions and their peers from 1995-2003.By choosing the period of 1995-2003, the study models two years before the policy was adopted (1995 -1996) and two years of lag time for institutions to respond to the policy (1997 -1998).The knot for Model A is placed in 1999, allowing us to compare the slope of retention rates before institutions were able to influence their outcomes (1995)(1996)(1997)(1998) with the years following the policy introduction (2000)(2001)(2002)(2003).
The second analysis (Model B) examines the impact of the adoption of six-year graduation rates as a performance-funding measure in 1997 by comparing Tennessee public four-year institutions and their peers from 1998-2006.The time span allows for the comparison, before and after the policy adoption, of four graduation cohorts.The knot for Model B was placed in 2002.
Finally, in order to examine the impact of the State's decision to double the money tied to retention and six-year graduation rates, the study conducts a third analysis that models the change in retention rates at Tennessee public four-year institutions and their peers from 2001-2009 (Model C).Because 2009 is the latest year of available data, the spline knot for Model C was placed at 2005, the same year the policy was introduced to allow for four years of observation after the policy was implemented.The impact of the State's increased financial incentive on six-year graduation rates could not be examined, because the policy adoption occurred in 2005 and the first six-year cohort will not graduate until 2011.
In addition to the random effects, which account for institutional differences, the study also includes additional controls to prevent the results from being driven by other institutional factors related to retention and six-year graduation rates (Fitzmaurice, Laird, & Ware, 2004).First, the student body characteristics of an institution are likely to impact six-year graduation and retention rates.For example, research has shown that there is a positive relationship between a student's socio-economic status and degree completion (Adelman, 1999(Adelman, , 2006;;Cabrera, Burkum, & Nasa, 2005;Titus, 2006aTitus, , 2006b)).Furthermore, students' academic preparedness has been shown to be related to degree completion (Adelman, 2004;DesJardins, Ahlburg, & McCall, 2002). 1dditionally, there is a gap in the graduation rates of minority students compared to white students (Adelman, 2004;Horn, 2006;Titus, 2006aTitus, , 2006b;;DesJardins, et. al., 2002).Adelman (2006) and Kim (2007) found that part-time attendance is negatively related to degree completion.To account for these factors, the analyses control for the log of total enrollment, the percent of the student body that received Pell grants, the percent of the student population that is part-time, and the percent of the population that is minority.The financial makeup of an institution also impacts six-year graduation and retention rates.For instance, the percentage of revenue from tuition is positively related to degree completion (Kim, 2007;Titus, 2006a).Additionally, Titus (2006a) found that the level of institutional expenditures on educational expenditures is positively associated with degree completion.For this reason, the study also accounts for the percentage of revenue from tuition, and the percent of expenditures that an institution spends directly on education.Finally, the study also controls for Carnegie classification in order to account for mission differentiation.The dependent variables included in the analyses are institutional retention (Model A and Model C) and six-year graduation (Model B) rates.

Results
Initially, graphical and descriptive analyses were performed to explore patterns in the data.Table A-3 presents the descriptive statistics for the variables included in the analyses.The analyses and results are presented and discussed by research question.

(Model A) Research Question 1: What is the impact of Tennessee's adoption of retention rates as a performance measure in the state's performance-funding program in 1997 on institutional retention rates at Tennessee public fouryear institutions compared to their peers?
In 1995 Tennessee institutions had a mean retention rate of 0.69 (SD = 0.05), by 2003 their mean retention rate was 0.68 (SD = 0.07).The average retention rate had declined slightly over the period.A paired samples t-test was used to test for a statistically significant difference between the retention mean at the beginning of the timeframe compared to the mean in 2003.The results of the test, t(8) = 0.257, p = 0.80, suggests that there is no difference in the change of retention rates for Tennessee institutions over the time period.Similar results were found for Tennessee peer institutions, t(43) = 1.13, p = 0.26, in 1995 (M =0.75, SD =0.07) andin 2003 (M =0.73, SD = 0.12).Finally, the two groups' mean averages were compared at both time points to see if there was a statistically significant mean difference in retention scores at the beginning and end points of the analysis.The independent samples t-test revealed an intercept effect, t(68) = 2.42, p ≤ 0.018, for mean retention rate by performance-funding group.Specifically, Tennessee public institutions' mean retention rate was lower than their peers in the initial year of the study.However, there was no statistically significant difference in the mean retention rate in 2003, t(59) = 0.290, p =0.773).
These results suggest that there may be no effect of performance-funding on retention rates over the period studied.To better understand the impact of the adoption of the performancefunding policy in 1997 on retention rates, a spline linear mixed model was utilized to test for an interaction effect between performance-funding group and the mean retention rate change before and after the policy adoption.Table 1 presents the results of the fitted model.The model shows that performance-funding did not account for statistically significant variation in retention rates in the beginning year of the study, t(537) = -1.51,p = 0.13, nor was it associated with change over time, t(537) = 1.82, p = 0.06.Institutional revenue was the lone covariate that accounted for significant variation in intercepts, t(537) = 3.72, p < 0.05, with higher revenue associated with higher intercepts.Lower Carnegie classifications, t(537) = -2.04,p < 0.05, the percent of part-time students, t(537) = -2.43,p < 0.05, and the percent of Pell students, t(537) = -3.23,p < 0.05, are all negatively associated with changes in retention rate over time.Performance-funding group membership was not statistically related with differences in intercepts or retention rate change over time.
Table 2 presents the results of the test of the hypothesis of interest, where there are no group differences in change in retention rates after the performance-funding policy was implemented: 0 6 8 : 0 H ! != = , under Model A. The test of the hypothesis reveals that the two parameters of interest are not statistically different from zero, F(2, 527) = 0.53, p = 0.22.This evidence is further corroborated by the model comparisons presented in Table 3 under Model A. The goodness of fit statistics reveal that the reduced model fits the data better than the full model ( ! " 2 (2) =1.04 , p = 0.22).The results show that the implementation of performance-funding was not associated with statistically different changes in retention rates at Tennessee institutions compared to their peers.(Model B) Research Question 2: What is the impact of Tennessee's adoption of six-year graduation rates as a performance measure in the state's performance-funding program in 1997 on institutional six-year rates at Tennessee public four-year institutions compared to their peers?Four-year institutions in Tennessee had a mean six-year graduation rate of 0.38 (SD = 0.08), compared to their peers, whose average six-year graduation rate was 0.45 (SD = 0.14).By 2007, Tennessee four-year institutions' mean six-year graduation rate had increased to 0.41 (SD = .08),and their peers had increased to 0.51 (SD = 0.15).A paired samples t-test was used to test for a statistically significant difference between the six-year graduation rate in 1999 and 2007.The results of the test, t(8) = .785,p = .45,suggest that there is no difference in the change of graduation rates for Tennessee institutions over the time period.Tennessee's peer institutions' mean difference was statistically significant, t(44) = 2.37, p < .05,suggesting that Tennessee's peer institutions' mean in 2007 was significantly higher than in 1999.Finally, the groups were compared for mean differences in 1999 and 2007.The results of the independent samples t-test revealed there is no intercept effect, t(73) = 1.53, p = 0.129; however, there was a statistically significant difference in the mean six-year graduation rates in 2007, t(74) = 2.023, p < .05).
Unlike the 1997 retention model, the descriptive analysis suggests that there may be an effect of performance-funding groups on six-year graduation rates over the period studied.A spline linear mixed model was utilized to test for an interaction effect between performance-funding group and the mean six-year graduation rate change before and after the policy adoption.The results of the fitted model are presented in Table 4.The model shows that there is no significant variation in sixyear graduation rates over time, t(536) = -0.80,p = 0.42; however there is an intercept effect, t(536) = 3.13, p < 0.05.Lower Carnegie classification, t(536) = -4.20,p < 0.05, and higher percentages of part-time students t(536) = -3.36,p < 0.05, account for significant variation in intercepts.No covariates included in the model accounted for unique variance in slopes.As was the case in the 1997 retention model, performance-funding group membership did not account for unique variance in six-year graduation rates' intercepts or slopes.
Table 5 presents the results of the test of the hypothesis of interest of no group differences in change in retention rates after the new performance-funding policy was implemented: The test of the hypothesis reveals that the two parameters of interest are not statistically different from zero, F(2, 605) = 1.06, p = 0.34.This evidence is further corroborated by the model comparisons presented in Table 3 under Model C. The goodness of fit statistics reveal that the reduced model fits the data better than the full model ( ! " 2 (2) = 2.19, p = 0.33).Once again, the results show that the implementation of performance-funding was not associated with statistically different changes in retention rates at Tennessee institutions compared to their peers.The results of the test of the hypothesis of interest of no group differences in change in sixyear graduation rates after the policy was implemented: 0 6 8 : 0 H ! != = are presented in Table under Model B. The test of the hypothesis reveals that the two parameters of interest are not statistically different from zero, F(2, 536) = 2.03, p = 0.13.The researchers also confirmed these results by conducting model comparisons using Log likelihood.The results of the model comparison are presented in Table 3 under Model B. The goodness of fit statistics reveal that the reduced model fits the data better than the full model ( ! " 2 (2) = 4.09 , p = 0.13).Consistent with the 1997 retention rate model (Model A), the results show that the implementation of performancefunding was not associated with statistically different changes in six-year graduation rates at Tennessee institutions compared to their peers.
(Model C) Research Question 3: What is the impact of Tennessee doubling the monetary value associated with its retention rate measure in the performance-funding program in 2005 on institutional retention rates at Tennessee public four-year institutions compared to their peers?Four-year institutions in Tennessee had a mean retention rate of 0.69 (SD = 0.04) in 2001, compared to their peers whose average retention rate was 0.78 (SD = 0.08).By 2009, Tennessee four-year institutions' mean retention rate had increased to 0.72 (SD = .05),and their peers had failed to increase (mean = 0.78; SD = 0.08).A paired samples t-test was used to test for a statistically significant difference between the mean retention rates for both groups in 2001 and 2009.The results of the tests suggest that there are no statistically significant differences in the change of retention rates for Tennessee institutions, t(8) = 1.092, p = .30,or their peers, t(37) = 2.97, p < .05,over the time period.Additionally, the groups were compared for mean differences in 2001 and 2009.The results of the independent samples t-test showed that there is an intercept effect, t(73) = 2.97, p < 0.05, as well as a statistically significant difference in the retention rates in 2009, t(74) = 2.14, p < .05.
In order to better model the linear retention rate change over time by performance-funding group membership, a spline linear mixed model was utilized.The utilization of a spline linear mixed model allows the researcher to test for an interaction effect between performance-funding group and the mean retention rate change before and after the policy adoption.Table 5 presents the results of the fitted model.The results show that there is no significant variation in intercepts, t(605) = 1.16, p = 0.24, or slopes, t(605) = -1.60,p = 0.10, over time.The larger the percent of Pell students, t(605) = -4.21,p < 0.05, and the higher the percentage of part-time students, t(605) = -3.04,p < 0.05, are associated with lower retention rate intercepts.The higher the percentage of minority students, t(605) = -3.04,p < 0.05, and being a member of the peer group in the performance-funding group variable, t(605) = -2.44,p < 0.05 is associated with higher retention rate intercepts.No covariates included in the model accounted for unique variance in slopes.While performance-funding group membership was significant in explaining intercept differences, group membership did not explain any unique variation in retention rate slopes.

Policy Implications
This study examined: 1) the impact of Tennessee's adoption of retention and six-year graduation rates into the performance-funding program in 1997, and 2) the impact of the State doubling the monetary value of these two measures in the performance-funding model in 2005.Treating the adoption of these performance-funding policies in Tennessee as a quasi-experiment, the study compared the impact of the policy adoptions on Tennessee institutions' rate of change in retention and six-year graduation rates before the policy was adopted with the rate of change for the years following the policy adoption compared to their peer institutions.The results of the analyses can be summarized as follows: The addition of retention rates into the performance-funding model in 1997 was not associated with a significant difference in the rate of change in Tennessee institutions' retention rates compared to their peers.The results were similar for the addition of six-year graduation rates into the performance-funding model by the State in 1997.The graduation rate of Tennessee institutions did not change at a statistically different rate compared to the pre-policy adoption period when the institutions were compared to their peers.Finally, the attempt by the State to further incentivize improvement in institutional outcomes by doubling the money tied to the retention and graduation rate measures in 2005 was unrelated to the institutions' change in retention rates.
Despite these monetary incentives, this study's findings suggest that the adoption of new performance-funding policies has little impact on altering institutional outcomes at their current funding levels.Our findings support previous hypotheses that institutions do not have enough financial incentive to emphasize improving student outcomes (Petrides, McClelland, & Nodine, 2004;Burke & Serban, 1999).
Additionally, public four-year institutions are major organizations, where change occurs incrementally and is often costly.Increasing 6-year graduation and retention rate performance takes time and considerable resources.Simply put, the small percentage of funds tied to performancefunding (usually around 5%) may not provide enough financial incentive to influence changes in institutional outcomes.These incentives may have been further weakened as state appropriations continue to decline as a percentage of institutional budgets over the last three decades (Kane, Orszag, & Gunter, 2003;Cheslock & Gianneschi, 2008), leading public institutional leaders to comment that their institutions, which were once state-supported, then state-assisted, are now only state-located (de Vise, 2011).These comments highlight one of the challenges states face in trying to influence institutional outcomes while supplying a smaller proportion of institutional revenues.
Furthermore, during the sixth cycle of performance-funding in Tennessee (2005-2010), institutions, as a whole, averaged receiving over 91% of the potential performance-funding revenue they were eligible to receive (THEC, 2011).The results of the current study found that on at least two of the measures, however, institutions in Tennessee showed no significant difference compared to their peer institutions in the measures' changes over time.The rate at which Tennessee institutions are achieving their potential performance-funding suggests that institutions can achieve their performance goals with relative ease.Our findings suggest that institutions are able to meet the performance requirement without improving their outcomes.At the current funding level, the consequences of failing to improve institutional outcomes are low.In order to incentivize improvement in institutional outcomes, policymakers are left with two options: 1) consider raising the stakes around performance-funding, or 2) pursue new strategies to achieve their goals.
Regarding the first option, beginning in FY 2012, Tennessee is increasing the proportion of state appropriations tied to institutional performance outcomes.Recognizing that higher education is key for economic development and the future success of the state, Tennessee recently passed major legislation (the Complete College Tennessee Act of 2010), which focuses on more closely tying institutional performance to state priorities.A key piece of the new legislation was the creation of a new outcomes-based funding model.No longer will performance measures be tied to just 5.45% of state appropriations.Under the new model as much as 80% of an institution's appropriations from the state will be tied to these measures.
While the current study has shown that tying five percent of appropriations to performance measures has not been significant in influencing changes in institutional outcomes, the verdict remains unsettled on whether Tennessee's new policy of tying a higher percentage of appropriations to these measures will sufficiently incentivize the desired change in institutional outcomes.Future research should examine the impact of Tennessee's new funding policy on institutional performance.Additionally, the evolution of the proportion of Tennessee's state appropriations tied to performance measures raises the question: What proportion of appropriations tied to performance measures is enough to incentivize changes in institutional outcomes?
With regard to the second option, while performance-funding policies may be politically desirable, our results suggest that policymakers may want to consider utilizing alternative policies to achieve improved institutional outcomes.Research on the use of performance-based accountability policies and their effectiveness in government has a rich history in the performance management literature, which may be informative for the ongoing policy dialogue.Specifically, scholars have found little evidence that other performance-based budgeting programs have produced the desired results (Andrews & Hill, 2003;Moynihan & Andrews, 2011).Given that scholars have found similar results across multiple fields, the effectiveness of these policies to generate the desired results is questionable.While the research findings speak to the tangible benefits of these policies, it is important not to devalue the importance these policies may hold both politically and culturally.
Finally, we believe it would be a mistake to conclude from our findings that performancefunding has not been an effective policy.It may be more appropriate to reframe discussions of its effectiveness around implementation and quality assurance rather than performance.The state's long history of performance-funding, which spans over three decades, established a new framework for accountability for higher education that focused on institutional performance and outcomes.Performance-funding has changed the conversation and culture of expectations of both the public and higher education in Tennessee by tying some state appropriations to outcomes.Given that the study's results showed that institutional outcomes did not improve over the period studied compared to their peer institutions, the high success rate of Tennessee institutions in receiving their performance-funding dollars may raise concerns over how effective the measures are that are being used to evaluate institutional performance.Regardless, the state's long history of tying state appropriations to outcomes may have made tying 80% of state appropriations to outcomes more politically palatable and culturally feasible.In short, participation in performance-funding created a climate that was conducive to expanding the reforms embedded in the Complete College Tennessee Act.

Conclusion
In an attempt to increase accountability and incentivize institutional performance, numerous states have followed Tennessee's example by establishing performance-funding programs.Additionally, at least three states (Indiana, Louisiana, and Texas) have recently adopted or are considering adopting performance-funding programs (MHEC, 2009).While the study's findings are obviously particular to Tennessee, the study's results have implications for state policy makers considering adopting new performance-budgeting programs, higher education governance boards that are advising legislative leaders on the merits of these programs, and the public.
This paper provides evidence that public institutions in Tennessee have not responded to the current monetary incentives created by the State's adoption of performance-funding policies.The introduction of retention and six-year graduation rates as a measure included in performancefunding in 1997 did not result in a statistically significant difference in the mean retention or six-year graduation rates at Tennessee institutions compared to their peers.Additionally, the doubling of the monetary incentive associated with the retention and six-year graduation rate measures by the State in 2005 was not associated with increases in retention rates at Tennessee institutions compared to their peer institutions.
These results suggest that States' adoption of performance-funding programs, such as the one in Tennessee, may not incentivize the change in institutional outcomes as desired by state leaders at their current funding levels.Previously, Shin and Milton (2004) recommended that policy makers should sustain performance-based funding programs until sufficient time has passed and they either result in increased performance or are shown to be ineffective.Based on the study's results, policy makers may want to: 1) consider increasing the financial incentives tied to these policies in order to elicit their desired change in institutional outcomes, or 2) consider other methods to improve institutional outcomes.