The Effects of Performance Budgeting and Funding Programs on
Graduation Rate in Public Four-Year Colleges and Universities
Jung-cheol Shin
Sande Milton
Florida State University
Citation: Shin, J., Milton, S.,
(2004, May 26). The effects of performance budgeting and funding
programs on graduation rate in public four-year colleges and universities.
Education Policy
Analysis Archives, 12(22). Retrieved [Date] from
http://epaa.asu.edu/epaa/v12n22/.
Abstract
This study was conducted to determine whether states with
performance budgeting and funding (PBF) programs had improved
institutional performance of higher education over the five years
(1997 through 2001) considered in this study. First Time in
College (FTIC) graduation rate was used as the measure of
institutional performance. In this study, the unit of analysis is
institution level and the study population is all public
four-or-more-year institutions in the United States. To test PBF
program effectiveness, Hierarchical Linear Modeling (HLM) growth
analysis was applied. According to the HLM analysis, the growth
of graduation rates in states with PBF programs was not
greater than in states without PBF programs. The lack of
growth in institutional graduation rates, however, does not mean
that PBF programs failed to achieve their goals. Policy-makers
are advised to sustain PBF programs long enough until such
programs bear their fruits or are proven ineffective.
|
Introduction
The purpose of this study was to examine the effects of
performance budgeting and funding (PBF) programs on the
performance of public higher education institutions in the United
States using longitudinal performance data. As of 2001,
thirty-six states in the U.S. utilized either performance
budgeting or performance funding programs, or both, in
institutions of higher education (Burke & Minassians, 2001).
This trend began during the 1990s, as state policy-makers sought
to enhance accountability in higher education, find more effective
budget allocation methods, and in some cases, trim budgets.
Although performance budgeting (PB) and performance funding (PF)
are different in focus, both create a link, to different degrees,
between budget allocation and institutional performance. In
addition, both types of programs are based on the underlying
premise that higher education institutions are motivated to
improve their performance when performance is linked to budget
allocation.
The trend toward increased use of performance budgeting and
funding programs is not limited to the U.S. The United Kingdom
led other European countries in initiating a market-driven higher
education accountability program, first introduced by the Thatcher
government. Influenced by trends in the U.S and Europe, the
Canadian provinces of Alberta and Ontario have also adopted
performance-based funding mechanisms to assess university
efficiency and quality (Barnetson 1999). In Australia, 40 public
research universities and two private institutions are subject to
an accountability framework that links their funding to
performance requirements (Atkinson-Grosjean & Grosjean,
2000). In spite of vigorous protests from students and faculty,
New Zealand established a new accountability system in 1998. In
the early 1990s, South Korea adopted a performance-based funding
program consisting of nine independent sub-programs, each with a
particular purpose, to which funding was tied (Kim, 2001).
Current Status Of Pbf Programs
Major Stakeholders: State Politics and Business
Involvement
The major stakeholders in higher education reform of the 1990s
have been state legislatures, state chancellors and state
agencies, governors, and higher education institutions (Blackwell
& Ciston, 1998). Among these stakeholders, governors and
state legislatures play the primary role in the adoption of PBF
programs (Burke & Minassians, 2001). In contrast, a
relatively small number of initiatives (in 24 states) have been
conducted within state agencies or university systems
themselves.
A more complete understanding of the widespread adoption of PBF
programs since the 1990s must encompass more than the vested
interests of state legislatures and governors. In the 1990s,
higher education reform has been guided by economic
values—competition, productivity, and efficiency—more
than ever before. This shift in priorities might be explained in
part by the growing interest of business leaders in state higher
education policy-making (Usdan, 2002; Zernike, 2002; Zumeta,
2000).
Business interests have changed the picture of higher
education. As Levin (2001) argues, higher education is becoming
like a “globally competitive business” (p.237). In
some cases, business leaders are actively involved in PBF program
development. For example, in South Carolina, which links 100% of
higher education funding to institutional performance, two of the
12 members of the Higher Education Joint Legislative Study
Committee represented the business community. [Not one of the 12
members of the committee came from higher education (China,
1998)].
However, other major stakeholders in higher
education—institutional administrators and
faculty—have always been more concerned about institutional
improvement than accountability, unlike politicians who emphasize
external accountability through the implementation of PBF programs
(Burke, 1998). Because of these perceptual and value differences,
tensions almost invariably exist between both groups around the
issue of PBF programs.
Type of Program: Performance Budgeting or Funding?
Performance budgeting and funding are similar in that both
programs link institutional performance with budget allocation,
but the methods differ in the way each ties institutional
performance to state funding. While performance funding (PF)
programs link budget to institutional performance in a direct,
automatic, and formulaic manner, the link in performance budgeting
(PB) programs is loose, indirect, and uncertain (Burke et al,
2000).
Because of the loose link between performance and budget
allocation, PB programs may be less effective in enhancing
institutional performance. Notwithstanding this limitation, state
policy-makers typically prefer performance budgeting to
performance funding because of the flexibility in implementation
(Burke, et al., 2000).
Although more states adopted PB programs than PF programs,
recent trends indicate, “both programs are borrowing
elements from the other approach to gain its benefits while
evading their own problems” (Burke et al, 2000, p.3).
Further, in order to minimize the limitations and maximize the
strengths of each program, an increasing number of states have
adopted both programs at the same time (Burke et al, 2000).
Commonly-Used Indicators for measuring institutional
performance
Ruppert (1995) conducted an in-depth case study on performance
indicators (PIs) and performance reporting in ten states that have
been among the leaders in designing and using PIs in higher
education. (Note 1) He
found that most states share a common core of performance
indicators. Graduation outcome data, for instance, is one measure
that responds to policy concerns —rising college costs and
the economic return to the state of college-educated citizens.
A survey by the State Higher Education Executive Officers
(SHEEO) conducted between 1996 and 1997 identifies some indicators
used frequently in the U.S. at that time: graduation rate (32
states), transfer rate (25 states), faculty workload/productivity
(24 states), follow-up satisfaction (23 states), and externally
sponsored research funds (23 states) (Christal, 1998). Another
study (Burke, 1998) of 10 states conducted in 1997 found less
commonality among PIs in different PBF programs. Among the states
surveyed, the following performance indicators were in use:
graduation and retention rates (10 states), two-to-four-year
transfers (6 states), faculty workload (5 states), institutional
choice (5 states), graduation credits and time to degree (4
states), licensure test score (4 states), transfer graduation
rates (4 states), workforce training and development (4 states),
and external research fund ratio (3 states). Overall, the common
performance indicators across these studies were graduation rate,
retention rate, transfer rate, faculty workload, and sponsored
research funds.
Previous Evaluations of the Impacts of Pbf Programs
Most evaluative studies of the impacts of PBF programs focus on
a specific state or set of institutions within a state. As many
studies assert, the effectiveness of PBF programs is a critical
concern of policy-makers. Nevertheless, few academic evaluation
studies of PBF program impacts have been conducted, save the
occasional evaluation project or dissertation. State-level
evaluations for budget allocation purposes mainly use quantitative
data on specific performance indicators. Florida and South
Carolina, for example, annually evaluate each institution’s
performance in order to guide budget allocation.
In a book on PBF programs, Bogue (2002) explores the impact and
effectiveness of the PBF program in Tennessee. From the actual
1994-1998 performance data, he found that the universities have
consistently scored above the national norm on the College BASE.
One interesting outcome of the Tennessee PBF program is the clear
increase in the proportion of academic program accreditations,
which improved from 65 percent to 100 percent (Bogue, 2002). He
found, however, that institutions generally failed to
significantly raise persistence-to-graduation rates and overall
job-placement rates over the same period.
Florida’s Office of Program Policy Analysis and Government
Accountability (OPPAGA) (2001) evaluated the performance budgeting
program of the State University System in Florida as part of an
evaluation of performance budgeting programs at each state
agency. In the evaluation, OPPAGA reported that graduation and
retention rates have increased since the state adopted a
performance budgeting program, and externally-funded financial
support for the research program has significantly increased as
well. Additional confirmation of these findings exists in
Florida’s relatively high score on completion indicators
versus other evaluation indicators in the ‘Measuring Up
2000’ study. OPPAGA claims this result occurred because the
Florida Legislature decided to make raising the completion or
graduation rates a priority in its indicator system.
The South Carolina Commission on Higher Education (CHE) (2002)
has been studying the impacts of performance funding program in
South Carolina as a three-year project for the Fund for
Improvement of Postsecondary Education (FIPSE) grant. In this
evaluation, the South Carolina CHE used diverse methods (journal
analysis, surveys, interviews, and historical data) to review its
PF program. According to preliminary findings, institutional
policies have changed to accommodate the PF program and some
outcomes are changing as well. Interestingly, the impacts of the
PF program seem to vary depending on which evaluation method is
used in the evaluation process. For example, respondents to the
survey answered that the PF program has had no impact on
institutional quality or efficiency. Yet historical data on
institutional performance shows that performance on SAT scores,
minority enrollment, and externally funded research have all
increased, and that graduation rates are slowly rising as well.
These results suggest that the actual performance of South
Carolina higher education institutions has been increasing since
South Carolina adopted the PF program.
In a review of performance data from Missouri higher education
institutions, Marcus et al. (2000) found that student assessment
scores, licensure exam pass rates, first time in college (FTIC)
graduation rates, minority graduation rates, and student job
placement rates have all increased since Missouri adopted a
performance funding program. While the causal relationship
between PF and increased performance was not conclusive, the PF
program was determined to be “clearly responsible for the
identification of priorities for funding, for the establishment of
assessment measures, and for helping institutions to accept that
part of their state allocation is linked to results”
(p.215).
The studies mentioned above indicate that PBF programs have had
a positive impact on institutional performance. However, these
results are based on limited time periods, and the interpretation
of the performance data can differ from one researcher to
another. For example, Dallet, et al. (2002) analyzed the same
performance data used in an earlier study by OPPAGA (2001) to
discuss the impact of Florida’s PBF program. While OPPAGA
reported that the PBF program had had a positive impact on
Florida’s higher education institutions, the conclusion of
Dallet and his colleagues was quite different. They argued that,
although the graduation rate and retention rate in Florida’s
higher education institutions increased somewhat after the
inception of the PBF program in 1994, this growth actually
occurred at a level similar to that of the 1990-91 academic year,
when no such policy was introduced. In Colorado, which adopted a
PF program from 1994/95 to 1996/97, the effect of performance
level on budget reward was found in a regression analysis to be
lower than expected (Bridges, 1999). Instead, there was a higher
correlation between institution size and performance funding than
between performance funding and performance level.
Research Questions
The overarching purpose of this study is to explore the impacts
that PBF programs have had on institutional performance. Thus,
there are two basic research questions addressed in this
study:
- Has institutional performance in states with PBF programs (PBF
states) improved more than in states without PBF programs (non-PBF
states)?
- Has institutional performance in states with both programs (PB
and PF) improved more than in states with only a PB or PF
program?
Research Design And Method
Unit of Analysis and Population
The unit of analysis for this study is higher education
institution, rather than the state, because the target of PBF
programs is institution-level change, and state-level performance
is simply the aggregation of each state’s institution-level
performance. The study will be limited to public higher education
institutions because private institutions are not the main objects
of state policy. Further, although some states have the same PBF
programs for public four-year institutions and community colleges,
many states (e.g., Florida, California, New York) have different
PBF programs for public four-year institutions and community
college systems (Burke, 1998). In addition, as Hoyt (2001)
addresses, the missions of community colleges are different from
those of four-year colleges. Therefore, for this study the object
of interest is limited to public four-year institutions, and
excludes public community colleges. Also, because PBF programs
focus mainly on undergraduate education, graduate-only
institutions will be excluded from the analysis.
Thus, the target population of this study is all public
four-year higher education institutions in the United States.
Based on the criteria above, 456 institutions were selected for
the study population.
(Note 2) The information on the annual PBF program status of
each state is available from surveys conducted each year from 1997
through 2001 (Burke & Minassians, 2001; Burke et al., 2000;
Burke & Modarresi, 1999; Burke & Serban; 1998; Burke &
Serban, 1997).
Dependent Variable
The dependent variable used in this study is the First Time in
College (FTIC) graduation rate of each institution during the
years 1997 through 2001. This dependent variable was selected
based on three criteria: 1) it is the most commonly-used
performance indicator nationwide; 2) nationwide data are readily
available; (Note 3)
and 3) there is internal validity in using graduation rate as a
measure of performance in higher education. The term is also in
accordance with the Student Right-To-Know Act (SRTK), which
mandated institutions to report the graduation rate of only
“first-time” and “full-time” students who
spent “up to 150 percent of normal time to complete their
degrees” (9PL 101-542; Federal Register, 1992).
In conducting a broad policy studies such as this, one must
consider the question of whether a given policy is accomplishing
its goals (Affholter, 1994). In PBF programs, the attainment of
program goals is measured using performance indicators that are
chosen by state legislatures, coordinating boards, or in some
cases, higher education institutions themselves. According to an
analysis of performance indicators nationwide (Christal, 1998),
graduation rate is the most commonly-used indicator in U.S. higher
education institutions with PBF programs. Of the 33 states with
PBF programs in 1997, 32 states used graduation rate as one of
their performance indicators.
In addition, when an inappropriate or weak indicator is chosen
as a dependent variable, it can invalidate the results of a policy
study. Each institution admits applicants who are deemed
qualified to study at the institution. These students study
approximately four years to satisfy graduation requirements that
are set forth by the department, college, university, or state.
If many students do not graduate in a timely manner, these
students have either not yet satisfied the graduation
requirements, have chosen to leave, or have transferred to another
institution. Any of these cases poses a problem for the
institution, which, due to the very nature of its mission, seeks
to maintain a high graduation rate. As a result, low graduation
rates may imply that the institutions are falling short in
performing some of their functions. Therefore, graduation rate is
a persuasive and valid indicator to represent how well the
institution performs in at least one of its many goals. (Note 4)
FTIC graduation rate is thus calculated using the completions
within 150% of normal time to degree divided by the total number
in a particular year’s cohort. The Integrated Postsecondary
Education Data System (IPEDS) provides total cohort information,
completion within 150% of normal time to degree, and the final
FTIC graduation rate. Therefore, for the purpose of this study,
the final graduation rate provided by the IPEDS will be used
without any adjustment.
Independent Variables
Independent variables in the statistical model will be of two
types: program (or treatment) variables, and control variables.
The program variables are the PBF program- related factors. The
control variables include factors other than program-related
factors that influence graduation rate. PBF program variables
include:
- PBF program adoption
- PBF statesare states that have adopted PBF programs in
at least three of the five years 1997-2001.
- Non-PBF statesare states that either did not adopt a PBF
program or had adopted one for only one year during 1997-2001. (Note 5)
- PBF type
- PB & PF states are states that have concurrent
performance funding and performance budgeting programs.
- PF or PF statesare states that only have one type of
program: PB or PF.
Control variables are used to control for other factors that
affect FTIC graduation rate. Based on the literature, several
variables were chosen to control for exogenous influences on the
graduation rate in order to accurately capture the effects of PBF
programs. These control variables are divided into
institution-level control variables and state-level control
variables.
Institution-level control variables are variables that have been
shown to affect graduation rate at the institution level. The
average tuition has been associated with graduation rates (Heller,
1997; Hsing & Chang, 1996; Dayhoff, 1991). The availability
of financial aid (grants, loans, work study, etc.) has a positive
impact on enrollment (John et al., 2001; Braunstein et al, 1999;
Sheridan et al., 1994; Cabrera et al., 1992). The opportunity to
live in a campus residence hall, especially during the freshman
year, has also been shown to influence retention rates and time to
completion (Astin, 1997; Sheridan et al., 1994).
- Average Tuition is defined as [(instate tuition + out of
state tuition)/2].
- Student Dormitory Ratio is defined as (space availability
in dormitory facilities)/(full-time students).
- Financial aid student ratio is the percentage of
full-time, first-time degree and certification seeking students
who receive any financial aid (grants, loans, assistantships,
fellowships, tuition wavier, tuition discounts, veterans benefits,
employer aids and other monies).
State-level control variables are state level characteristics
which have been shown to influence graduation rate, and which must
be included in the causal model in order to capture accurately the
effects of PBF programs. Astin (1997), Schmitz (1993), and Kahn
and Nauta (2001) found that college preparation factors (mean
entrance test score on ACT or SAT, and freshmen’s high
school grades) are positively associated with graduation rate.
Also, unemployment rate has been shown to be negatively associated
with graduation rate (Heller, 1999; Hsing & Chang, 1996).
Statewide family ability to pay college costs is the variable that
reflects financial support from the family to the students
(Braunstein et al., 1999; John, 1993) as well as the economic
conditions of each state.
- College preparation is defined as “Overall
score” of college preparation (Note 6) as described in
“Measuring Up 2000” (NCPPHE, 2001).
- Family ability to pay college costs is defined as the
student’s family’s ability to pay at public four-year
colleges, as described in “Measuring Up 2000” (NCPPHE,
2001).
- Unemployment rate is defined as the number of unemployed
as a percent of the labor force (U.S. Department of Labor,
2002).
Data Sources
Study data are available directly or indirectly from diverse
sources. Data on PBF program variables have been collected in
annual surveys conducted by Burke and associates in the years 1997
through 2001 (Burke & Minassians, 2001; Burke et al., 2000;
Burke & Modarresi, 1999; Burke & Serban, 1998; Burke &
Serban, 1997). Nationwide data on state-level control variables
were included in the nationwide performance evaluation study,
“Measuring Up 2000” (NCPPHE, 2001). The data on
unemployment rate are available from the U.S. Department of Labor
(U.S. Department of Labor, 2002). All the other control
variables, and the dependent variable, graduation rate, are
available from the IPEDS database.
Method and Analysis Procedures
Identifying the data structure of a dependent variable and
independent variable is the first step in choosing the most
appropriate statistical method. In this study, the dependent
variable, FTIC graduation rate, is a continuous variable, while
the independent variables consist of continuous variables and
dichotomous variables. The data to be analyzed have two important
characteristics. First, the data related to the dependent
variable are longitudinal in nature, and will allow for an
analysis of changes in the FTIC graduation rate in the years 1997
through 2001. Second, as discussed in a previous section, the
data related to the independent variables have two hierarchical
structures: institution-level variables and state-level
variables.
Considering the data structure and research questions, recently
developed Hierarchical Linear Modeling (HLM) growth model will be
applied. HLM growth analysis enables to describe changes over
time in longitudinal data (unlike a pre-test and post-test
design), and analyzes program impacts within a particular period
of time specified by the researcher (unlike time-series
analysis). What is more, when the data structure is hierarchical,
the HLM growth model can better estimate the contribution of
variables at each level (Arnold, 1992; Bryk & Raudenbush,
1992). Thus, under a hierarchical data structure, the HLM growth
model is more relevant and useful than other methods such as
pre-and post-test design or interrupted time series design.
One strength of the HLM growth model is that HLM considers each
different variable at each different level, and each different
level is formally represented by its own
sub-model—institution-level and state-level in this study.
These sub-models express relationships between variables at each
level, and specify how variables at one level influence relations
occurring at another (Bryk & Raudenbush, 1992). When the
change in FTIC graduation rate (growth trajectory) is included in
the analysis, this study has three different hierarchical
structures. Accordingly, three different types of sub-models will
be generated. The first is the institutional growth model, which
includes each individual institution’s growth trajectory for
FTIC graduation rate. The second is the within-state model, which
reflects institutional characteristics. The third is the
between-states model, which analyzes the effects of state-level
policy variables.
The first step in model building is to identify the growth
trajectory of each institution between linear and polynomial
models. To identify the growth trajectory, a visual inspection of
all the institutions’ growth trajectories was conducted, and
the average growth trajectory was generated and visually inspected
as Bryk and Raudenbush (1992) recommend. Based on the identified
growth trajectory, a linear within-institution model (Level-1
model) was generated.
Prior to specifying institution-level and state-level models, it
is useful to fit the unconditional model which does not include
explanatory variables at the institution and state levels. The
major purpose of the unconditional model is to collect information
about the growth trajectory and point of origin (i.e., the
estimated FTIC graduate rate in 1997). Based on the information
obtained through unconditional models, the first step is to
determine if sufficient variability among institution- and
state-level variables exists. For example, if the growth rate
within a given state is the same for every institution, it means
that the parameter is constant across all the institutions within
a state (Level-2). In this case, the parameter is retained at
Level-2 and the corresponding random effect term is set to zero,
but no Level-2 level predictors are included in the conditional
model.
Based on the results of the analysis of the unconditional model,
the conditional model, which includes explanatory variables at
Level-2 and Level-3, is considered. When the coefficients of the
unconditional model are significant at each level, the variables
are included in the conditional model. By introducing explanatory
variables at each level, the total variation of FTIC graduation
rate can be apportioned by levels. At Level-2, institution-level
control variables are included, and state-level control variables
and program variables of interest are included at Level-3.
Results
Results of the Unconditional Model
Fitting the unconditional model at Level-2 and Level-3 exclusive
of any Level-2 or Level-3 predictors provided useful information
about the general pattern of growth and the difference in the
growth rates between institutions and states.
Fixed Effects
The state-level fixed effect of graduation rate at the beginning
of the five-year period indicated that, averaged across all
institutions in the 41 states included in the analysis, the
estimated graduation rate in 1997 was 39.92 percent (t = 28.23;
p<. 001). The estimated state-level fixed effect of
slope indicated that, averaged across all institutions in the 41
states, the growth in graduation rates was estimated to be 0.62
percentage points per year. This growth trajectory was
significantly different from zero (t = 4.47; p<.
001). (See Table 1.)
Random effect
The estimated variance components appear in the lower panel of
Table 1. At the institution level (Level-2), the large chi-square
values (1545.11 and 585.84) indicate that there were
large amounts of random variations (p<. 001) among
institutions for both initial graduation rate and the growth in
graduation rate. Therefore, further conditional analyses at the
institution level were warranted. Across the states (Level-3), a
significant amount of random variation also existed in the state
mean initial graduation rate and state mean growth rates. The
chi-square values for the two parameters were 132.72
(p<. 001) and 121.10 (p= <.001),
respectively. Again, further conditional analysis for each
parameter at the state level was also indicated.
Table 1
Fixed Effects and Random Effects
(Unconditional Model)
Model Specification at Institutional Level and Sate Level
Based on the results of the unconditional model, the next step
was to fit the institution-level model (Level-2 model) in order to
account for random variation found in the Level-1 growth
parameters. To search
for Level-2 predictors with sufficient predictive power, all
Level-3 models were temporarily left unconditional. Then, the
Level-3 predictors were considered when the Level-2 model was
specified.
Institution Level Model (Level-2 Model)
To explain the variability between institutions, institutional
characteristics were considered at Level-2. Among the three
potential Level-2 variables, average tuition and financial student
ratio were excluded in the analysis because both variables were
highly correlated with other included variables.
For the institution’s initial status,
the variance component associated with student dormitory ratio
(277.36) was relatively large, and also had statistically
significant predictive power (t = 6.598; p< .001).
Also, the residual variance for student dormitory ratio was
significant (chi-square = 63.70, p = 0.008). For the
institution’s growth rate, dormitory
ratio was the only significant predictor (chi-square = 85.40,
p < 0.001).
Therefore, student dormitory ratio was included for
institution’s initial status and for institution’s
growth rate. The final Level-2 conditional models were specified
as:
State Level Model (Level-3 Model)
To explain the differences between state variations, state
characteristics were included in the models. The results of a
correlation analysis suggest that family ability to pay college
costs was the strongest potential predictor. (College preparation
and unemployment rate were highly correlated with family ability
to pay, and were thus excluded from the analysis because of
multicollinearity.) Therefore, the final state-level models
were:
The Effects of PBF Programs
The next step was to test the PBF program effects by including
the program-related variables in the model. The first research
question was to compare the growth in institutional graduation
rates between PBF states and non-PBF states. To test for PBF
program effects, the final model was expanded by adding a PBF
program variable. This program variable was included only in the
state average growth rate parameter – not the initial status
parameter—because the research question concerned the
effects of the PBF program on the growth in graduation rates.
Therefore, the final models to test program effects were:
In the model, “PBF” represented a contrast between
PBF states and non-PBF states. The effect of PBF programs on FTIC
graduation rate was tested in two ways. First, the model fit
improvement was tested to know whether significantly more
state-level variance in graduation rate was explained by the
introduction of PBF program variable. This test was conducted
using a model fit test with D-statistics. Second, single
parameter tests of the contrast of interest — PBF states vs.
non-PBF states — were conducted.
The model fit was not improved significantly by adding the PBF
program variable (chi-square = 0.188; df = 2;
p>.500). This result suggests that the PBF program
variable did not help to reduce the unexplained variance at the
state level. (See Table 2.) No matter whether the states were PBF
or non-PBF states, the inclusion of the PBF program variable did
not influence significantly the growth in graduation rates. In
addition, the results for the single parameter tests showed that
the contrast for PBF states vs. non-PBF states was not
statistically significant (for state mean growth rate t =
-0.50; p >.500, and for Dorm Rate t = 0.044;
p = >.500). (See Table 3.)
Table 2
Fixed Effects of Final Model
(Testing PBF Program Effects)
Table 3
Random Effects of Final Model
(Testing PBF Program Effects)
The Effects of Program Types
Research question 2 concerned the effects of PBF types on growth
in graduation rates within the PBF states. That is, did states
with both PB and PF programs have higher growth in graduation
rates than states with only a PB or PF program? Therefore, a new
data file with only PBF states (30 states) was obtained by
splitting the original data file. Fitting the model followed the
same steps described above.
The variance for the Level-2 model indicates that there were
significant amounts of random variations (p<. 001) among
institutions for both initial graduation rate and growth of
graduation rate. Therefore, further conditional analysis at the
institution level was warranted. Across states, too, a
significant amount of random variation existed in the state mean
initial status, and in the state mean growth rate.
Based on the unconditional model, institutional characteristic
variables were included at Level-2 models. Each predictor
variable for an institution’s initial status and growth rate
were the same as for the previous model specification. Through
the model specification procedures, student dormitory ratio was
selected for the institution’s initial status predictors and
for the institution’s growth rate predictor.
Based on the results of the Level-2 conditional model,
state-level characteristics were included to construct a
conditional Level-3 model. The model specification procedures
were similar to those of the previous model, which was based on 41
states. Family ability to pay for college proved significant and
was included in the final model. In addition, the performance
program type variable was included in the final model to address
the research question. Therefore, the final models were:
By adding a PBF program type variable in the models, the model
fit improved significantly (chi-square = 8.12; df = 2;
p = 0.017). In addition, single parameter test results
showed that the PBF program type was significant overall. The
effects of PBF program type on the state mean growth in graduation
rates was statistically significant (t = 2.36; p =
0.026). Also, PBF program type had a significant effect through
student dormitory ratio: in states with both PB and PF programs,
the ratio of dormitory beds has a greater impact on graduation
rate than in states with only one program (t = 2.61;
p = 0.015) as shown in Table 4. The test results
demonstrate that the states with both PB and PF programs performed
better than the states with only a PB or PF program.
Table 4
Fixed Effects of Final Model
(Testing PBF Program Types)
Summary And Discussion
The purpose of this study was to determine whether states with
performance budgeting and funding (PBF) programs had increased the
effectiveness of their public institutions of higher education
over the five years considered in this study. To explore this
question, two primary research questions were generated. An HLM
analysis made it possible to test PBF program effectiveness: 1)
between PBF states and non-PBF states; 2) between the states with
both PB and PF programs and with only PB or PF program. The
following is a summary of the research findings:
(1) During the five years under study, the growth of graduation
rates in PBF states was not greater than in non-PBF states.
(2) The growth in graduation rates in the states with both PB
and PF programs was higher than in the states with only one (PB or
PF) program.
These results could be disappointing for state policy-makers who
support performance-based reforms, or might be good news to those
who disapprove of state government-initiated reforms in higher
education. The following discussion explores the results and
considers their implications.
The first finding to consider is that the growth in graduation
rate in PBF states was not found to be different from that in
non-PBF states. In interpreting these results, it is important to
consider: a) the nature of change in graduation rates at
institutions of higher education; b) the amount of funding tied to
PBF in a given state; and c) the degree to which PBF programs
influence college- and departmental-level decision making.
- As many higher education administrators have recognized, the
change of the graduation rates among colleges and universities
generally is very slow. This tendency is confirmed by a review of
a decade of changes in graduation rates in Florida (Florida Board
of Education, 2001), where the graduation rate at state colleges
and universities varied by only four points from 1992 through 1999
(57.2 % in 1992, 58.7% in 1993, 59.5% in 1994, 59.4% in 1995,
59.5% in 1996, 59.5% in 1997, 61.1 % in 1998, 59.6% in 1999, and
59.9% in 2000).
Despite changes in budgeting and other policies made by the
state during this decade, the graduation rate of Florida
institutions of higher education remained very stable, a fact that
could be attributed to the intrinsic nature of change in higher
education institutions and one which could have limited the impact
of PBF programs on graduation rate. Such an alternative
explanation for the slow growth of the graduation rate in PBF
states may account for the lack of difference seen between PBF and
non-PBF states, which would otherwise suggest that PBF programs
are ineffective.
The tendency of graduation rates to change slowly calls into
question the use of FTIC graduation rate as a program indicator,
particularly for performance funding (PF) programs, which tightly
link budget with institutional performance. When the graduation
rate is selected as a performance indicator for a PF program,
policy-makers should not expect instant results. Thus, the
findings of this study would suggest that PF states which use
graduation rate as a performance indicator take into account the
caveats described above.
Another possible explanation on the ineffectiveness of the PBF
programs might be the small amount of funding tied to
institutional performance. Although some researchers (Hoyt, 2001;
Serban, 1998; MGT of America, INC., 1999) argued that the amount
of money may not be crucial in PBF programs, the SHEEO survey
results show that the proportion of money linked to performance is
too small—from 0.5% to 4% across all PBF states except South
Carolina, the only state that allocates 100% of its higher
education budget based on institutional performance (Christal,
1998). A considerable amount of program ineffectiveness might be
explained by the low monetary stakes tied to institutional
performance.
In addition, although many states adopted PBF program, these
states still allocate most portion of their budget based on
traditional criteria. As Wildavsky (1988) argued, even when state
governments adopt new budgeting policies, the tradition of
incrementalism remains entrenched in many state agencies and state
legislatures. As is well known, the main budget allocation
criteria in higher education are traditionally the number of full
time equivalent (FTE) students. Therefore, one possible scenario
in budgeting allocation is that low-performing institutions with
high student enrollment might receive more money than
high-performing institutions with lower student enrollments, as
Bridges (1999) found in his dissertation on the PF program in
Colorado.
From the point of program implementation, one critical issue
related to ineffectiveness of the PBF programs is the level at
which the PBF program has an impact within the institution.
Expectations related to a PBF program might be communicated and
tied to funding at the institutional level, departmental level, or
faculty level. Considering that faculty plays a large part in the
performance of higher education institutions, it is important that
they be ‘on board’ in order to improve institutional
performance. As Burke and his associates (2000) found, however,
performance funding tends to become invisible on campus below the
level of the vice- president. In some extreme cases, Poisel
(1998) found that presidents of community colleges do not exactly
understand the programs at the beginning of the PBF program
implementation.
The second finding to consider is the greater growth of the
graduation rate in states with both PB and PF programs than in
states with only one of these programs. To complement the
strengths and accommodate the weaknesses of the two programs, ten
states adopted both programs at the same time, thereby combining
the flexibility of performance budgeting and the certainty of
performance funding.
The states with both programs might transfer strong political
intentions to their colleges and universities by adopting the PF
program; and they may allow for needed flexibility by adopting the
PB program. Regardless of whether the PBF programs were mandated
by state legislature, and whether performance indicators are
prescribed or not, the flexibility and participation of
institutional administrators and faculty in the design and
implementation of PBF programs is crucial. Participants in the
program implementation process tend to feel more responsible for
the outcomes (Coulson-Clark, 1999). As Van Vaught (1994) argued,
some of the failures of government-initiated changes in higher
education may be attributed to the independent nature of the
academic profession. This characteristic of higher education
institutions ensures that government-initiated reforms in higher
education systems will fail unless they are flexible in their
implementation – and take faculty culture into account.
Limitations
This study has two limitations in generalizing its findings.
First, although the most common performance indicator, FTIC
graduation rate, is used to compare institutional performance,
some states do not use FTIC graduation rate as a performance
indicator. However, there is no comprehensive study of
performance indicators nationwide, except a State Higher Education
Executive Officers (SHEEO) survey which found that 32 of 33 states
in the U.S. include graduation rate as one of their performance
indicators (Christal, 1998).
Second, FTIC graduation rate is only one of the definitions of
graduation rate. The report ‘Measuring Up 2000’
(NCPPHE, 2000) used two different terms in its calculation of
completion. One approach was FTIC graduation rate and the other
type involved the proportion of total completion to total
enrollment. Only one of these terms, FTIC graduation rate was
used for this study because most states use FTIC graduation rate
as their graduation rate indicator.
Conclusions
In the 1990s, higher education institutions faced the pressure
of externally-imposed reforms designed to link budget with
institutional performance. Among these reforms were performance
budgeting and performance funding programs. As traditionally
autonomous institutions, however, colleges and universities were
slow in responding to the demand for change posed by such
programs. As the results of this study show, the implementation
of PBF programs did not have the immediate or dramatic impact on
higher education institutions that policy-makers may have
expected. Institutional performance—FTIC graduation rate in
this study—did not improve markedly after the adoption of
PBF the programs by states. This outcome might be attributed to
the tendency of higher education institutions to change slowly, to
the use of graduation rate as a measure of institutional
performance or effectiveness, or to problems intrinsic to the PBF
programs themselves.
The lack of growth in institutional graduation rates, however,
does not mean that PBF programs failed to achieve their goals.
More time may have been necessary for changes to become apparent,
or changes might have appeared indirectly rather than directly.
From a strategic point of view, then, legislatures might do well
to encourage institutions to engage in actions which will lead to
long-term change. Concluding that PBF programs are not useful
based simply on changes in graduation rate over a short period,
then, is not advised.
Another issue comes to light when evaluating the effectiveness
of PBF programs, that is, the common practice of state
legislatures of allocating monies for higher education annually.
As long as state governments measure institutional performance
annually and allocate budgets based on this annual measurement,
institutions will direct more money and effort towards short-term
rather than long-term fundamental changes. The pressure of annual
budgetary decisions is accentuated when states review and
sometimes change their performance indicators annually. In such
cases, institutions are not sure if today’s performance
indicators will be next year’s indicators. Under these
circumstances, it is not surprising that institutions tend to
focus more on short-term than long-term efforts to increase their
institutional performance.
In consideration of the findings of this study and the
evaluation issues discussed above, policy-makers are advised to
sustain PBF programs long enough until such programs bear their
fruits or prove ineffective. In addition, a distinction between
short-term and long-term performance indicators is essential.
Long-term indicators should be considered in the budget allocation
process after a reasonable time has passed, or weighted in such a
way as to guide institutions to focus more on long-term than
short-term changes. If policy-makers attend to such details, PBF
programs may prove effective in fostering the sort of
institutional change that benefits all involved with higher
education.
Notes
1. This study was
done during 1993 to 1994, and the ten states included were
Colorado, Florida, Illinois, Kentucky, New York, South Carolina,
Tennessee, Texas, Virginia, and Wisconsin.
2. Branch campuses
will generally be excluded from the analysis, except in cases
where adequate information is available.
3. In 2001, the
Integrated Postsecondary Education Data System (IPEDS) began to
provide nationwide graduation rate data, which had been collected
since 1991 using a student tracking system.
4. In some cases, a
low graduation rate reflects the initial intentions of students
who ultimately wish to transfer to another institution, a scenario
most frequently seen at community colleges (Whigham, 2000).
5. We excluded
states with two years PBF experience. This excluded nine states
from the analysis.
6. The
“Overall score” of preparation is calculated weighted
based on students’ high school completion, K- 12 course
taking, and K-12 student achievement.
References
Affhoilter, D. P. (1994). Outcome monitoring. In
J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds),Handbook
of practical program evaluation, San Francisco:Jossey-Bass
Inc.
Arnold, C. L. (1992). An introduction to
hierarchical linear models. Measurement and Evaluation in
Counseling and Development, 25, 58-89.
Astin, A. W. (1997). How good is your
institution’s retention rate? Research in Higher
Education, 38(6), 647-658.
Atkinson-Grosjean, J., & Grosjean, G. (2000).
The use of performance models in higher education: a comparative
international review. Education Policy Archives,
8(30). Retrieved from
http://epaa.asu.edu/epaa/v8n30.htm
Barnetson, B. (1999). Alberta’s
performance-based funding mechanism. Paper presented at the
Coalition for the Advancement of Applied Education Meeting. (ERIC
Document Reproduction Service No. ED 430 428)
Blackwell, E. A., & Ciston (1998). Power and
influence in higher education: The case of Florida. Higher
Education Policy, 12, 111-122.
Bogue, E. G. (2002). Twenty years of performance
funding in Tennessee: A case study of policy intent and
effectiveness. In Joseph C. Burke and Associates, Funding
public colleges and universities: popularity, problems, and
prospects. Albany, NY: The Rockefeller Institute Press.
Braunstein, A., McGrath, M., & Pescatrice, D.
(1999). Measuring the impacts of income and financial aid offers
on college enrollment decisions. Research in Higher Education,
40(3), 247-259.
Bridges, G. L. (1999). Performance-funding of
higher education: A critical analysis of performance funding in
the state of Colorado. Unpublished doctoral dissertation,
University of Colorado.
Bryk, A. S., & Raudenbush, S. W. (1992).
Hierarchical linear models. Newbury Park, CA: Sage
Publications, Inc.
Burke, J. C., & Minassians, H. (2001).
Linking state resources to campus results: From fad to
trend—The fifth annual survey. New York: The Nelson A.
Rockefeller Institute of Government.
Burke, J. C., Rosen,J., Minassians, H.,, &
Lessard, T. (2000). Performacne funding and budgeting: An
emerging merger? The fourth annual survey. New York: The
Nelson A. Rockefeller Institute of Government.
Burke, J. C., & Modarresi, S. (1999).
Performance funding and budgeting: Popularity and
volatility—The third annual survey. Albany, NY:
Rockefeller Institute of Government.
Burke, J. C. (1998). Performance funding
indicators: Concerns, values, and models for state colleges and
universities. New Direction for Institutional Research, 97,
49-67.
Burke, J. C. & Serban, A. M. (1998).
Current status and future prospectus of performance funding and
performance budgeting for public higher education: The second
survey. Albany, NY: Rockefeller Institute of Government.
Burke, J. C., & Serban, A. M. (1997).
Performance funding and budgeting for public higher education:
Current status and future prospects. Albany, NY: Rockefeller
Institute of Government.
Cabrera, A. F., Nora, A., & Castaneda, M. B.
(1992). The role of finance in the persistence process: A
structural model. Research in Higher Education, 33(5),
571-593.
China, J. W. (1998). Legislating quality: the
impacts of performance based funding on public South Carolina
technical colleges. Unpublished doctoral dissertation,
University of Texas at Austin.
Christal, M. E. (1998). State survey on
performance measures: 1996-97. Denver, CO: State Higher
Education Executive Officers/Education Commission of the
States.
Coulson-Clark, M. (1999). From process to
outcome: Performance funding policy in Kentucky public higher
education, 1994-1997. Unpublished doctoral dissertation,
University of Kentucky.
Dallet, P. H., Wright, D. L., & Copa, J. C.
(2002). Ready, fire, aim: Performance funding policies for public
postsecondary education in Florida. In Joseph C. Burke and
Associates, Funding public colleges and universities:
popularity, problems, and prospects. Albany, NY: The
Rockefeller Institute Press.
Dayhoff, D. C. (1991). High school and college
freshmen enrollments: the role of job displacement, Quarterly
Review of Economics and Business, 31(1), 91-103.
Florida Board of Education (2001). State
University System Accountability Report. Tallahassee, FL:
Author.
Florida Office of Program Policy Analysis and
Government Accountability (2001). Justification review of the
state university system (Report No. 01-28). Tallahassee, FL:
Author.
Heller, D. E. (1997). Student price response in
higher education: An update to Leslie to Brinkman. The Journal
of Higher Education, 68(6), 624-655.
Hoyt, J. E. (2001). Performance funding in higher
education: The effects of student motivation on the use of
outcomes tests to measure institutional effectiveness. Research
in Higher Education, 42(1), 71-85.
Hsing, Y., & Chang, H. S. (1996). Testing
increasing sensitivity of enrollment at private institutions to
tuition and other costs. The American Economist, 40(1),
45-56.
John, E. P. St., Hu, S., & Weber, J. (2001).
State policy and the affordability of public higher education: The
influence of state grant on persistence in Indiana. Research in
Higher Education, 42(4), 401-428.
Kahn, J. H., & Nauta, M. M. (2001). Social
cognitive predictors of first- year college persistence: The
importance of proximal assessment. Research in Higher
Education, 42(6), 633-652.
Kim, K. B. (2001). A study of institutional
policy-makers’ perceptions of performance funding indicators
for higher education in Korea and United States. Unpublished
doctoral dissertation, University of Iowa.
Levin, J. S. (2001). Public policy, community
colleges, and the path to globalization. Higher Education,
42, 237-262.
Marcantonio, R. J. & Cook, T. D. (1994).
Convincing quasi-experiments: The interrupted time series and
regression-discontinuity designs. In Joseph S. Wholey, Harry P.
Hatry, & Kathryn E. Newcomer (Eds.), Handbook of practical
program evaluation. San Francisco: Jossey-Bass Inc.
Marcus, D., Cobb, E. B., & Shoenberg, R. E.
(2000). Missouri coordinating board for higher education: Funding
for results. Lessons learned from FIPSE project IV.
District of Columbia, U.S.: 05/2000. (ERIC Document Reproduction
Service No. ED 443 300)
National Center for Public Policy and Higher
Education. (2000). Measuring-up 2000: The state-by-state report
card for higher education. Retrieved from http://measuri
ngup2000.highereducation.org
Poisel, M. A. (1998). An evaluation of
performance-based budgeting in the Florida Community College
System. Unpublished doctorate dissertation, Florida State
University.
Ruppert, S. S. (1995). Root and realities of
state-level performance indicators systems. New direction for
higher education, 91, 11-23.
Schmitz, C. C. (1993). Assessing the validity of
higher education indicators. The Journal of Higher Education,
64(5), 503-521.
Sheridan, P. M., & Pyke, S. W. (1994).
Predictors of time to completion of graduate degrees. The
Canadian Journal of Higher Education, XXIV-2,
68-88.
South Carolina Commission on Higher Education
(2002). Studying the effects of performance funding on
planning, budgeting and student outcomes. The paper presented
in FIPSE South Carolina conference of performance funding,
February, 2002.
U.S. Department of Labor (2002, SEP.). Local
area unemployment statistics. Retrieved from http://www.bls.gov/au/ast
ch98htm; http://www.bls.gov/lau/
astrk98.htm; and http://www.bls.gov/lau/l
auastrk.htm
Usdan, M. D. (2002, Spring). The new state
politics of education. The State Education Standard,
14-18.
Van Vught, F. A. (1994). Policy models and policy
instruments in higher education: The effects of governmental
policy-making on the innovative behavior of higher education
institutions.Higher education: Handbook of theory and research,
X.
Wildavsky, A. (1988). The dance of dollars:
in the new politics of the budgetary process. Glenview,
Illinois: Scott, Foresman/Little Brown College Division.
Zernike, K. (2002, August 4). Accountability hits
higher education. The Chronicle of Higher Education, p.
B20.
Zumeta, W. (2000). Accountability: challenges
for higher education. Retrieved from http://www.nea.or
g/he/mea/ma2k/a00p57.pdf
About the Authors
Jung-cheol Shin, Ph. D.
Research Associate, Center for Educational Research and Policy
Studies
Florida State University
Stone Building 312-F
Tallahassee, FL 32306-4463
Phone: 850-668-3962 Fax: 850-644-1592
Email: jcs6205@yahoo.com
Sande Milton, Ph. D.
Professor, Educational Leadership and Policy Studies
Florida State University
Stone Building 113
Tallahassee, FL 32306-4451
Phone: 850-644- 6777 Fax: 850- 644-1258
Email: sdmilton@mindspring.com
|
