Stigma Without Sanctions : The ( Lack of ) Impact of Private School Vouchers on Student Achievement

Under the Florida A+ Accountability Program, Florida’s schools are graded based on student performance on the Florida Comprehensive Assessment Test. Previously, when schools would earn their second failing grade within four years, students assigned to these schools were offered school vouchers which parents and guardians could use to transfer students to a private or another traditional public school. In January of 2006 the Florida Supreme Court declared that private school voucher component of the Florida A+ Accountability Program was unconstitutional, eliminating the threat of having these students and funds leaving to attend private schools. This exogenous shock allows us to test whether private voucher threats and the funding tied to these students led to increases in student achievement. We find no evidence that the private school voucher threats drive academic improvement beyond what is seen in schools when this private school voucher threat is removed.


Introduction
School systems have addressed lagging student achievement with a variety of education reforms and accountability policies.The No Child Left Behind Act of 2001 (NCLB) has made highstakes, accountability-based standardized testing paired with school choice policies increasingly common throughout the United States (Betebenner, Howe, & Foster, 2005).Though not exactly identical, many states have implemented accountability-based programs quite similar to Florida's A-Plus Accountability Program (A+) since the enactment of NCLB (Camilli & Bulkley, 2001).Since the A+ program precedes NCLB and was the model applied to the entire nation with the passage of NCLB, the vast panel data available from Florida has allowed researchers to explore the consequences of NCLB-type reform programs.
Under A+, schools are assigned letter grades (A -F) and receive sanctions or rewards based on student performance on the Florida Comprehensive Assessment Test (FCAT).Schools' grades are determined by an algorithm that includes their students' proficiency levels and test score gains. 1hen earning a second failing grade (F) within four years, the state places a school under state sanctions, provides additional resources, and offers vouchers to students assigned to attend these schools.
From the perspective of market-based education reformers, private school choice policies are fundamentally necessary for improving public school performance through competition (e.g., Friedman, 1962).Chubb and Moe (1990) contend that failing institutions are not likely capable of effectively implementing reform from within when it was likely the actions of those in charge that brought about the failing status (or at least the inability to make improvements).Chubb and Moe (1990) even make a case for "the notion that choice is a panacea" (p.217).However, despite this assertion, there is also a basis for choice alone being insufficient if parents do not have information regarding school performance and that uniting choice and test-based accountability is necessary for improving outcomes (Hastings & Weinstein, 2008).Therefore, state policymakers might more successfully improve school performance by motivating district and school administrators with sanctions and incentives in addition to choice (Bettebenner, Howe, & Foster, 2005).
While traditional public schools are typically already competing with charter and private schools for students, private school vouchers can expand schooling options, especially to students from low socioeconomic status (SES) households.Moreover, the private school vouchers also provide a threat to school districts' funds that are tied to student enrollments.Greene and Winters (2003) elaborate on this concept: "The theory behind the A+ Program is that chronically failing public schools will have an incentive to improve if they must compete with other schools for students and the funding they generate" (p. 1).However, Carnoy (2001) contends that the need for choice policies within test-based accountability programs is likely overstated, and Ladd and Glennie (2001), directly addressing Greene's (2001) earlier study on this issue, find that "Greene has inappropriately attributed the differential gains to the voucher program rather than to the other effects of being labeled a failing school, such as shame, increased scrutiny, and possibly additional resources" (p.49).
As state education policymakers continue to assess ways to increase standardized test score performance at lower-performing schools, it is critical to examine the effectiveness of particular components within these accountability programs.Examining one particular component of an accountability program is often difficult as multiple policies are typically implemented simultaneously as part of a comprehensive reform plan.In instances where these types of private voucher sanctions lead to positive results, it could be that NCLB-type accountability programs could have produced comparable effects without the additional threat of a private school voucher sanction and loss of funds (Harris, 2001).For example, in addition to the provision of private school vouchers, states also more closely supervise and allocate additional resources to sanctioned schools.These schools also receive a great deal of public scrutiny from parents and the media as a result of "underperforming".
To better estimate the effect of a private school voucher sanction within the context of a state accountability system, we take advantage of a major change to Florida A+ that provides a natural experiment that allows us to isolate this sanction's impact.In January of 2006 the Florida Supreme Court declared the private school voucher component of A+ was unconstitutional, immediately eliminating the private school voucher threat2 for students attending schools designated as failing (Bush v. Holmes, 2006). 3This immediate, exogenous change allows us to test the effectiveness of the private school voucher threat under a state's test-based accountability program, independent of the other accountability policies.Therefore, the purpose of this study is to examine whether the threat of private school voucher sanctions produces significant effects for students attending "lower-performing" public schools.
Testing the effectiveness of this private school voucher threat provides insight into whether the threat of losing students and funding incentivizes public schools to bring about increases in student achievement on state standardized tests.For this study it is important to note that we rely on the assumptions that producing gains on these tests is a valued objective of the designers of this accountability program and that standardized test score improvements represent a meaningful educational outcome.It is important to note that prior research has demonstrated that there are concerns with interpreting high-stakes assessment results, especially when these results are partnered with accountability measures (e.g.,Jones, 2007;Nichols & Berliner, 2007;Rubin, 2011;Wang, Beckett, & Brown, 2006). 4Therefore, while our study is specifically intended to inform the discussion of whether a particular aspect of state-based accountability systems are successful in driving changes in standardized test scores, these findings do not speak to or further examine the issue of whether these outcomes necessarily signify real or desirable benefits in terms student learning.
We find no evidence to support the claim that the threats associated with a private school voucher program produces test score gains beyond what is seen in lower-performing schools without this sanction in place.This finding has implications with regard to the design and implementation of state school accountability policies and programs.Private school vouchers are not necessarily harmful to the academic growth of public school students who are attending schools most likely to feel the pressures of such a sanction.However, this sanction, at least within the context of an NCLB-inspired accountability program, does not seem to significantly increase test score results beyond the improvements that occur without a private school choice sanction in place.
In the next section we provide an overview of the previous literature on test-based accountability systems as well as private school voucher sanctions.Next, we explain the methods and data used to conduct this study.Then, we present our findings and discuss their implications.Finally, we conclude with a summarization of some key takeaways in addition to some limitations and suggestions for further analyses on this subject.

Prior Research Standardized Test-Based Accountability
Prior research has shown that NCLB-inspired, test-based accountability programs have generally produced significant increases in student achievement as measured by test score gains (Braun, 2004;Carnoy & Loeb, 2002;Dee & Jacob, 2011;Hanushek & Raymond, 2005;Lee, 2008;Rosenshine, 2003).When examining the impact of what are typically the most severe sanctions for underperforming schools, Figlio and Rouse (2006) and Greene (2001) find that accountability programs tend to be effective at increasing student performance for those students attending lowerperforming schools.Subsequent analyses call the magnitude and interpretations of these gains or the mechanisms by which schools achieve these gains into question (e.g., Amrein & Berliner, 2002;Amrein-Beardsley & Berliner, 2003;Camilli & Bulkley, 2001;Koretz & Barron, 1998;Kupermintz, 2001;Nichols, Glass, & Berliner, 2006).However, some studies provide empirical evidence that seems to suggest that these programs produce lasting and potentially meaningful improvements in student achievement on state-based assessments (e.g., Chiang, 2009;Rouse, Hannaway, Goldhaber, & Figlio, 2007;West & Peterson, 2006).
A common concern with test-based accountability programs is that school administrators and faculties respond to incentives in ways that only artificially improve student achievement as measured on high-stakes assessments (e.g., Chakrabarti, 2007;Jacob & Levitt, 2003;Kupermintz, 2001).Some of these responses include schools putting more emphasis on producing gains with students near proficiency cuts (Neal & Schanzenbach, 2010) and disproportionately excluding lowachieving students from testing (Heilig & Darling-Hammond, 2008).There is also evidence that suggests school accountability programs have substantially changed the way teachers view assessments in general (Brown & Harris, 2009).Where assessment was originally a way to decide how best to improve classroom instruction, it is now seen primarily as a school accountability tool.The results of these studies can help explain initial school-level responses as well as the influences that these stigmas or threats may have over school behaviors.
There is some evidence that sanctioned schools in test-based accountability programs do in fact change their practices in meaningful ways such as allocating more resources to assist the instruction of lower-performing students (e.g., Booher-Jennings, 2005;Rouse, Hannaway, Goldhaber, & Figlio, 2013).However, other studies show that these score gains on states' highstakes tests may not necessarily signify true learning gains in terms of other assessments used to measure student success (e.g., ACT, SAT, and NAEP) (e.g., Amrein & Berliner, 2002;Figlio & Loeb, 2011;Klein, Hamilton, McCaffrey, & Stecher, 2000).The lack of translation is plausibly the result of changes in stakes that cause schools and teachers to produce artificial gains (e.g., Chakrabarti, 2007;Jacob & Levitt, 2003;Neal & Schanzenbach, 2010).Therefore, while schools appear to substantially respond to test-based accountability programs and even produce lasting gains in student achievement, the extent to which these responses actually benefit student learning is strongly contested.

Private School Voucher Sanctions
Several studies have found that the voucher sanctions and corresponding financial incentives are key contributors to the improvements that occur when accountability programs introduce competition to underperforming schools (e.g., Greene, 2001;Greene & Winters, 2003;West & Peterson, 2006).These studies appear to align with a general finding that competition from vouchers and tax credit scholarship programs tend to lead to system-wide improvements in student achievement (Egalite, 2013).Specifically, Greene and Winters (2003) conclude that schools experiencing greater pressures (i.e.schools already under private school voucher sanctioning or very close to sanctioning) produce greater gains relative to similar schools not feeling the same pressures.They attribute these differences in achievement gains to the magnitude of a private school voucher threat and suggest it as evidence for these threats independently driving gains in a test-based accountability system.West and Peterson (2006) find that public school choice stemming from failure to meet Adequate Yearly Progress (AYP) under NCLB, was not producing results comparable to those of the private school vouchers issued under Florida's A+ Program.This finding could suggest that private schools vouchers play a significant, independent role in increasing student achievement levels in lower-performing schools.This role can likely be attributed to the fact that private school vouchers put more pressure on district funds than public school choice.
Other studies, however, have attributed the positive results from accountability programs to schools' responses to other aspects of these types of sanctions (e.g., Figlio & Rouse, 2006;Goldhaber & Hannaway, 2004;Ladd & Glennie, 2001;Mintrop & Trujillo, 2005).For example, Figlio and Rouse (2006) find that the addition of the voucher sanction does not significantly improve student performance.They conclude that other sanctions, namely the state-provided additional resources and support as well as the stigmatizing of poor-performing schools, have greater influence over improvements in student achievement.Goldhaber and Hannaway ( 2004) conduct black box-type analyses to get a better sense of school-level responses to test-based accountability sanctions.These types of analyses are especially informative when it comes to examining intermediate outcomes that potentially have substantial roles on policy outcomes.They survey teachers and administrators to try and get a better sense of how school faculty, staff, and administrators were reacting to A+ incentives.Goldhaber and Hannaway (2004) conclude that school personnel are significantly more conscious of the reputation that accompanies their schools' grades as opposed to the possibility of students acquiring private school vouchers.Since teachers and administrators appear to be more aware of public scrutiny than private school choice sanctions, there is reason to believe that the former does more to directly motivate school personnel than the latter.This finding potentially informs the results of our study and is therefore further explored in the greater test-based accountability context in the discussion of our results.
We hope to further inform this discussion by examining how the sudden discontinuation of a private school voucher sanction affects test score performance for students attending schools that have been deemed to be underperforming.Unlike many prior studies, we are able to examine how the same population of students performs both with and without the presence of this sanction.Moreover, the fact that we examine the discontinuation rather than the introduction of this policy allows us to analyze whether the lack of impacts from these sanctions is attributable to results that only materialize after some time.In the following sections, we describe our data and methods, provide the results of our analyses, discuss the implications of our findings as well as limitations, and conclude with takeaways for state policymakers as well as proposals for future research.

Data
This study utilizes a dataset provided by the Florida Department of Education containing test scores in reading and math as well as demographic information for all students enrolled in Florida public schools for grades 3 -10 from 2002 -2008.The student level data are supplemented with school level information on the point total and grade earned by each school over the same time period.This dataset consists of over 8 million student/school/year observations.For this analysis we focus on a small subset since we are utilizing a time differences in regression discontinuity (RD) model. 5We use only students in grades 4 -6 attending schools that earned a grade of D or F within 0.5 standard deviations of the D/F break point.The vast majority of students attending these schools were eligible for free and reduced-price lunch (90 percent) and were either Black or Hispanic (72 percent), and 21 percent were identified as students with special needs.The full set of descriptive statistics for our analyses are broken out by grade level and are presented in Appendices A through C.

Methodology
Our methodology for analyzing the effect of the private voucher sanctions (or threats) on student math and reading achievement is similar to the Rouse et al. ( 2013) strategy.Since the point system used to grade schools changed the same year the voucher threat was removed (Florida Department of Education, 2010a), we are unable to use a cubic function of the schools' point totals to account for school quality.To measure only the impact of a school receiving an F and minimize picking up regression to the mean tendencies, we employ an RD approach and limit the sample to students in schools within 0.5 standard deviations of the D/F point breakpoint.For the 2002 -2005 school years in our dataset, this includes D-graded schools earning between 280 and 305 points, and schools earning an F between 254 and 279 points.The school grading scale changed for the 2006 school year, so for later years D-graded schools are included if they earn 395 -428 points and Fgraded schools are included when earning 362 -394 points.These restrictions leave us with several thousand student-level observations from which we estimate the following model: Where T indicates the test score for student i in school s during year t; f(T ist-1 ) is a cubic function of a student's test score in the prior school year; X it is a vector of observed student characteristics in year t; Y is a year indicator (the calendar year of the fall semester); F st-1 indicates the school attended by student i in year t received an F in the previous school year; F*Threat indicates the school earned an F in the previous year and the private school voucher threat was still present; and st is a stochastic error term clustered by school and year.
We estimate this model for each level, from fourth to sixth grade, for reading and math test scores separately.Third grade is excluded because it is the first tested grade level, so no previous year test scores are available for prediction.Students above grade six are excluded since there were very few junior high and high schools earning F grades under A+.Any impact of an F we find for secondary schools is likely to be school specific rather than a result of the statewide policy.We estimate a separate equation for each grade to allow for different achievement growth patterns by grade level as the curriculum, test difficulty, and other cohort level conditions vary.The coefficients of interest are 4, which estimates the overall effect of receiving a high F as opposed to a low D, and 5 , which estimates the differential effect of receiving an F when voucher threats are still present.

Results
The results from estimating the model using math test scores are presented in Table 1.As expected, previous test scores and student demographic controls have a highly statistically significant influence on current test scores for every grade level. 6We estimate two sets of models, one set (1-3) with 2008 included and another set (4-6) with 2008 excluded since the point accumulating school grading system was altered that year.7In all models the excluded year is the 2005-2006 school year.This is the year during which the Florida Supreme Court ruling (Bush v. Holmes, 2006) was announced and it is not clear whether schools anticipated a change to the private voucher threat while the lawsuit was pending, and the point accumulating system was also altered the same year.
For math scores, the coefficient on the school receiving an F in the previous year is positive and statistically significant in four of the six equations, indicating that students in schools recently earning a high F experience larger test score gains than students in schools recently earning a low D. The coefficient on the interaction term for a school with an F grade while the threat was in place is negative but statistically insignificant, indicating the threat of students receiving private school vouchers has no additional impact beyond the other consequences that come with receiving an F. The results obtained from the data on reading test scores are found in Table 2.As with the math results, previous test scores and demographic controls are statistically significant and in the direction consistent with other results.Again, we see positive results when schools earned an F in the previous year; although, they are only statistically significant in three of the six equations.When the F grade is interacted with the existence of the voucher threat, the coefficients are negative, have an absolute value greater than the F coefficient, and are statistically significant in four of the six equations.This result suggests that the threat of private school voucher sanctions actually contributes to lower student test performance in reading.

Discussion
We now discuss the results of our analyses and provide some policy implications.In the majority of our model specifications, elementary students attending schools that recently received grades of F make statistically significant gains in both math and reading.This finding corroborates with studies that have found that test-based accountability programs do indeed produce significant gains in students' state assessment test-score gains (e.g., Carnoy & Loeb, 2002;Dee & Jacob, 2011;Figlio & Rouse, 2006;Hanushek & Raymond, 2005).When examining the interaction of a failing grade with the presence of a private school voucher sanction, we find no impact on math scores and a statistically significant decrease in reading.Therefore, our results also provide evidence to support the conclusions of Figlio and Rouse (2006), Ladd andGlennie (2001), andHarris (2001) that an accountability-based private school voucher sanction does not independently produce increases in achievement for students enrolled in lower-performing schools.
There are at least a few possible explanations for these findings.First, the lack of effect for sanctions in math likely indicates that the possibility of losing students through private school vouchers does not incentivize schools in ways that necessarily increase test score results.However, these schools are motivated to remove the reputational effects of a failing grade or benefit from the additional attention and resources that the state provides.It is also plausible that schools in this study did not take the private school voucher threat seriously due to low take-up rates in the years that the private school vouchers were available.The year with the highest Florida Opportunity Scholarship Program enrollment was 2004-05 with only 763 students statewide using the scholarship to enroll in private schools (Florida Department of Education, 2010b).This explanation is consistent with the findings from Goldhaber and Hannaway ( 2004) that school personnel were very much unaware of the private school voucher sanctions, especially relative to public scrutiny and state attention.Since teachers, principals, and district administrators did not really experience a massive student exodus with the offer of private school vouchers, it is unlikely that this sanction provided any significant, additional incentive.However, these explanations do not explain the negative effect of sanction threats on reading performance.
Despite the exhaustive data available, we are not currently able to explain the negative effect of the threat on reading performance definitively and must rely on theory and intuition to guide this aspect of the discussion.One potential explanation for this finding is that school improvements implemented as a result of accountability initiatives take time before showing significant effects in student achievement.If student performance is improving over time as a result of these accountability measures, test scores would be highest in the most recent years.Therefore, it could be the case that the maturing and effectiveness, at least for the A+ Program as reflected in reading scores, happens to coincide with the removal of the private school voucher sanction, leading to a negative bias in the estimated impact of the private school voucher sanction.
Another possible explanation is that something else, which is unobservable, changes when the voucher threat was removed, such as parental attitudes or involvement.Jacobsen, Saultz, and Snyder (2013) suggest that parental satisfaction and public support fall as schools go on the verge of sanctioning.Voucher threats potentially have a negative influence on school-community morale.For example, some parents might lose the motivation to be as active and work with these struggling schools to improve academic performance when vouchers are potentially available.Once the private voucher program is eliminated, school-community members might now have a stronger incentive to become more involved.The fact that this effect is found with reading and not math may support this interpretation since reading achievement is often more closely tied to parental involvement relative to other subject areas (Cronin, Kingsbury, McCall, & Bowe, 2005;Heyneman, 2005).
Finally, there are also limitations to consider when examining the results of this study.While we wish to further inform the broader discussion on NCLB-type test-based accountability programs, it is important to note that our data are limited to Florida's elementary-aged student test-score achievement over a six-year period.We also do not know the extent to which the private school voucher threat significantly impacts the performance of schools not near the cut point for being deemed as underperforming.It is likely that these schools on the bubble are the most likely to change significantly from the addition or removal of these types of sanctions, but we cannot say with certainty how private school voucher threats influence schools at different performance levels.Finally, we focus on achievement only in terms of standardized test score gains.Private school voucher sanctions could have significant impacts on other educational outcomes not examined in this particular study such as student retention, graduation rates, and college enrollment.

Conclusion
The findings from this study suggest that a state-operated private school voucher sanction does not incentivize lower-performing public schools in ways that increase students' test scores beyond the effects from the stigma and attention that come with being deemed as underperforming.Despite the ineffectiveness of the private school voucher sanction, schools do appear to take action and improve on standardized test-based measures when school performances are graded and made public.
There are some lingering issues that future research could address in order to still better inform the and implementation of test-based accountability programs.While raising student achievement in terms of test score gains is a major concern for education stakeholders, researchers should evaluate the impact that the incorporation or discontinuation of private school voucher sanctions has on other important outcome measures, such as student grade promotion and degree attainment.The fact that there was a low private school voucher take-up rate in Florida also raises the question of whether this sanction would have greater impact with more students (and funds) actually leaving a public school district.Lastly, the finding of a negative effect in reading scores raises questions that beg for additional research on the mechanisms going on inside the black box.Unfortunately, we are not able to address these issues with our data, but hopefully future research will conduct analyses like that of the Goldhaber and Hannaway ( 2004) study to better address these important questions.
The strong, positive response to a negative stigma (in addition to the null/negative impact of the voucher sanction) still has significant implications with regard to constructing optimal education policy.There is considerable debate about the role of private school vouchers in accountability programs.Some school choice proponents argue that these vouchers incentivize schools to perform better when faced with the possibility of losing students and funding, but many voucher skeptics contend that publicly reporting performance and the extra public attention and state resources are primarily responsible for the success of test-based accountability programs.Our findings indicate a state-funded private school voucher sanction does not significantly increase student achievement, independent of other aspects of an accountability program.While state-provided private school vouchers may not be a necessary component of effective school reform programs, there is, clearly, still much research to be done so policy makers can confidently design programs to use states' education resources most efficiently.

Table 2
Stigma and Voucher Sanction Threat Impacts on Reading Scores