Citation: Laitsch, D. (2002, December 4). K-12 voucher programs and education policy: An exploratory study of policy maker attitudes and opinions, Education Policy Analysis Archives, 10(48). Available http://epaa.asu.edu/epaa/v10n48/.

Introduction

On June 27, 2002, the U. S. Supreme Court ruled that the use of vouchers to pay tuition at private K-12 schools in Cleveland, Ohio, did not violate the establishment clause of the U.S. Constitution (Zelman v. Simmons-Harris, 2002). This ruling energized the pro-voucher movement, and resulted in plans by state legislators across the country to introduce new voucher legislation (Toppo, 2002). While there is debate as to how far reaching this decision may ultimately be (some individuals have compared the decision to the 1954 Brown v. Topeka Board of Education decision), observers generally agree that the policy focus, both legislative and judicial, regarding the use of vouchers for K-12 tuition, will now turn to the states (Gehring, 2002; Toppo, 2002). In fact, less than two months after the Zelman decision, Justice Kevin P. Davey of the Leon County Circuit Court, struck down Florida's state-wide voucher program as violating that state's constitutional prohibition on state aid to religious institutions (Holmes v. Bush, 2002). The Institute for Justice has also filed law suits in two states (Maine and Washington), seeking to have state-level prohibitions against the use of public funds at religious schools overturned (Institute for Justice, 2002).

As voucher policy is drafted, introduced and debated, the views and beliefs of state legislators will play a critical role. To help determine just how policy makers react to vouchers and school choice issues, a survey of state legislators was conducted in November, 2000. The results of this research are detailed in the accompanying article.

Despite interest from policy makers and advocacy groups, little research has been completed that would lend to a clearer understanding of the effectiveness of arguments used to promote or inhibit voucher plans and how policy makers respond to such arguments. The study  conducted here attempts to examine policy maker reaction to these arguments, using the following questions:

• What role do policy makers feel vouchers play in the larger context of reform?
• How do policy makers react to specific policy arguments about vouchers?
• How does the educational philosophy of individual policy makers relate to their attitude regarding vouchers?
• How do the demographic traits of policy makers relate to an their philosophy of education?
• How do the demographic traits of policy makers relate to their views on vouchers?

This study specifically examined efforts to privatize education services through the use of vouchers by looking at three states where vouchers were enacted (Ohio, Wisconsin, and Florida), as well as three states where vouchers have not been enacted, but where serious attempts have been made to establish such programs (Michigan, New Mexico, and Pennsylvania). By examining policy maker reaction to arguments designed to promote or prevent voucher programs in states where vouchers were established or under serious consideration, this research sought to help both proponents and opponents focus on the concerns of policy makers in education reform.

Background

The education market movement is a compilation of at least three different efforts to expand the role of private providers in public education. These efforts include contracting out support and curricular services (such as food service, student transportation, and curriculum programs); contracting with a private company for management of an entire school or district (such as through Edison schools); and privatization of school governance (charter schools, tax-credits, and vouchers). While the term privatization may encompass all of these movements, as it is used here, privatization, vouchers, or market reform will generally refer to the privatization of governance through the transfer of public funding from public schools to the private sector.

Method

It is not until recently that the use of vouchers as a mechanism for creating a free-market system has been seriously considered; and not until the 1990s that a useful model of a voucher system established. The social and political contexts under which voucher programs were established then are relatively recent. This recent history presented some unique methodological opportunities (the chance to survey actual participants in the policy making process) and difficulties (no conclusive outcome or evaluative measures, continued doubt as to the constitutionality [at the state level] of such programs). For a detailed description of the methodology used in this research, see Appendix I.

In general, a quantitative methodology that allowed for wide spread application and comparative analysis, was sought, and consequently, a survey of legislators was chosen as the best option for answering research questions across a wide geographic range and large body of potential data sources. The focal points of this study (Florida, Michigan, New Mexico, Ohio, Pennsylvania, and Wisconsin) were chosen using both critical case sampling and politically important case sampling (Martella, Nelson, & Marchand-Martella, 1999).

The survey was constructed using pro- and anti- voucher arguments uncovered during a review of voucher literature. During the validation process, the survey was amended to include questions related to broader issues, including policy maker views on the purpose(s) of education, as well as the potential effectiveness of a variety of current reform proposals. Demographic data, including race/ethnicity, gender, age, and religious affiliation, was gathered to test for any specific response patterns. On October 18, 2000, the survey was mailed to 936 state legislators in Florida, Michigan, New Mexico, Ohio, Pennsylvania, and Wisconsin. Of the 936 surveys mailed, 89 were returned, for an overall return rate of 9.5% (see Table A1, Appendix 1 for a detailed discussion of the return rate and Appendix 2 for a discussion of methodological issues).

General Overview of Findings

Just under half of the policy makers surveyed (48%) supported some sort of voucher reform, however, when asked to evaluate vouchers in isolation, and when examined in relationship to other reform options, their support weakened, with vouchers ranking last among eleven other reform options. Policy makers also viewed the broader concept of school choice more negatively, with three of the four lowest ranked reform options on this survey related to school choice (teacher preparation and professional development, early childhood initiatives, phonics based reading programs, and greater use of technology all rated higher as reform strategies).

While choice was not favored in comparison to other reforms, policy makers still generally seemed to accept pro-market arguments for voucher programs, as well as statements related to allowing consumers to regain control of educational systems. While respondents seemed to accept at face value statements that competition will improve services, they did not support statements related to the potentially negative impact of market forces.

Even though there was broad agreement on the benefits of a pro-market approach, there were concerns expressed relative to resource allocation. Respondents did not support the idea that vouchers would help equalize funding, or provide new schools, increased investment, or improved cost controls. In fact, they felt that voucher programs might result in reduced services to special education students and increased competition for the "best" students. Respondents were also concerned that private schools would raise tuition and fees, ultimately limiting accesses to the schools by poorer families.

One of the most common warnings of voucher opponents is that voucher programs may result in a separation of students by race. Three items were related to this concern were included in the survey. Voucher advocates supported two of the more positively worded items (that vouchers would force schools to focus on customers seeking specific academic, social, or religious programs, and that such programs would result in schools with specific religious affiliations or racial/ethnic compositions), while voucher opponents were more likely to support the more negatively worded suggestion that voucher programs would increase segregation by race, religion, or income.

While some policy makers expressed concerns that vouchers would result in private schools loosing independence, others worried that there would be a jump in the number of low quality schools, as entrepreneurs sought access to newly available public funding (this dichotomy between independence and oversight has played out in all of the programs enacted so far, which currently include either no, or very weak, evaluation and oversight components).

When examining education more generally, respondents agreed that the overriding purpose of education is to ensure academic excellence in students. Respondents also supported the statement that education should ensure that students are prepared to meet the needs of businesses and employers, however voucher advocates were much more likely to want schools to instill strong moral character in students, while voucher opponents wanted schools to create good citizens. There was also a general reluctance to rank social purposes-including the suggestion that education should be used to promote social mobility or diversity-strongly. Women as a group tended to rank the creation of good citizens more strongly than men, but that emphasis came at the expense of business.  Respondents who were minority group members were slightly more likely to rank social mobility higher than Caucasian respondents. For both women and minorities, however, the degree of difference between their opinions and that of the majority was not statistically significant.

Item Analysis¹

Demographic Data

The typical respondent to this survey was a white protestant male, however, the diversity represented by the respondents is similar to that of legislators nationwide and across the study states (see Figures 1 & 2). Approximately  79% of respondents were male, and 19% female, with 2% not answering (this figure remains constant and so will not be repeated). Approximately 80% of respondents were white, with 6% identifying themselves as black, 11% Hispanic, and 1% Native American. The age range of respondents is shown in Figure 3. The large majority of respondents were actively religious (87%) and Protestant (65%). 26 percent were Catholic, and 2% were Jewish. Most respondents had completed a Bachelors degree and many had gone on to post-secondary education (BA, 46%; Professional degree, 15%; MA, 14%; Ph.D., 7%). A slight majority of respondents were Republican (53%) while Democrats made up 45% of the responders. A large majority had also served on the education committee (66%). Most respondents attended public K-12 schools (72%), and none attended private non-parochial schools at the K-12 level. Most of the policy makers also attended a public college (59%), while those who went to a private colleges and universities were spilt between religious institutions and nonreligious. While 72% of respondents who had children sent them to public schools, the 28% of respondents who sent their children to private schools chose parochial schools over non-parochial ones (83% to17%). While education service was generally low (ranging from 0 - 6% in most categories), of those individuals who had worked in education, 27% said they had worked as a public school teacher, 22% had volunteered in a public school, and 9% had volunteered at a private school.

Figure 1. Survey/across state demographic comparison

Figure 2. Survey/across national demographic comparison

Purpose of Education

Participants were asked to identify what they thought the primary purpose of education is, given six options (instill academic excellence, ensure students can meet the needs of business, create good citizens, instill moral values, promote social mobility, promote diversity), as well as an option to write in an option of their own. The majority of legislators responding to this survey clearly felt that the primary purpose of education is to ensure academic excellence in students. 61 percent of respondents identified this purpose as their primary choice and 80 percent ranked this option as first or second. The second choice of respondents was to ensure that students are adequately educated to meet the needs of businesses and employers. 45 percent of respondents ranked that option as first or second. Very few respondents considered social mobility or the promotion of diversity an important function of schooling. Figure 4 shows how the respondents reacted to each option.

Figure 3. Age of respondents

Figure 4. Purpose of Education

Success of Education

To evaluate the perceived need for education reform, respondents were asked to rate the effectiveness of schools in addressing the primary purpose of education they identified in the first item. As with the annual polls conducted by Phi Delta Kappa and Gallup, which have consistently shown that the public rates local schools higher than they do schools nationally (Rose & Gallup, 2000), the policy makers responding to this survey also rated local schools more positively. While the overall difference was relatively small (local schools had an average rating of 2.8 compared to schools nationally, at 3.4), it is statistically significant. More respondents gave local schools a one, two or three at each level (the highest ratings) than they did schools nationally, and more schools nationally were rated with a four, five, or six at each level (the lowest ratings), than were local schools. There was however a strong tendency toward the middle ranges, with 44% of respondents giving local schools a three or four and 72% rating schools nationally with a three or four.

Reforms

Even at the local level then, policy makers see significant room for improvement in school performance. While this research focused primarily on vouchers, market reforms are only a part of the entire reform picture. Research from Tennessee and Wisconsin has shown class size reduction as an effective tool to help raise achievement (Pritchard, 1999), while research by William Sanders (Sanders & Rivers, 1996) has shown teacher effectiveness, and by extension teacher preparation, to be the largest influence on student achievement. Policy makers across the country have also been strengthening testing and accountability procedures, and Charter schools are now supported in most states, educating more than 250,000 students (Shokraii-Rees, 2000). When policy makers consider voucher programs, it is within this wide variety of reform options. To determine the degree of effectiveness policy makers feel these reforms may have in improving education, respondents were asked to rate eleven possible reforms: class size reduction; site based management; vouchers; open enrollment; standards, testing, and accountability; greater use of technology; early childhood initiatives; phonics based reading programs; teacher preparation; increasing teacher salaries; and charter schools. In addition, respondents were given the opportunity to add some other reform if they desired.

Figure 5. Estimated effectiveness of selected reforms—highest to lowest

The highest rated option (of the choices offered) was reforming teacher preparation and professional development; however, the difference between teacher preparation and early childhood initiatives was not statistically significant. 77 percent of respondents rated teacher training reforms with a one or a two, and the item's average rating was 1.90. Early childhood initiatives had the same number of "1" ratings, but only 67 percent of respondents rated such initiatives a 1 or a 2, resulting in a 2.02 average (see Figure 5). The reform respondents had the least overall confidence in was vouchers, followed by charter schools and raising teacher salaries. Despite their apparent closeness, the difference between vouchers as a reform, and charter schools or raising teacher salary, is statistically significant. This apparent lack of confidence in vouchers as a reform is surprising given that the majority of respondents actually supported some type of voucher program (as discussed later in the paper­). It is possible that respondents may see vouchers as only a limited solution while the other reform options are seen as more available to all students, and so in comparison, vouchers were rated less favorably. As shown in Figures 6 and 7, it is also possible that voucher opponents skewed the average rank downward by choosing the lowest possible options, while voucher supporters spread their ratings more evenly across the scale (there were 39 ratings of five or six for vouchers, while the rest of the responses were spilt across ratings 1-4).

Figure 6. Education reform

Support for Vouchers

Respondents were asked to rank order their support for vouchers given options from support for no type of voucher program to support of an unlimited system (see Figure 8). The response ranked first most often was "no voucher system", however, when the first ranked responses for all types of voucher programs were combined, more respondents supported vouchers than opposed them (48% to 45%, with 7% undecided). Another key finding relates to what options respondents ranked second and third. While voucher opponents were forced to consider what type of voucher system was least objectionable (and they preferred vouchers for poor students first, then students at poor performing schools), voucher advocates had the option to choose another type of voucher system, or no voucher system at all. This all or nothing approach allowed an evaluation of the relative strength of the stated support for vouchers. Two thirds of respondents who ranked vouchers for poor students as their top choice chose an unlimited voucher program as their last choice. While this trend was not as strong with supporters of vouchers for families at low achieving schools (25%), it does suggest that the support of individuals who back narrowly tailored voucher programs might not generalize to support to all voucher programs. Not a single respondent ranked vouchers limited to non-parochial schools as their first choice.

Figure 7. Evaluation of vouchers as a reform

Because of the small number of undecided policy makers, their responses were generally not analyzed as a group, however five of the six respondents expressed a tendency to support some type of voucher system in their second place ranking, and so were included in the pro-voucher statistics. The other policy maker, who expressed a tendency toward no support for vouchers, was included in the anti-voucher analysis.

There is however a moderately strong correlation (r = -0.69) between support for vouchers for impoverished families and vouchers for students in poor performing schools (given the general correlation between high poverty and low performance, this may not be surprising). 89 percent of respondents supporting vouchers for impoverished students also supported voucher use at low performing schools. The reverse was also true, although slightly less so, with 75% of respondents who supported provision of vouchers to students in poor performing schools, also supporting provision of vouchers to impoverished students.

Figure 8. Support for voucher programs

Pro-voucher Statements

Respondents were asked to rank order 15 pro-voucher statements which were organized along three lines: excellence, access and equity, and education structures. Within the category of excellence, the highest ranked statement of the five options (a statistically significant difference when compared to the second ranked option) was "...voucher programs will make public schools more accountable by allowing consumers the option to take their business elsewhere." 48 percent of respondents ranked this item first, and its average ranking was 2.04. The next ranked item, with an average of 2.67 was "...voucher programs will stimulate innovation, research and development as schools seek better ways to increase achievement." The lowest rated item (3.81) was "voucher programs will result in increased service as schools are forced to offer incentives and enticements to potential students." Both opponents and supporters of voucher programs placed these statements in the same order (on average).

With regard to access and equity, the highest ranked statement (of six options) was "voucher programs will force all schools to offer strong academic programs, increasing access to quality education for all students." This item's ranking was not statistically different from the second highest ranked statement, "voucher programs will allow funding to follow the student, making it important for public schools to meet the needs of every student." The lowest ranked item, with an average rank of 4.56 and no number one ranking, was "voucher programs will increase funding by allowing new investors and entrepreneurs to enter the market." While the top three items in average rank were the same for both supporters and opponents of vouchers, voucher opponents gave a lower overall ranking to the statement, "ease the entry of new schools into the market place, increasing access to education."

In the final pro-voucher grouping, education structures, respondents strongly felt that voucher programs would "return control to parents" (a statistically significant difference when compared to the second ranked option). Respondents ranked as last the claim that costs would be reduced as fiscal management practices are emphasized. While the overall total was the same for the highest and lowest ranked items, voucher opponents ranked the statement, "reduce bureaucratic oversight since poor performing schools will be forced by consumers to improve or close" first twice as often as they did the "return control to parents" item.

Anti-voucher Statements

The patterns in responses to the anti-voucher statements are much less defined. For instance, under "Excellence" the statement with the most number one rankings (of six options) was item 47, "voucher programs will force schools to compete for only the best students in order to maximize achievement scores," however, a large number of "six" rankings lowered the mean score of this item to 2.89, placing it second by average. The item with the highest mean rank was item 49, "voucher programs will result in an increased emphasis on standardized tests." Another complicating factor in this analysis is the mode. In this case, item 49 had a mode of "3", while item 47 had a mode of "1", and the third and fourth ranked items both had a mode of "2". There was also a low degree of variability in the averages. The highest ranked items averaged 2.89 and 2.75 respectively, while the third and fourth ranked items averaged 3.21 and 3.36, and the fourth and fifth ranked items averaged 4.31 and 4.32 (none of these pairings had rankings that were statistically different). The lowest ranked items were item 51, "voucher programs will cause public schools to transfer existing resources to advertising and marketing," and item 50, "result in less spending on research and development as costs are cut."

Voucher supporters and opponents did not rank either the highest or lowest average ranked statement together. Supporters preferred the statement with the overall highest ranking, while opponents preferred item 49. Proponents rated item 51 lowest while opponents ranked item 50 lowest.

The responses to anti-voucher statements in the access and equity section resulted in a more clearly defined top ranked item (that was statistically difference from the second ranked option), but an even more confused picture of the other five items. The top ranked statement, vouchers would "reduce services to students traditionally more expensive to educate (special education)," had a mode of two (no item had a mode of one), and an average of 2.71, while the other items all averaged in the three range. With a variance in their average rankings of .08, the other five items were virtually indistinguishable. An examination of their modes did not clarify the rankings much, with the modes running a three, two fours, a five and a six, with a correlation between the average rank and mode occurring at only four and six. The lowest ranked item (six mode and sixth by average) was, "voucher programs will result in reduced services for students in rural or low attendance areas." Despite the generally confused picture of the rankings on this item, the first and last ranked items were the same for both voucher supporters and opponents.

While the modes and averages are confused for the next section (Structures of education) as well, a greater variability between the averages lends some clarity to the picture (even though the difference between the first and second ranked items was not statistically significant). The item with the highest average, 2.69, and a mode of 2, was, voucher programs will result in "the diverting of resources from the public sector to the private," while the next ranked item (average of 2.81 and a mode of 1) was "private schools losing independence as politicians attempt to ensure oversight of public funding through regulation." The lowest ranked items were again virtually indistinguishable, and included, voucher programs would result in "a proliferation of low quality schools designed to quickly access public funding with little or no oversight," "lower educational quality due to a lack of state and district oversight," and "increased segregation by race, religion, or income." The last two statements had the lowest average rankings at 4.061 and 4.059 respectively.

The highest ranked item by voucher supporters and opponents was, "voucher programs will result in the diverting of resources from the public sector to the private." Proponents, on average, ranked "increased segregation" last, while opponents ranked "a lack of oversight" last.

Written Comments

Generally written comments were in response to options marked "other", and comprised a very low percentage of responses to such items. The one exception was the first item: the Purpose of Education. Five comments were made regarding the "other" option on this item, and they were:

1. All of the above are important parts of the whole.
2. Ensure equity among all ability levels.
3. Ensure students are given the tools they need to make positive change and keep learning.
4. Prepare every student for life participation in our economy.
5. Ensure people are adequately educated to lead a good family and financial life.

Four comments were evaluative of the survey in general. Two respondents commented that they felt the survey was slanted against vouchers, while one commented that there was clearly a pro-voucher bias. A fourth respondent indicated discomfort with being forced to rank items that he/she did not support.

Cross-Sectional Analysis

Support for Vouchers

One of the main purposes of this research was to examine the relationship between support for vouchers and the other items on the survey. To do this, a correlation coefficient was generated between support for or opposition to vouchers in general, and then by type of program, and each item on the survey. Once an initial relationship was detected, a X² analysis was conducted to examine relationships within the correlation.

A respondent's ranking of the voucher statement "I support the concept of no voucher system" (item 22) was used to identify the level of support for such programs. While the wording of the item suggests opposition to vouchers, a ranking of 5 or 6 indicated support for vouchers rather than opposition. This item was used rather than the statement of support for an unlimited voucher system (item 26) because that allowed for inclusion of support for all types of voucher programs. There was a relatively strong negative correlation between the responses on item 22 and item 26, support for unlimited voucher programs (-0.74), which would support the use of item 22 in identifying support for voucher programs.

A respondent's ranking of the statements in item 22 also strongly correlated with item 12, evaluation of vouchers as a reform strategy (-0.898), and moderately with item 16 (evaluation of early childhood education as a reform strategy, r = 0.56), as well as item 70, party affiliation (r = -0.54). There was no linear relationship between support for vouchers and item 56, decreased access to quality schools as private schools raised tuition, (r = .000).

To further test the relationship between these items, a X² test was performed, with H₀: item X and item Y are independent; and H_a: item X and Y are not independent. Using a significance level of .05, and combining upper, middle, and lower rankings (1,2; 3,4; 5,6) so as to ensure at least 5 expected cell counts, the null hypothesis was rejected in favor of the alternate hypothesis, that is: item 22 and item 12 are not independent (p = 2.7(10)^-14 < 0.05). In this test, the two groups compared were voucher supporters and voucher opponents. Upon further examination of the distribution of rankings, it turned out that every respondent opposed to vouchers ranked the likely effectiveness as a 4, 5, or 6, while voucher supporters generally ranked the reform highly. While this may not seem surprising, it does buttress the internal validity of the document, as well as the original rejection of a Likert scale for the instrument (which was rejected because it was feared that respondents would tend to the extreme and rank the statements on the high or low ends of the scale).

An X² test could not be performed on item 12 as useful categorizations could not be constructed. The correlation may also be the result of a lack of variance in the rankings, with respondents choosing a 1, 2 or 3 for most every estimate of the effectiveness of early childhood initiatives. A X² test was successfully used to examine the relationship between party affiliation and support for vouchers and once again, the null hypothesis was rejected in favor of the alternate hypothesis (p = 0.0000056). Figure 9 illustrates the relationship. As could be expected, although there is some crossover appeal, the bulk of support for or opposition to vouchers falls along party lines.

Figure 9. Party affiliation

Stronger correlations became evident when controlling for the type of voucher program supported. When looking at individuals who supported only voucher programs for impoverished students and students in failing schools, moderate to strong correlations were found with ten items in the survey (items 19, 39, 40, 49, 54, 56, 60, 61, 62, and 84). This group was identified by controlling for support for vouchers but opposition to an unlimited voucher system (item 26). The strongest correlation, at 0.94, was between limited support for voucher programs and item 54, the statement that voucher programs would "result in a reduction of services and opportunities for all students as public schools enact stricter cost controls." Because support for limited voucher plans was identified by looking for a negative evaluation of unlimited voucher programs, this correlation could signify a general lack of support for the statement, even though the correlation is positive. Such an assumption is confirmed through further examination of the data (70% of respondents supporting limited voucher programs did not ranked this statement strongly).

A strong relationship  (-0.81) was also found between support for limited programs and item 40, the statement that voucher programs would ease the entry of new schools into the market place, increasing access to education. In this case then, the correlation could signify support for the claim that vouchers will increase access to education. Upon further inspection of the data, however, this is not necessarily the case. While 50% of the group did rank the statement strongly (1, 2, or 3), the other 50% ranked the statement negatively (4, 5, or 6). When looking at the total evaluations of this statement however, there does seem to be slightly greater support than was evident in the larger population of voucher supporters (where 78% ranked the statement with a 4, 5, or 6).

While there was no moderate or strong correlations between support for or opposition to voucher programs and the other survey items not highlighted, there were some trends that should be mentioned. In previous writings, women have been identified as opposing voucher programs. While there was only a weak correlation between gender and voucher support (i.e.., males and females analyzed together), 63% of female respondents (n = 16) did oppose voucher programs. In addition, while race or ethnicity generally did not correlate strongly with support for voucher programs, every African-American respondent supported vouchers (n = 5). Hispanic respondents were mixed in their support, with 4 opposed to all types, 2 supporting a limited program, and 1 supporting an unlimited program. These n's are especially small and should not be generalized to the larger minority population.

Evaluation of school effectiveness

There was not a strong correlation between a respondent's evaluation of the effectiveness of schools and any other item on the survey, although there was a positive correlation between a respondent's view of school success locally and nationally. Respondent's who ranked local schools poorly (that is, with a 4, 5, or 6), were more likely to support some type of voucher plan than the population as a whole (63% to 48%), while respondents ranking schools strongly were less likely to support voucher programs (69% to 45%). To further examine this relationship, a X² test was applied comparing the estimated effectiveness of local public schools with support for voucher programs as determined by item 22. Scores were counted as (1,2,3) and (4,5,6) to supply the required expected cell count, and a level of significance of 0.05 chosen. The resulting p value was 0.37, which means there is not sufficient evidence to cause rejection of the null hypothesis. While it appears that these items might be independent, the X² test confirms that this is not the case.

Support for statements and voucher support

An X² analysis was used to examine the relationships between respondent support for vouchers and the policy statements offered in the survey. Using a level of significance of .05 and grouping the responses by pairs, the null hypothesis, that the items are independent, was rejected in eight of the 33 items. The statements, along with their "p" scores, are identified in Table 1.

As Table 1 shows, there is a large disparity in the rankings of voucher supporters and voucher opponents on five of the statements. Voucher advocates generally viewed the statement "Voucher programs will force public schools to focus on the academic, social, or religious demands of specific customer groups" more positively than did opponents. A similar statement that was worded more negatively also appears in the table: "voucher programs will result in increased segregation of students by race, religion, or income." When worded in this manner, voucher proponents ranked the item negatively, while anti-voucher respondents ranked the item more strongly. Another item related to the issue of student-body composition and school focus appears on the list. The statement, "voucher programs will result in a proliferation of schools with specific religious affiliations or racial/ethnic composition," had a higher average rank with voucher advocates than with voucher opponents. The other two items with a large difference in average rank were: "voucher programs will result in private schools losing independence as politicians attempt to ensure oversight of public funding through regulation," and "voucher programs will result in a proliferation of low quality schools designed to quickly access public funding with little or no oversight." Voucher advocates ranked the statement regarding loss of independence higher on average that opponents, while voucher opponents were more concerned with a proliferation of low quality schools.

A final note needs to be made regarding interpretation of the items in this section. While care was taken to examine relationships between items, and between supporters and opponents of voucher programs, the analysis conducted here cannot suggest causation, and can only suggest areas where there appear to be relationships. In addition, the structure of the survey forced policy makers to rank a series of items. As with any forced ranking system, a low ranking is not necessarily negative. Respondents may agree or disagree strongly with all the items, but still rank them as the survey asks. While this effect is mitigated as the number of responses increases it would be unwise to make any absolute statements regarding the impact of the items on this instrument.

Discussion

Before reviewing the findings from this research, it bears repeating that while these conclusions can be used to inform the controversy surrounding the voucher debate, they cannot be interpreted or generalized to the larger body of legislators. The sample size, though largely composed of legislators in key leadership positions on education issues, and demographically reflective of legislators nationally, was small. This is also the first time this survey has been administrated, and no trend data are available; thus these data are representative of a single point in time and cannot be portrayed as sustainable.  Because this effort can be viewed as a pilot, researchers interested in replicating this research, or conducting similar legislative surveys, may be able to draw useful lessons by examining the survey application and study design.

While similar surveys of legislators have shown slightly higher rates of return, surveys of policy makers generally appear to have lower response rates than surveys of other populations. A review of dissertations available from ProQuest, an on-line repository of dissertations ( http://www.umi.com/hp/Support/DServices/products/da.htm), showed survey response rates for legislators ranging from 24% to 61%.(Note 2) Separate from this study, research into why policy makers respond poorly to surveys, as well as techniques that might be used to improve response rates, could fill a significant gap in the data (see Appendix 2 for a detailed discussion of the survey returns and how future research might address issues uncovered here).

This research sought to answer five basic questions:

• What role do policy makers feel vouchers play in the larger context of reform?
• How do policy makers react to specific policy arguments about vouchers?
• How does the educational philosophy of individual policy makers relate to their attitude regarding vouchers?
• How do the demographic traits of policy makers relate to an their philosophy of education?
• How do the demographic traits of policy makers relate to their views on vouchers?

An analysis of each of these questions follows.

What role do policy makers feel vouchers play in the larger context of reform?

While 48% of policy makers surveyed support some sort of voucher reform when asked to evaluate vouchers in isolation, when examined in the relationship to other reform options, their support appears much weaker, ranking last among eleven other options. Though apparently counter-intuitive, this is perhaps not an unusual finding. Annual polls sponsored by Phi Delta Kappa show just less than a majority of people they survey support vouchers generally (44% in 2000), however, when asked, "Which would you prefer - improving and strengthening the existing public schools or providing vouchers for parents to use in selecting and paying for private and/or church-related schools?" only 27% of respondents chose providing vouchers (Rose & Gallup, 2000).

If we look at the concept of choice and deregulation as broadly identified in this survey, policy maker support continues to be low. Three of the four lowest ranked reform options on this survey related to school choice (vouchers placed 11th, charter schools placed 10th, open enrollment placed 8th), while another option related to decreased regulation, site based management, placed 7th. The top rated reforms also appear more representative of the concept of change from within. Greater support for teacher preparation and professional development, early childhood initiatives, phonics based reading programs, and increased use of technology, suggest that policy makers still feel the "inputs" are important. That is, that change in "what" the schools do, rather than "how" they do it, is still a primary emphasis of policy makers.

How do policy makers react to specific policy arguments about vouchers?

In general policy makers showed a surprising unity of opinion regarding pro-voucher arguments. On average, both opponents and advocates gave 11 of 15 pro-voucher statements the same rank. The items ranked highest were pro-market in nature, suggesting that this has been an area where voucher advocates have successfully stated their positions. In addition, the highest ranked pro-voucher statements could also be characterized as related to the control of educational decisions, that is, consumers retaining control of educational systems. This is not a new concern in educational circles, as one of the primary results of education reform in the 20th Century was a consolidation and centralization of educational services. While this consolidation often results in a broader range of educational opportunities, it has also brought on a sense of isolation and loss of power in many communities (Cremin 1964; Ravitch 1983; Tyack, 1996). If pro-voucher support groups are able to capitalize on this discontent they could gain a significant philosophical boost, moving their arguments beyond the traditional market advocacy.

While there was broad agreement on the pro-market and control statements, there was also agreement on the lower ranked statements, which were primarily related to resource allocation. Statements concerning equalization of funding, the opening of new schools, increased investment, and better controlled costs were all ranked low. While such low rankings do not necessarily suggest a lack of faith in these effects, they at least suggest that this is an area were voucher opponents could make inroads against some traditionally argued pro-voucher positions. In addition, voucher proponents should take seriously concerns related to resource allocation within voucher systems. Such a strategy was used in Florida, Ohio and Wisconsin, where voucher programs also came with increased public school resources or resource guarantees, but was missing in the recently defeated ballot measures in California and Michigan.

As stated previously, there was a notable lack of unity in responses to the anti-voucher statements. Such a lack of consensus suggests that opponents of vouchers may not have been as successful as proponents in focusing policy makers on their concerns. While this lack of unity was true for the respondents in general, some broad patterns are still discernible. Both supporters and opponents of voucher programs exhibited some equity concerns. Statements predicting reduced services to special education students and increased competition for the "best" students were highly ranked by both groups. Respondents were also concerned that private schools would raise tuition and fees, ultimately limiting accesses to the schools by poorer families. As identified earlier, concerns related to resource allocations were highly rated. These statements were most frequently made by voucher opponents, and such support suggests that the message has been successful in reaching at least some policy makers. These concerns should also be noted anew by voucher supporters in designing pilot programs. Such findings confirm advice offered by Coons and Sugarman (1999), suggesting that advocates seek to establish narrowly tailored programs that take resources and equity concerns seriously, and then later attempt to expand the programs.

As evidenced in Milwaukee, and to some extent in Florida, it is easier to expand a program once it is already in effect. The findings also indicate that advocacy of a format similar to Florida's statewide program might be a successful approach for other states, since support of vouchers for impoverished students (the Milwaukee and Cleveland model) correlated with support of vouchers for students in poor performing schools (the Florida model). Such an approach could then be used to extend voucher programs merely by changing the definition of "poor performing schools", whereas changing the definition of poverty would undoubtedly be more difficult.

As with the pro-voucher statements, respondents generally did not rank very highly statements related to reduced services because of efforts to cut or control costs. This could signal a possible disconnect in the market model. While respondents seem to accept at face value claims that competition will improve service, they do not accept statements looking at negative market forces. While this dichotomy is present here, and in the advocacy literature as well, an understanding of why this disconnect exists could prove useful. For example, are negative market influences seen as unlikely because they haven't been generally considered (in which case voucher opponents could have a new position to pursue), or is there a belief that such influences are a good thing (forcing bad schools to go out of business), or that they won't affect good schools?

One of the most common warnings of voucher opponents, and a major reason the NAACP has taken an anti-voucher position, is that voucher programs may result in a separation of students by race. Historically, when schools were desegregated voucher programs were intentionally established to maintain racial separation, most notably in Virginia (although such programs were ruled unconstitutional at the time). Three items in the anti-voucher section were included related to this concern. On two of the positively worded items (that still imply student separation by race, religion, or social status), voucher advocates ranked the items strongly. The items stated: voucher programs will "force public schools to focus on the academic, social, or religious demands of specific customer groups," and, voucher programs will result in "a proliferation of schools with specific religious affiliations or racial/ethnic composition." While voucher opponents were less likely to rate these statements strongly, they did support the statement that such as programs would "increase segregation by race, religion, or income."

The narrow line between meeting constituent demands, and the creation of an illegal or unethical segregation of students, is one requiring close examination. Clearly, supporters feel that addressing constituent needs is important, but opponents may have valid concerns that such efforts will result in a segregated system. Further definition and clarification of this narrow line, and how constituent needs can be met without resulting in a segregated system should be a concern of voucher advocates. At the same time, this could continue to be an effective angle for voucher opponents to exploit.

Because of the low number of undecided respondents (n=6), there has been little mention of their responses pattern on these items. There were however, two key items on which they responded similarly. As a group, five of the six were Democrats, and four of the six respondents expressed concern that vouchers would result in private schools loosing independence.

This was also a concern expressed by voucher advocates. Voucher opponents did not generally share this concern; they did, however, rank strongly a statement that vouchers would result in a "proliferation of low quality schools designed to quickly access public funding with little or no oversight." This dichotomy is perhaps one of the most difficult to solve in program design, and one which voucher opponents can continue to highlight. This difference in perspective has played out in all of the programs enacted so far, which currently include either no, or very weak, evaluation and oversight components. Proponents typically argue that market forces will perform the oversight function, and that oversight is a "poison pill," believing that private schools will avoid systems that involve even a small amount of government oversight or regulation. The PDK poll cited earlier lends support to the use of oversight as limiting strategy by voucher opponents, as 76% of respondents to that survey thought private voucher schools should be required to meet the same accountability standards as public schools (Rose & Gallup, 2000).

How does the educational philosophy of individual policy makers relate to their attitude regarding vouchers?

In general, respondents agreed that the overriding purpose of education is to ensure academic excellence in students. There was, however, a small divergence of opinion when looking at the second and third ranked statements, which within each group were indistinguishable. Both groups supported the statement that education should ensure that students are prepared to meet the needs of businesses and employers, however voucher advocates were more likely to want schools to instill strong moral character in students, while voucher opponents felt schools should work to create good citizens.

Voucher advocates were also more uniform in their responses to these items. Through their first four rankings, they generally moved through the options in groups (1-6-2-4, or 6-1-4-2), diverging on items three and five. While voucher opponents generally ranked the first item strongest, they quickly diverged, splitting on items two and six, with no discernible response pattern among the remaining items. This suggests that while these groups share a sense of common purpose overall and then initially within the groupings, there is more divergence of opinion for voucher opponents. While it is a stretch to label either group "liberal" or "conservative", one of the more common political generalizations over recent years has been that conservatives have generally been more unified in their approach to policy, whereas liberals have put forth a less unified vision. At the very least, these results suggest that efforts of advocates to frame vouchers as a "social equity" issue might be effective to a degree. The general reluctance, however, of either group to rank social purposes strongly suggests that the effects would be small (for example, only two voucher supporters, and four voucher opponents ranked social mobility or promotion of diversity as a primary [1 or 2 ranking] role of education).

How do the demographic traits of policy makers relate to their philosophy of education?

It is difficult to determine a relationship between educational philosophy and demographic traits because of the unity of opinion related to the primary purposes of education and because of the small "n" for demographic differences. Even so, there were some disparities. Women as a group tended to rank the creation of good citizens more strongly than men. The emphasis on citizenship came at the expense of business, which had a lower average rank for females than it did for males (in effect, women as a group ranked citizenship second, business needs third, while the rankings were the reverse for men).  Respondents who were minority group members were slightly more likely to rank social mobility higher than white respondents. The first three rankings, however, were the same for each group (promote academic excellence, meet business needs, and create good citizens). In either case, for women and minorities, the degree of difference between their opinions of educational purpose and that of the majority of respondents was not statistically significant. The same was true when disaggregating by religion.

How do the demographic traits of policy makers relate to their views on vouchers?

Just as with the purpose of education, there is no apparent demographic pattern to those who support or oppose voucher programs. Where there is a difference, because the "n" in each subgroup is so small, the differences are generally the result of only one or two respondents, and therefore, not significant. While such a lack of disparity might suggest that support for vouchers is broad based, that, too, is a generalization that should nor be made in light of the small number of minority and female respondents.

Additional Research

Clearly, there is a need for more research into the impact of voucher programs on student learning. Given recent claims by some researchers that voucher programs stimulate public school improvement, research into the corollary effects of voucher programs on the education system should also be conducted. It is possible that a pilot voucher experiment could partially answer these questions. Given the failure at Alum Rock, however, along with the tendency of elements on both sides to back advocacy research over hard science, success of such a program, to say nothing of the possibility of starting one, seems tenuous.

While not directly cited here, research into press coverage of voucher issues may also be significant. While pro-voucher research groups like the Heritage Foundation, the Manhattan Institute, and the Project on Education Policy and Governance have proved skillful in getting their research into news stories, voucher opponents seem to have been much less successful. An understanding of the interaction between the advocacy organizations, voucher researchers, the press, and public opinion, could all be further areas of inquiry

Additional research into survey methodology, specifically as it relates to policy maker response rates, could also prove useful. As discussed in Appendix 2, many of the techniques used to increase response rates on surveys of the general public may not apply to policy makers, since courtesies such as personal salutations and signatures are expected, while postage costs are not a concern due to franking privileges.

Continued examination of policy maker reactions to the various advocacy arguments identified here is important. It seems likely that the debate will remain a philosophical one, and as such, attempts to sway legislators to one side of the issue or the other may ultimately rely on arguments based on emotion, faith, politics, and beliefs rather than hard data.

Final Observations

As highlighted earlier, the four major findings of this research are similar to findings of previous surveys oriented to the general public. First, there is little support for vouchers as a systemic reform strategy. While many legislators surveyed supported the concept of vouchers in some form, the large majority opposed the concept of unlimited voucher programs. Additionally, when offered a variety of reform options, legislators tended to have more confidence in reforms designed to work from within the current system (teacher education, early childhood education, and phonics based curricular reform) rather than reforms designed to fundamentally change the form of the current system (such as vouchers, charter schools, and even open enrollment).

Second, despite such dubious support for vouchers when compared with other reforms, policy makers responding to this survey generally believed in the concept of market reforms and the power of competition to act as a positive force in education. At the same time, they did not buy into arguments that the market philosophy could hurt education. While this may seem a dichotomy, it may not be. Legislators in three of the six study states (Florida, Ohio, and Michigan) are term limited, meaning that they have most likely only legislated during strong and expanding economies. It will be interesting to observe whether support for market reforms remains strong during times of recession.

Third, there continues to be an ill-defined line between the positive effects of schools meeting constituent needs, and effectively resulting in the creation of a system where participants can segregate themselves by race, religion or other values. It is difficult to predict the effect of a voucher program on integration because the cities in which vouchers are currently used, Milwaukee and Cleveland, are already majority minority systems. Research from the effects of charter schools on integration is also mixed, with some studies vulnerable to the same claims of bias as are present in the debate over voucher effectiveness. Various studies (Cobb & Glass, 1999) indicate a tendency toward segregation, while others (Center for Education Reform, 2000c) show charter schools acting as an integrating force.

Finally, while party affiliation remains a strong determinant of voucher support or opposition, the majority of undecided policy makers were Democrats. While this could signal a place where voucher proponents could make significant inroads, it seems unlikely that in the long term such Democratic support would result in a broadly accessible voucher system. When Democrats did support vouchers, support was primarily for vouchers directed toward poor families and students in failing schools. While this tendency could result in new programs, it seems unlikely given the opposition to unlimited programs, that vouchers would expand beyond the narrowly tailored plans currently available. Florida's program, however, could prove to be the linchpin in expanding voucher programs. The voucher program for special education students is already showing tremendous expansion, and, should the program emerge from the courts entac, it seems possible that the voucher program for students in poor performing schools will expand significantly as the state's accountability program becomes fully enacted.

Notes

1. To see the text of each item discussed in this section, see the survey in Appendix 2.
2. The dissertations reviewed involved surveys administered to legislators using a variety of methodologies and across many topics. No surveys were found detailing efforts at cross-state surveying of legislators, so that all rates cited are relevant to policymakers surveyed by a researcher at an institution within their states. The review was not exhaustive, and conducted to generate response rate figures for discussion within the context of this section only.

Acknowledgment

The work presented in this paper was taken from research completed for the degree of Ph.D. awarded in the Fall semester, 2001 from American University, Washington, D.C. The opinions expressed in the article are those of the author and not any institutions and not of any institution with which he is affiliated.

American Statistical Association. (1998). More about mail surveys. ASA Series: What is a survey. Alexandria, VA: Author.

Cobb, C. D. & Glass, G. V. (1999). Ethnic segregation in Arizona charter schools. Education Policy Analysis Archives (7)1. January 14, 1999. Retrieved October 21, 2002 from http://epaa.asu.edu/epaa/v7n1/

Center for Education Reform (2000). What the research reveals about charter schools. Washington, DC: Author. Retrieved October 23, 2002 from http://www.edreform.com/pubs/charters.htm

Coons, J. & Sugarman, S. (1999). Making school choice work for all families. San Francisco, CA: Pacific Research Institute. [On-line] Available: http://www.pacificresearch.org/edu/choice/choice.html (October 14, 1999).

Cremin, L. A. (1964). The transformation of the school: Progressivism in American education 1876 - 1957. New York: Vintage Books.

Creswell, J. W. (1994) Research design: Qualitative and quantitative approaches. Sage Publications,  Thousand Oaks, CA.

Devore, J. & Peck, R. (1997). Statistics: The exploration and analysis of data (3^rd ed.). Pacific Grove, CA: Duxbury Press.

Fox, R. J., Crask, M. R., & Kim, J. (1988). Mail survey response rate: A meta-analysis of selected techniques for inducing response. Public Opinion Quarterly. 467 - 491.

Gehring, J. (2002). Voucher battles head to state capitals. Education Week 21(42) 1,24,25.

Ruth D. Holmes et al., v. John Ellis "Jeb" Bush et al. CV 99-3370 (Leon County Circuit Court, 2002). Retrieved August 13, 2002 from http://199.44.225.4/courtDockets/pdf/CV993370/99-3370-51.pdf

Institute for Justice (2002). Advancing school choice. Retrieved October 21, 2002 from http://www.ij.org/cases/school/.

Jones, W. H. (1979). Generalizing mail survey inducement methods: Population interactions with anonymity and sponsorship. Public Opinion Quarterly. pp. 102 - 111.

Majchrzak, A. (1984). Methods for policy research. Applied social research methods Series (Vol. 3). Newbury Park, CA: Sage Publications.

Martella, R. C., Nelson, R. & Marchand-Martella, N. E. (1999). Research methods: Learning to become a critical research consumer. Needham Heights, MA: Allyn & Bacon.

Neuman, W. L. (1997). Social Research Methods: Qualitative and Quantitative Approaches. Needham Heights, MA: Allyn and Bacon. p. 243.

Pritchard, I. (1999). Reducing class size: What do we know? Washington, DC: National Institute of Student Achievement, Curriculum and Assessment: Office of Education Research and Improvement, U. S. Department of Education. Retrieved October 23, 2002 from http://www.ed.gov/pubs/ReducingClass/

Ravitch, D. (1983). The troubled crusade: American education 1945 - 1980. NY: Harper Collins.

Rose, L. C. & Gallup, A. M. (2000). The 32^nd annual Phi Delta Kappa/Gallup poll of the public's attitudes toward the public schools. Bloomington, IN:  Phi Delta Kappa International.

Sanders, W. L. & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville: University of Tennessee Value-Added Research and Assessment Center.

Shokraii-Rees, N (2000). School choice: What's happening in the states. Washington, DC: The Heritage Foundation.

Tedin K. L., & Hofstetter, C. R. (1982). The effect of cost and importance factors on the return rate for single and multiple mailings. Public Opinion Quarterly. pp. 122 - 128.

Toppo, G. (2002, Aug. 6). Voucher backers put forth bills. Associated Press.

Tyack, D. B. (1996). The one best system. Cambridge, MA: Harvard University Press.

Yammarino, F. J., Skinner, S. J., & Childers, T. L. (1991) Understanding mail survey response behavior: A meta-analysis. Public Opinion Quarterly. 613 - 639.

Zelman, Superintendent of Public Instruction of Ohio, et al. v. Simmons-Harris et al. 536 U. S. 00-1751 (2002).  Retrieved August 13, 2002 from http://www.supremecourtus.gov/opinions/01pdf/00-1751.pdf.

About the Author

Dan Laitsch
Senior Policy Analyst
Association for Supervision and Curriculum Development
1703 Beauregard St.
Alexandria, VA 22311-1714
(703) 575-5628
dlaitsch@ascd.org

Dan Laitsch is Senior Policy Analyst for the Association for Supervision and Curriculum Development. He was previously Associate Director of the State Issues Clearinghouse at the American Association of Colleges for Teacher Education, and has taught high school English at a number of alternative schools in Fairfax County, including six years with the Enterprise School in Vienna, VA. He has co-taught middle school English classes in Japan as part of the Japan Exchange Teacher Program. He holds BA in English from Virginia Polytechnic Institute and State University, an M.Ed. in    Social Foundations of Education from the University of Virginia, and a Ph.D. in Education Policy from American University in Washington, DC. This paper is based on the work of his dissertation research. Laitsch's current interests include teacher quality; research use in, and impact on, education policy; school effectiveness; and public school reform issues.

Appendix I: Methodology

Overview

In choosing an approach to take to answer the identified research questions, both quantitative and qualitative approaches were examined. While the history of voucher programs in Ohio, Florida, and Wisconsin is both important, worthy of study, and could have contributed to identifying arguments used effectively in those states, the ability to examine policy maker reactions to current issues surrounding voucher proposals, both in states with such programs, and in states where the programs have been a hotly debated issue, presented a unique research opportunity. For this reason, a quantitative methodology that allowed for wide spread application and comparative analysis, was sought. A survey of legislators was chosen as the best methodology for answering the research questions across a wide geographic range and large body of potential data sources.

Site Selection

In choosing the states to examine in this study, two related techniques were used: critical case sampling, and politically important case sampling. Sampling is often done randomly, but can also be collected purposefully; that is with the goal of selecting cases rich in information critical to the research questions. The focal points of this study (Florida, Michigan, New Mexico, Ohio, Pennsylvania, and Wisconsin) were chosen using both critical case sampling and politically important case sampling. Critical case sampling is used to select cases that are of interest for a particular reason while politically important cases are selected because either the site or the topic (or both) is of political importance (Martella, Nelson & Marchand-Martell, 1999). Specifically, Ohio and Wisconsin were determined to be critical because they are the only two settings with established voucher programs. Florida was chosen because it is the first state to enact a state-wide voucher program, while Michigan, New Mexico, and Pennsylvania, were all considered for inclusion in the sample because of the prominence of the voucher debate within those states (voters in Michigan recently considered vouchers through a ballot initiative, while the governors in New Mexico and Pennsylvania have strongly advocated for voucher systems in their states).

Data Sources

In this study, the primary methods for collection of data were through administration of a researcher prepared survey (using a self-administered survey) as well as a review of relevant documents. Using these techniques data related to vouchers and education reform was collected from policy makers, while historical data was collected to help provide context. These methods for the collection of data are consistent with survey research, as well as historical research designs and policy analysis (Majchrzak, 1984; Martella, et al., 1999; Neuman, 1997). An added benefit of the historical data collection was the ability to partly synthesize the data as the study evolved, allowing for more focused research as information was processed. Because of the ability to synthesize data in this manner, the focus of the survey was expanded to include components designed to evaluate policy maker opinions regarding the purpose of education as well as the role of vouchers within the larger context of reform.

Policy Participants

Expert observers active in the voucher debate were contacted both to help establish the policy atmosphere in which vouchers were enacted, and to help validate the survey. The people contacted included university scholars; members of national think-tanks, education organizations and associations; and members of unions and other advocacy groups. Individuals were contacted at pro-voucher groups including the Center for Education Reform and the Institute for Justice; groups opposed to voucher programs such as the American Association of School Administrators, and the National School Boards Association; and neutral groups, including the National Conference of State Legislatures, the American Association of Colleges for Teacher Education, and the Center for Education Policy . Faculty at American University also assisted in construction and validation of the survey.

Survey

As the primary data collection tool for this research, great care was taken in development of the survey instrument, however, some difficulties arose during the application procedure.

Surveys are typically used to identify how people feel, think, act, or vote (Martella, et al., 1999), which fits well with research designed to gauge how policy makers react (feel) to specific pro- and anti- voucher statements. The advantages attributed to survey research include comparative cost effectiveness, the ability to target specific populations, and the tendency of respondents to feel their confidentiality is protected (American Statistical Association, 1998).  Although associated difficulties include deciding contact methods, controlling costs, setting deadlines, protecting fairness, and performing proper follow-up, a survey methodology was determined to be the best way to obtain the data needed to answer the research questions.

Items and Content

Because the debate about vouchers is often one polarized to extremes, a methodology that would force respondents away from the extremes was developed. The survey was divided into three sections. The first section examined school reform issues. The second looked at the voucher statements, and the third section was concerned with demographic data.

Section I: Reform issues

The issue of vouchers cannot easily be separated from the concept of education reform and the general role of education. In addition, the perception of the success of our current system in meeting that role needed to be examined. Section I of the survey briefly examined all three of these issues.

Because the need for reform is based on the perceived effectiveness of the education system, that is, an effective system would not need reform while an ineffective system may need radical reform, legislators were asked to evaluate the current system's success. Before such an evaluation could take place, however, "what" is being evaluated needed to be defined. To accomplish this, legislators were be asked to rank order six possible "purposes" of education, along with the option to add a seventh purpose if they desired. Next, respondents were asked to evaluate the degree of effectiveness of the current system, both locally and nationally in addressing the purpose they previously identified. The next item asked policy makers to indicate their level of support for 11 specific reforms, vouchers included.

The final question in this section looked at support for vouchers on the continuum of voucher programs; from an unrestricted voucher system, to systems regulated by income or school effectiveness, to no voucher system at all.

Section II: Voucher positions

Section II asked respondents to rank order pro- and anti- voucher statements. The statements were divided along three tension lines found within the education system: excellence, access and equity, and structure. These statements were designed to evaluate the type of impact respondents felt vouchers would have on the current education system.

Section III: Demographic data

Demographic data that has been previously related to support or opposition to voucher programs in other studies (race, religion, gender and educational experiences) was included. Where possible the items in this section were modeled on other demographic surveys and data sources.

Development

The survey was constructed in four phases. In the initial phase, a draft instrument was constructed by reviewing the research and advocacy literature related to voucher programs, as well as literature related to survey construction. This initial survey contained many of the policy arguments included in the final survey (as well as the demographic questions), however the rating scale used was a Likert scale. It was in this phase that the primary policy arguments were identified and the arrangement of item groupings first noticed.

Use of the Likert scale was dropped for the policy arguments during phase two primarily to avoid any response set bias (Neuman, 1997). It was also felt that due to the extreme positions held by many individuals, the Likert scale might not elicit the kind of data that would allow for detailed differentiation between item responses. In other words, people already holding a position would recognize the various pro- and anti- voucher statements and "strongly agree" with all items interpreted as supporting their position, while "strongly disagreeing" with all items interpreted as being against their beliefs. In such a situation, the middle ground, or swing arguments, would be lost. The survey was instead reorganized to encourage differentiation between items by asking respondents to rank-order related items. In this way, the strongest and weakest arguments for each grouping could be analyzed. Since the arguments were organized into pro- and anti- voucher sections of the survey, analysis of arguments with cross-position appeal could also be made. It was during this phase that similar subgroups of the policy arguments were identified within each position. The subgroups are: access, equity, and educational structures. In other words, the policy arguments could be aligned by evaluating the impact of vouchers (as perceived by policy makers) on: access to education; equity of educational opportunity; and their potential impact on current educational structures. Since the arguments were already separated into pro- and anti- voucher groupings, the result was six final categories, allowing for the evaluation of up to 36 policy arguments (33 were ultimately identified).

In the third phase, the scope of the survey was expanded to include items designed to gauge policy maker reaction to other reform options, as well as the purpose and effectiveness of public education. This was done to help provide some context to the data interpretations. While identifying which arguments seem most effective could prove useful in a limited way, placing those conclusions within the broader context of reform and educational philosophy provided a richer understanding of the data. A Likert scale was used in this part of the survey, since policy makers where being asked to guage the degree of effectiveness of the schools generally, and various reforms specifically. A rank order response was used for the section related to the purpose of education.

Finally, the survey was sent to experts in the field of vouchers and school reform to assist with instrument validation. Despite the positive feedback from evaluators, difficulties experienced during this phase of development significantly effected the process involved in finalizing the survey. While the general instrument was completed and disseminated to reviewers in early March of 2000, some reviewers took up to five months to review the document. This significant delay in the review process ultimately precluded a formal pilot test.

The final step in the typical development of a survey is the pilot-test. This survey was not formally pilot-tested for two critical reasons. First, a representative target audience for a pilot test was not readily available. While a legislative body in a local state could have served as the target population, gaining detailed feedback from the audience regarding the survey would have been difficult and time consuming (and such information had already been obtained through the expert reviewers, who were asked to first take the survey themselves and then provide feedback as to the validity of the survey content, application and structure).

A second, and more perplexing difficulty with completing a pilot-test arose from the delay caused by the unexpectedly long review process. The addresses and electronic contact information for the sample to be surveyed were provided by StateNet, an organization that supplies legislative research resources to private clients. Due to the 2000 elections, this source data was only accurate up until November 8 (election day). Contact with legislators after that deadline would be effected by changes in the sample brought about by the elections (this was the case with a second follow-up e-mail sent after the election day). An extensive pilot-test was therefore precluded because of the need to apply the survey before the target population changed.

Application

Because the target audience for the survey was defined as legislators in the selected study states, the entire population was available to be surveyed. While a random, or a stratified sample of legislators was considered, the potential benefits of targeting the entire population were viewed as outweighing the utility of a smaller sample. The total number of participants surveyed was 936.

The survey was applied according to the recommendations of the American Statistical Association (1998), which parallel other application recommendations (Creswell, 1994). A cover letter was included with the survey, as well as a pre-addressed stamped return envelope. Follow-up contact was attempted two weeks later, although the follow up procedure differed from the initial contact in that it focused on electronic contact. E-mail messages were sent to each legislator available, totaling 780 (approximately 715 were successfully delivered). The follow-up message asked legislators to return the hard copy sent to them, or fill in an on-line version of the instrument.

Two areas where the application procedure deviated from ASA recommendations include a failure to send a pre-contact notice and failure to offer an incentive for completion of the survey. These guidelines were not followed since legislative offices are generally very careful with the mail they receive, so that while a survey may not have be returned, it was felt that it most likely would be opened and reviewed. An incentive was viewed as inappropriate for this audience and for this type of research.

Validity

There are three types of validity typically used to establish the appropriateness of a measurement device: construct, content, and criterion. Education activists and professionals were contacted to help establish the face validity of the survey (content validity). While face validity is not the strongest tool for establishing validity, it was one of the only tools available in this instance. Because this was the first administration of this device, construct validity could not be established (Martella, et al., 1999; Neuman, 1997). In addition, while individual items were constructed based on other measures of support for vouchers, no similar measure existed for the device as a whole (criterion validity).

Data Analysis

The data obtained from the survey was analyzed individually by item and section, as well as summatively across sections and items. In addition to basic analyses (mean, median, mode, standard deviation, and variance, as appropriate), a correlational analysis was conducted examining the relationships between demographic variables and each section independently, as well as the relationships between support for or opposition to vouchers and policy maker opinion of school purpose and effectiveness. A comparison of reactions to policy statements in states with voucher programs and states without such programs, was also conducted.

The data analysis did not attempt to establish any causal relationships since survey data cannot generally be used in that manner (Martella, et al., 1999). In addition, because the return rate was relatively small, the available data was limited. Therefore, correlational data involving demographic characteristics was not attempted at the state and local levels. A final impact on the complexity of the data analysis stems from the nature of the data, which in much of the survey is categorical (qualitative). While categorical data does not preclude analysis of the data using a multiple regression model, it does significantly expand the predictors used in each model and the model complexity (Devore & Peck, 1997).

Data Collection

On October 18, 2000, the survey was mailed to 936 state legislators in Florida, Michigan, New Mexico, Ohio, Pennsylvania, and Wisconsin. Policy makers were asked to respond by October 27, so that follow-up contact could be made before the November 8 elections. A follow-up contact was made via e-mail on November 5 asking policy makers to return the survey or complete an on-line version. Finally, on December 12, the last reminder e-mail was sent notifying legislators that the survey would remain on-line until the end of January, 2001.

Survey Completion and Non-completion

Of the 936 surveys mailed, 89 were returned, for an overall return rate of 9.5%. Of the items returned, 20 included sections only partially completed, and four were filled out incorrectly and therefore excluded from analysis. While the majority of returns were through the mail, 14 surveys were returned on-line, accounting for 16% of all surveys returned.

Completion Rate Figures

The completion rate of viable surveys was 9.21% (see Table 1). Twenty incomplete surveys were included in the return estimate because they were completed correctly and contained useful data for most sections of the survey. Those items that were left blank or were completed incorrectly were not included in any analyses that examined the compromised sections. In all cases included in the estimate, Section 1 was completed correctly, while some incomplete data was present in Section 2. Demographic data was also completed accurately by all but two respondents. Items determined as "unusable" were those that were filled out incorrectly throughout, or had too little data to be included in any of the analyses (one survey had only a note at the top "I oppose all voucher plans").

Sampling Error

By attempting to survey every legislator in the study states and maintain the focus of any conclusions on those states, it was hoped that the potential for sampling error could be minimized, however, because of the low response rate, a significant potential for error exists. In addition, while the demographic traits of respondents are similar to the body of legislative policy makers in each state, some traits were clearly skewed (for example, two thirds of respondents said that they have served on the committee responsible for education). While these legislators clearly have an interest in education, service on the education committee is not a common characteristic of the broader body of legislators. It can be argued, however, that data gathered from legislators with experience on the education committee lends to the significance of the survey results by providing insight into the responses of legislators who are or have been in key leadership positions on education issues.

In addition to the returns, 17 letters or e-mails were received from legislative offices in three of the study states. Many of the letters explained that the legislator had a blanket policy of not returning surveys because of the large number of such contacts they receive. One legislative office offered the opportunity for a face-to-face interview; however, the legislator was not interested in completing a phone interview or mail survey. Similar automatic responses to the e-mail contacts suggest that legislators in general were not disposed to respond to surveys from non-constituents.

Other replies referred to problems with the timing of the survey. One response noted: "It's the day before electiion [sic] day.  All I can tell you is that I'm vehemently opposed to vouchers.  I'm a Republican" (personal communication, November 7, 2000), while another stated, "Take me off of your list. I have been term limited out and yesterday was the last" (personal communication, December 14, 2000). Also complicating survey returns was the electoral controversy in Florida. Most e-mails sent to the legislators in Florida were met with the following auto-response: "Thank you for contacting me about Florida's recount effort.  Because I have received thousands of calls, letters and e-mails it is difficult to respond to each communication individually."

One return from a legislator in Pennsylvania managed to capture the controversy in that state:

Why would you presume that every legislator in six states is interested in your research, especially when it requires them to answer several pages of inane questions about a subject which most of them would like to forget . . . Further an enterprising doctoral student might figure out that there are a variety of groups on both sides of the issue that ask candidates to respond to questionnaires during the heat of the campaign which might prove far more interesting than some academic exercise" (personal communication, December 14, 2000).

Data Analysis

In evaluating the results of this survey, a brief discussion of the scale used is important. Most non-demographic items were scored from 1 - 6 on either a Likert scale or as a rank order. In either case, the highest rating an item could receive was a "1", while a "6" was generally the lowest rating possible. The mean for each item was calculated and used to help determine the favorableness of each item. While by itself an average of 2.1 wouldn't tell us anything, especially for the rank order items (i.e.., there is no rank of 2.1), it does give us a sense as to how positive an individual item is seen to be. The variance of each item was also calculated so that a rough picture of the spread of answers within each item could be constructed. Where required for analysis, a .05 level of significance was used.

Appendix 2: Survey application and return rate

Survey application and design

To understand the low response rate to this survey, the literature around survey responses was revisited. While the research reviewed supported the summary research presented earlier, examination into the theory behind why specific techniques appear to improve return rates, and consideration of the impact such theory might have on policymakers suggests some possible explanations for the low return rate on this survey. Because this research was not structured to specifically examine responses rate effects, conclusions reached in this section should be viewed as hypothetical.

One critical gap noted in the research was an examination of the response rates of policymakers to surveys. Much of the survey research reviewed appears to have been conducted by researchers interested in marketing and surveys applied to the general public. No research could be found that dealt specifically with the impact of response rate strategies on policymaker return rates. Consequently, in the design of this research it was assumed that strategies successful in increasing the response rate of the general population would also be successful with policymakers. However, when looking at the theory behind the specific strategies, weaknesses in the potential impact of their effect on policymakers become apparent. While this is not to say that the return rate strategies used had no impact, it is possible to see how such impact might be minimized.

Research into survey response rates has generated a number of application strategies that seem to be successful across the board (American Statistical Association, 1998; Creswell, 1994; Fox, Crask, & Kim, 1988; Jones, 1979; Tedin & Hofstetter 1982; Yammarino, Skinner & Childers, 1991). These include the use of first class postage, pre-contacts, follow-up contacts, personalized addresses and salutations, use of cover letters, signed cover letters, stamped return envelops; guarantees of anonymity; and monetary incentives. The effect of non-application manipulations are less clear, but include survey length and complexity, page color, topic importance, sponsorship, and type of population. Of these topics, sponsorship, especially sponsorship by a university, has been found to effectively increase response rates, as has the importance of the topic.

While much of the research examined focused on the effects of the intervention, Tedin and Hofstetter also considered some of the theory behind these interventions (1982).  They point out that interventions the investigator generally can control relate to survey importance factors and cost factors. In effect, interventions seen by respondents as emphasizing the importance of their responses (repeated contact, personal letters, first-class postage, monetary incentives, etc.), and interventions lowering participation "costs" (time and effort it takes to respond - length of questionnaire, enclosed and stamped return envelop, etc.), generally increase the response rates.

The response rate interventions used on this survey include the following: inclusion of a cover letter (including a personalized address and salutation and personalized signature), inclusion of a stamped return envelop, use of a short questionnaire (four pages), multiple contacts (in paper and electronic formats), consideration of topic importance, and a guarantee of anonymity. While such interventions may reduce costs and increase the perceived importance of responding for members of the general public, when examining the way in which legislative offices operate, such considerations may be less likely to have a strong impact.

Cost effects

Of the interventions identified above, two can be categorized as efforts to reduce respondent costs: inclusion of a stamped return envelop, and use of a short questionnaire. While the impact of a short questionnaire on responses rates has been difficult to determine (Fox, Crask, & Kim 1988; Tedin & Hofstetter, 1982), generally the use of first class postage has been shown to increase response rates; however, the volume of contacts legislators receive requires legislative offices to utilizing procedures which may mitigate this effect. For example, since policymakers receive a variety of postal benefits (franking privileges), the monetary cost saving of a stamp and envelop are negligible. In addition, because legislative offices are organized to respond to constituent contacts, the cost reduction in the effort to return the survey offered by inclusion of a stamped envelope may also be reduced.

Importance

The other four interventions- use of personalized addresses and cover letters, multiple contacts (in paper and electronic formats), consideration of topic importance, and a guarantee of anonymity-can be categorized as effecting the respondents perception of survey importance (the perception of respondents that it is important that the survey be returned). Personalization emphasizes that the individual's opinion is of particular importance, as does the use of multiple contacts. The topic itself, education reform, was also framed so as to emphasize the importance of the survey. Finally, the guarantee of anonymity adds to perceptions of importance by recognizing the issue as controversial and one in which an open, honest evaluation, free of constraints (in this case those of political party) is critical. Although each of these interventions is validated in survey research, the special circumstances involved in contacting legislators may reduce their impact on return rates.

While it may be unusual for a random sample of the general public to receive a personalized survey (thereby emphasizing its importance), legislators expect such personalized contact as routine. Contact advice posted on at least one of the state web sites confirmed this expectation. Perhaps even more problematic is the expectation that such contacts originate from or impact upon the legislator's constituents. The automatic responses received to the e-mail contacts highlighted this point, some specifically stating that the legislator only reviewed in-district e-mail. This reluctance to work with individuals who the policy maker does not represent, and who are not at least partly responsible for his or her election, may have kept response rates low, despite the interventions used.

Additionally, because of the large number of contacts policymakers receive, many offices have constructed pre-formatted replies related to specific issues. Although such replies are not directly applicable in this case (other than those stating that the policy maker does not respond to any surveys), it suggests that while a policymaker's response is important, individualization (such as required by surveys) is not. In the application of this survey, no steps were taken to associate the survey with the policymaker's constituent groups. In fact, neither the researcher, nor the researcher's institution were located within any of the study states.

Finally the timing of the survey application may have worked to minimize the attention paid to it. For reasons discussed earlier, the survey was administered shortly before the November 2000 elections, and follow-up contact was made during the post-election controversy over election returns in many states, most notably Florida. Because of the timing then, the survey was competing for the attention of policymakers with both election campaigning and post-election controversies. As one politician, cited earlier, noted:

.... there are a variety of groups on both sides of the issue that ask candidates to respond to questionnaires during the heat of the campaign which might prove far more interesting than some academic exercise" (personal communication, December 14, 2000).

Two other previously noted comments also highlight the poor timing of the survey: "It's the day before electiion [sic] day.  All I can tell you is that I'm vehemently opposed to vouchers.  I'm a Republican" (personal communication, November 7, 2000); and, "Take me off of your list. I have been term limited out and yesterday was the last" (personal communication, December 14, 2000). The e-mail responses from Florida also highlight the lack of immediate importance the survey may have had for many legislators ("Thank you for contacting me about Florida's recount effort. Because I have received thousands of calls, letters and e-mails it is difficult to respond to each communication individually."). In light of the 2000 election controversies, the survey completion rate may have been negatively impacted.

Population identification and research instrumentation

Changes in the methodology used to identify the target audience and application procedure used for this instrument might have resulted in a higher rate of return. For example, a smaller, stratified sample, would have permitted more focused efforts to obtain responses. A smaller sample drawn from the same states (or even fewer states), could have enabled the use of letter and phone pre- and post- contacts, and might even have allowed the survey to be administered in an interview format. Additionally, use of a stratified sample would allow use of a pre-contact strategy designed to elicit a commitment to participation by individual policymakers. Such a commitment could then allow the researcher to adjust of the actual survey application as a projected response rate is generated. These extensive efforts may then have been seen by policymakers to stress the importance of participation in the survey, increasing response rates.

While such efforts may well have increased the rate of return, manipulations of the sample might also have concealed some findings that would only emerge in a survey of the entire population (for example, the high interest level of education committee members as evidenced by their high rate of return), and greatly increasing the complexity of survey design, application, and analysis. Despite such complexities, however, increasing response rates in this case would have strengthened the research findings.

Recommendations for future research

Because policymakers operate under different expectations regarding mail contacts than the general public, it is possible that techniques traditionally effective in increasing survey response rates are not as effective with policymakers. In addition, because of situational complexities (most notably the application of the survey during the closing days of a general election), events external to the research design may have lowered response rates. There are, however, additional steps that might be taken to increase response rates of future surveys. These recommendations serve to emphasize the importance of the survey to policymakers.

1. Since sponsored surveys often receive a higher response rate, use of university letterhead on the cover letter may generate a higher rate of return. It should be noted that the National Council of State Legislatures was asked to sponsor this survey but due to the controversial nature of the topic they declined.
2. Because policymakers are more likely to consider the concerns of their constituents, future survey efforts might benefit from the use of proxy administrators located in each state or within institutions in each state. By seeking to generate the data from within each legislator's home state, greater attention may be attached to completion of the survey.
3. The survey should be administered just after the end of each state's legislative session, when legislators do not face as many demands on their time.
4. The survey should not be administered during an election year, or, if timing demands such application, the survey should be administered prior to election day and preferably before the summer campaign season.
5. A procedure should be established to allow pre-contact, as well as extensive post contact. A useful procedure might include a pre-contact letter of introduction, followed by a pre-contact phone call to attempt to get buy in by the legislator. After application of the initial survey (either in mail, interview, or Internet format), a post contact follow-up should be made either to thank the legislator for their cooperation, or to encourage those legislators not responding to complete the survey.
6. Consideration of the survey format could also be valuable. Changing from a mail-administered questionnaire to an interview format could greatly increase response rates, since mail surveys are used specifically to target a wide audience. Narrowing the sample through a change from questionnaire format to an interview format be worth the loss of breadth if,  by allowing targeted and frequent pre-contact and follow up, it raises the response rate.
7. If expense precludes the use of these strategies for multi-state or state-wide research, then researchers may want to consider using sampling techniques to narrow the population to be surveyed. Of the sampling choices available, stratified sampling could field a list of legislators within each strata who might be contacted until sufficient numbers for detailed analysis are generated. Additionally, because the data generated will be categorical in nature, stratified sampling could ensure that there will be enough legislators in each area of interest identified, so that the appropriate statistical examinations can be completed.

While these seven recommendations may not guarantee a higher rate of return, it is unlikely, given the low rate of return on this survey, that they would negatively effect return rates. It also seems likely that a survey administered under similar circumstances will result in a low rate of return.

General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu .