Performance Standards : Utility for Different Uses of Assessments

Performance standards are arguably one of the most controversial topics in educational measurement. There are uses of assessments such as licensure and certification where performance standards are essential. There are many other uses, however, where performance standards have been mandated or become the preferred method of reporting assessment results where the standards are not essential to the use. Distinctions between essential and nonessential uses of performance standards are discussed. It is argued that the insistence on reporting in terms of performance standards in situations where they are not essential has been more harmful than helpful. Variability in the definitions of proficient academic achievement by states for purposes of the No Child Left Behind Act of 2001 is discussed and it

is argued that the variability is so great that characterizing achievement is meaningless.Illustrations of the great uncertainty in standards are provided.
Measurement specialists generally conceive of achievement as a continuum and they prefer to work with scale scores with many gradations rather than with a small number of categorical scores.It is recognized that there are a number of purposes for which the scores need to be lumped into a small number of categories that require the identification of one or more cut scores.Some leading measurement specialists, however, have suggested that it is best to avoid setting performance standards and associated cut scores if possible.For example, Shepard (1979) advised that it is best to "avoid setting standards whenever possible" (p.67) and Green (981) concluded that "fixed cutting scores should be avoided whenever possible" (Green, 1981(Green, , p. 1005)).
There obviously are some purposes where the identification of one or more cut scores cannot be avoided because they are essential to the use of a test.Tests used to make licensure and certification decisions, must have a cut score identified that will collapse the score scale into only two categories -pass and fail.In other situations, the scores are collapsed into 4 or 5 categories.The College Board, for example, converts a weighted combination of scores on the multiple-choice and the constructed-response sections of their Advanced Placement (AP) Examinations to a final grade that is reported on a five-point scale: 5 -extremely well qualified 4 -well qualified 3 -qualified 2 -possibly qualified 1 -no recommendation.
The pass-fail dichotomy is required for the decision to be made in the case of a licensure or certification test.The use of 5 categories on AP examinations, on the other hand, also supports dichotomous decisions about whether or not a student will receive college credit based on his or her AP grade, but allows colleges and universities to determine the grade required to be awarded credit.
Certification tests and AP examinations are just two of many situations where the primary use of test scores is to determine whether the test taker has met a performance standard.The performance of the person who barely met the standard is more similar to that of the person who barely failed to meet the standard than to that of someone who exceeded the standard by a comfortable margin.Indeed, due to measurement error, there is a substantial probability that the person who barely met the standard may be a false positive while the person who barely failed may be a false negative.The two could easily switch places if they took an alternate form of the test.Although such classification errors are of concern and should be minimized to the extent possible, they cannot be avoided altogether and there are legitimate practical reasons that require that a decision be made.Thus, Shepard's (1979) and Green's (1981) advice to avoid the use of a fixed standard or cut score cannot always be followed because standards are an essential element of the use of the test results.In the last 10 or 15 years, however, performance standards have been mandated and used with increasing frequency in situations where they are not essential to the use being made of test results.

Nonessential Uses of Performance Standards
Performance standards began to be introduced in uses of tests where they were not really essential as the result of the criterion-referenced testing movement that was spawned by Glaser's (1963) classic article.Ironically, Glaser's conceptualization of criterion-referenced measurement did not require the establishment of a fixed standard or cut score, but the use of cut scores to determine that a student either did or did not meet a performance standard became associated with criterion-referenced tests (Hambleton & Rogers, 1991;Linn, 1994).Glaser's later discussions of criterion-referenced testing (e.g.Glaser & Nitko, 1971) recognized the use of criterion in the sense of a fixed standard noting that "[a] second prevalent interpretation of the term criterion in achievement measurement concerns the imposition of an acceptable score magnitude as an index of achievement" (p.653).Nonetheless, the setting of a fixed standard or cut score is not an essential to criterion-referenced measurement.
There are a number of reasons to question the wisdom of setting a performance standard for a test if the standard is not essential to the use of the test results.Green's (1981) desire to avoid setting fixed performance standards whenever possible was based on the recognition that a single item provides very little information by itself but "one item may well make the difference between passing and failing" (p.1005).Others who have been critical of the use of performance standards have focused on limitations of the standards.Based on his review of standard setting methods, Glass (1978), for example, concluded that standards setters "cannot determine 'criterion levels' or standards other than arbitrarily.The consequences of arbitrary decisions are so varied that it is necessary either to reduce the arbitrariness, and hence the unpredictability or to abandon the search for criterion levels altogether in favor of ways of using test data that are less arbitrary, and hence safer" (p.237).Although respondents to Glass's article noted that although standards are arbitrary in the sense that they are set judgmentally, they need not be capricious (Hambleton, 1978;Popham, 1978).
For a licensure test there is a clear context for standard setters who have a responsibility for thinking about the minimal skills required to protect the public from incompetent practitioners.The broader good of requiring some minimal level of performance on a test to be certified and therefore allowed to practice justifies the judgmentally established cut score.Moreover, the need to make certification decisions provides judges with a clear context for setting the standard.Standard setters may also have a clear sense of the proportion of candidates who have passed the examination in the past.Similarly, for an AP test there is a clear link between college grades assigned in courses for which credit may be awarded.Judgments regarding minimal acceptable performance when standards are set on tests that must be passed to graduate from high school or for promotion to the next grade have a more generalized context than the one that applies to setting standards for licensure examinations.The need to establish a cut score is essential to the use of a test as a high school graduation requirement and at least some of the consequences of the use are known to the standard setters who can weigh the potential benefits (e.g., restoring the meaning of a high school diploma or motivating greater effort by teachers and students, Mehrens & Cizek, 2001, p. 480) against the potential negative consequences (e.g., that the minimums will become the maximum or that more students will drop out of school, Mehrens & Cizek, 2001, p. 481).
In many instances the standards that are set are not used to make any pre-specified decisions about individual students.Instead the performance standards are used for reporting the performance of groups of students and for tracking progress of achievement for schools, states, or the nation.Examples include the setting of performance standards for the National Assessment of Educational Progress (NAEP) or a state assessment.The context that standard setters have had for setting a cut score to determine proficient, basic, or advanced performance for NAEP or a state assessment has, until recently, lacked any clear context other than a sense of an aspiration for high levels of student achievement.

Standards and Aspirations
Six broad educational goals, two of which concerned student achievement, were agreed to by Governors and the President at the Education Summit held in Charlottesville, Virginia in 1989.The National Educational Goals Panel was created and given the responsibility of monitoring progress toward the goals set at the Education Summit.Five years after the Charlottesville summit, the Goals 2000: Educate America Act of 1994 was signed into law .Goals 2000 encouraged standards-based reporting of student achievement.As defined by a technical planning group for the Goals Panel, "performance standards specify 'how good is good enough'" (National Educational Goals Panel, 1993, p. 22).Unanswered is the question: good enough for what?It is clear that the performance standards are expected to be absolute rather than normative.(Although normative comparisons have been eschewed by many proponents of criterion-referenced tests and standards-based assessments, norms have considerable utility for providing comparisons of relative performance across content areas and even for a single measure are often more readily interpreted than are criterion-referenced reports or reports of results in terms of performance standards (see, for example, Hoover, 2003).)In keeping with the zeitgeist of the time, it is also clear that they were expected to be set at "world class" levels.
The high aspirations of 1989 Education Summit which were encouraged by Goals 2000 and the Goals Panel provided the primary context for the setting of standards on NAEP and on many state assessments.As might have been expected the performance standards set on NAEP on a number of state assessments were set at high levels.In support of the judgment that the performance standards were set at ambitious levels consider the fact that in 1990, the first year that NAEP results were reported in terms of achievement levels, the proficient level on the mathematics assessment corresponded to the 87th percentile for 4th grade students, the 85th percentile for 8th grade students, and 88th percentile for 12th grade students (see, for example, Braswell, Lutkus, Grigg, Santapau, Tay-Lim & Johnson, 2001).Linkages of NAEP to the Third International Mathematics and Science Study (TIMSS) grade 8 mathematics results which revealed that no country is anywhere close to having all of their students scoring at the proficient level or higher (Linn, 2000) also attest to conclusion that the NAEP achievement levels are set at very ambitious levels.
Performance standards set by a number of states for their state assessments during the past decade have also been set at quite high levels in many cases.The high levels that were set by many states are evident from the linkage of state assessments to NAEP (e.g., McLaughlin & Bandeira de Mello; 2002, see also Linn, in press, for a discussion of the McLaughlin & Bandiera de Mello results).It is not unusual for a state to have the proficient performance standard set at the 70th percentile or even higher.Since performance standards on NAEP and on a number of state assessments have, in most cases in the past, had no real consequences for students and there is no requirement for actually achieving the aspiration of having all students at the proficient level or above, one might conclude that there is no harm in having high standards.There may be reason for concern that reports that, say, less than half the students are proficient may paint an unduly negative picture for the public, but many would argue that it is good to have ambitious goals even if they are never achieved."If you reach for the stars, you may not quite get them, but you won't come up with a handful of mud either" (Samuel Butler, as quoted by Applewhite, Evans, and Frothingham, 1992, p. 22).
It is one thing to set performance standards at the height of stars when there are no requirements of achieving them, but it is another matter altogether when there are serious sanctions for falling short such as those that have recently been put in place by the No Child Left Behind (NCLB) Act of 2001 (Public Law 107-110).NCLB sets the goal of having all children at the proficient level or higher in both reading/language arts and mathematics by 2013-2014.It also requires schools, districts, and states to meet adequate yearly progress (AYP) targets in intermediate years that would assure that they are track to having 100% of the students at the proficient level or above by 2013-2014.There are severe sanctions for schools that fall short of AYP targets for two or more years in a row.
The percentage of students who are at the proficient level or above for purposes of NCLB is to be determined by state assessments using performance standards established by each state."Each State shall demonstrate that the State has adopted challenging academic content standards and challenging student academic achievement standards that will be used by the State" (P.L. 107-110, Section 1111(b)(1)(A)).
All states had to submit plans explaining how they were going to meet the accountability requirements of NCLB by January 31, 2003.States that were in process of introducing new assessments or that had not yet set performance standards will be setting standards in quite a different context than existed prior to the enactment of NCLB.In light of the new context provided by NCLB, it reasonable to expect that they will set the standards at less ambitious levels than they would have been set a couple of years earlier.The standards recently set by Texas for their new assessment, the Texas Assessment of Knowledge and Skills (TAKS), are consistent with the expectation that states may set their sights a little lower in the context of NCLB.
States that already had their assessments and performance standards in place prior to the enactment of NCLB faced a dilemma.They confronted the question of whether they should stay the course, recognizing that their performance standards were set at levels that are unrealistic for all children to achieve within the next 12 years.Or should they lower their performance standards and risk being accused of dumbing down their standards?Some states, e.g., Colorado and Louisiana redefined their performance levels for purposes on NCLB.Colorado, for example, has reported results on the Colorado Student Assessment Program (CSAP) in terms of four levels: unsatisfactory, partially proficient, proficient, and advanced.Colorado will continue to use all four levels for reporting to parents, schools, and districts.For purposes of NCLB, however, Colorado has collapsed the partially proficient and proficient levels into one level called proficient.

NCLB Starting Points for States
In order to track their AYP toward the goal of 100% proficient or above by 2013-2014, states have to define percentage proficient starting points.The starting point for each subject (reading/language arts and mathematics) is defined to be equal to the higher of the following two values: (1) the percentage of students in the lowest scoring subgroup who achieve at the proficient level or above and (2) "the school at the 20th percentile in the State, based on enrollment, among all schools ranked by the percentage of students at the proficient level" (P.L. [107][108][109][110]Sec. 111 (b)(2)(E)(ii)).In most cases the latter value will be the higher one and define the starting point.
Because states have their own assessments and set their own performance standards it should not be at all surprising that state NCLB starting points are quite variable.Some states are yet to define their performance standards and/or starting points and some states have expressed their starting points in terms of scale scores that are make comparisons difficult.Percentage proficient or above for reading/language arts starting points are available for 34 states at grades 4 and 8 (Olson, 2003).At grade 4, the starting percentages range from a low (i.e., most stringent) of 13.6% for California to a high (i.e., most lenient) of 77.5% for Colorado, with a median of 49.35%.At grade 8, the stating points range from a low of 13.6% to a high of 74.6% with a median of 46.2%.As at grade 4, California and Colorado define the extremes at grade 8.At grade 4, eight states have starting point percentages of 34% or less and eight states have starting points 64% or more.The corresponding percentages at grade 8 are 35% and 60%.State NAEP results indicate that states do vary in terms of student achievement, but not nearly enough to explain the huge variability in NCLB percentage proficient starting points.For the 43 states that participated in the 2002 NAEP 4th grade reading assessment, for example, the percentage of students who were at the proficient level or above ranged from a low of 14% in Mississippi to a high of 47% in Massachusetts (Grigg, Daane, Jin & Campbell, 2003).
The variability in the starting points is of similar magnitude for mathematics at grades 4 and 8 as that found for reading/language arts.The range for mathematics at grade 4 is from 8.3% in Missouri to 79.5% in Colorado and at grade 8 the range is from 7% in Arizona to 74.6% in North Carolina.On the 2000 NAEP mathematics assessment, North Carolina students did perform somewhat better than Arizona students.Thirty percent of the North Carolina students were at the proficient level or above on the grade 8 mathematics compared to 21% in Arizona (Braswell, Lutkus, Grigg, Santapau, Tay-Lim & Johnson, 2001).The grade 8 mathematics achievement of students in Arizona and North Carolina appears much more similar on NAEP, however, than is suggested by the starting points of 7% and 74.6%.

Controversy Regarding Performance Standards
Performance standards have been the subject of considerable controversy.The performance standards called achievement levels set on the NAEP) assessments, for example, have been subjected to harsh criticism.Reviews by panels of both the National Academy of Education (NAE) (Shepard, Glaser, Linn, & Bohrnstedt, 1993) and the National Research Council (NRC) (Pellegrino, Jones, & Mitchell, 1998) concluded that the procedure used to set the achievement levels was "fundamentally flawed" (Shepard, et al., 1993, p. xxii;Pellegrino, et al., 1998, p. 182).The conclusions of the NAE and NRC panels were controversial and several highly-regarded measurement experts have defended the procedure used to set the NAEP achievement levels as well as the resulting levels (e.g., Cizek, 1993;Kane, 1993Kane, , 1995;;Mehrens, 1995).
There is an abundance of methods for setting standards, but there is no agreed upon best method.This point was made repeatedly at the Joint Conference on Standard Setting for Large-Scale Assessments sponsored by the National Assessment Governing Board and the National Center for Education Statistics held October 5-7, 1994 in Washington, DC.The Joint Conference included 18 presentations by scholars representing multiple perspectives.The papers dealt with a variety of issues ranging from technical to policy and legal issues.Crocker and Zieky (1995) prepared an executive summary of the conference which included the following summary conclusion.
"Even though controversies and disagreements abounded at the conference, there were some areas of general agreement.Authors agreed that setting standards was a difficult, judgmental task and that procedures used were likely to disagree with one another.There was clear agreement that the judges employed in the process must be well trained and knowledgeable, represent diverse perspectives, and that their work should be well documented" (p.ES-13).

Variability in Standards
As was indicated by Crocker and Zieky, there is a broad consensus in the field that different methods of setting standards will yield different standards.This consensus is consistent with Jaeger's (1989) summary of the literature on the comparability of standard setting methods."Different standard-setting procedures generally produce markedly different standards when applied to the same test, either by the same judges or by randomly parallel samples of judges" (Jaeger, 1989, p. 497).It is also agreed that different groups of judges will set different standards when using the same method, especially when the groups represent different constituencies (e.g., teachers, administrators, parents, the business community, or the general public).Moreover, there is general agreement that "… there is NO 'true' standard that the application of the right method, in the right way, with enough people, will find" (Zieky, 1994, p. 29).
Given that there is no "true standard or "best" method for setting a standard, it is reasonable to ask what should be made of the variability in results as the function of choice of methods or choice of judges.If one wants to make generalizations across methods or groups of judges then it would seem reasonable to treat the variability in results as error variance.In doing so, we would at least acknowledge that there is a high degree of uncertainty associated with any performance standard.

Variability Due to Judges
Attempts are often made to estimate the error variability due to judges as part of the standard setting process.A difficulty that is encountered, however, is that standard setting methods usually involve group discussion following an initial set of judgments which may be made independently.Group discussion obviously makes judgments in subsequent rounds dependent which makes it impossible to estimate the error variability due to judges in the final round of judgments.Furthermore, there are good reasons to believe that the person who leads the standard setting exercise may have an important influence on the outcome.Thus, what would be desired is something akin to duplicate-construction experiment that Cronbach (1971) proposed as a way to evaluate the content validity of a test.The duplicate-construction experiment would require that two teams of "equally competent writers and reviewers" (p.456) independently construct alternate tests.The parallel in standard setting would involve the use of independent panels of comparably qualified judges set the standards under the direction of equally competent leaders using the same method and instructions.The variance in the standards for the two panels would provide an estimate of the amount of error due to the panel of judges and standard setting leader.
Since the bigger sources of variability in standards is apt to come from the way in which judges are identified and the method that is used to set standard, even the parallel of Cronbach's duplicate-construction experiment would greatly underestimate the real degree of uncertainty in the standards.Some idea of the degree of variability due to the identification of judges is provided by results of a study conducted by Jaeger, Cole, Irwin, and Pratto (1980).Jaeger and his colleagues had three panels, consisting of samples of teachers, school administrators, and counselors, respectively, independently set passing standards on a North Carolina test.The differences in the standard set by the different panels can be gauged by the magnitude of the differences in the proportion of students who would have failed the test according to the different groups of judges.On the reading test the proportion who would have failed ranged from a low of 9% to a high of 30%.The variability in failure rates was even greater for the mathematics test, ranging from a low of 14.4% to a high of 71.1%.

Variability Due to Method
Variability due to choice of method can be evaluated based on results of several different studies that were reviewed by Jaeger (1989).One of those studies where multiple methods were used, for example, was conducted by Poggio, Glassnapp and Eros (1981).They had independent samples of teachers set standards using one of four different standard setting methods: the Angoff (1971) method, the Ebel (1972) method, the Nedelsky (1954) method, or the contrasting groups (see, for example, Jaeger, 1989) method.Teachers set standard for tests at grades 2, 4, 6, 8, and 11.There was substantial variability in the standards set by the four different methods at every grade.At grade 8, for example, the four different methods would set the minimum passing score on a 60 item reading test at 28, 39, 43, and 48 items correct.If the most lenient standard had been used, just over 2% of the students would have failed whereas approximately 29% would have failed if the most stringent standard had been used.
In his summary of 32 contrasts of standards set by different methods Jaeger (1989) found that the ratios of the percentages of examinees who would fail range from a low of 1.00 to a high of 29.75 with a median of 2.74.That is, the typical consequence of using a different method to set standards would be to alter the failure rate by a factor of almost 3.As Jaeger (1989) concluded the "choice of a standard setting method is critical" (p.500).He went on to endorse earlier suggestions by Hambleton (1980), Koffler (1980) and Shepard (1980;1984) that "it might be prudent to use several methods in any given study and then consider all the results, together with extra-statistical factors, when determining a final cutoff score" (Jaeger, 1989, p. 500).
Because of the practical cost considerations, the use of multiple methods as input to a final standard setting decision is rare in operational practice, but that was the approach recently taken in Kentucky for the assessments introduced in the state in 2000.As was recently reported by Green, Trimble and Lewis (2003), the Kentucky Department of Education used multiple methods as input to their final standard setting when the state introduced a new testing system, the Kentucky Core Content Test (KCCT) in 2000.First, three different methods, the bookmark procedure (Lewis, Green, Mitzel, Baum & Patz, 1998;Mitzel, Lewis, Patz, & Green 2001), the Jaeger-Mills procedure (Jaeger & Mills, 2001), and the contrasting group (see, for example, Jaeger, 1989) were used to set cut scores to distinguish four levels of performance (novice, apprentice, proficient and distinguished) on each of 18 different tests used in the KCCT system for various grade levels and content areas.The results of the bookmark, the Jaeger-Mills, and the contrasting groups standards setting efforts were input to a synthesis process where the results were considered by teacher committees that recommended cut scores to the Kentucky State Board of Education (Green, Trimble, & Lewis, 2003; see also CTB/McGraw-Hill, 2001;and Kentucky Department of Education, 2001, for more detailed descriptions).
Table 1 displays the percentage of students at the proficient level or above on each of six KCCT subject area tests administered at elementary school grades according to the three standard setting methods.Also shown are summary statistics across the three methods, mean, standard error, minimum, maximum, and range as well as the percentage proficient or above according to the standard set by the synthesis panel.The standard error is simply the standard deviation of the percentages for the three different standard setting methods since the standard deviation may be interpreted as a standard error if the goal is go generalize across standard setting methods.As can be seen, the standard errors are quite large, indicating that there is considerable uncertainty about the percentage of students proficient or above due to standard setting method.Tables 2 and 3 display results parallel to those in Table 1 for the KCCT tests administered at the middle school and high school grades, respectively.The results for the 12 subject area by grade combinations shown in Tables 2 and 3 are similar to those shown in Table 1 for the tests administered at the elementary school grades.The mean standard error across the 18 subject area by grade combinations in Tables 1 through 3 is 10.82 and they range from a low of 4.26 for the middle school mathematics test to a high of 26.35 for the middle school reading test.Even in the best case there is a good deal of uncertainty about the percentage of students who are at the proficient level or above as the consequence of the method used to set the performance standards.A standard error of even 4 points is large relative to the annual change in percentage proficient or above that is likely to be required to meet the AYP target for NCLB.
A standard error of 26 points is gigantic in that same context.Of course Kentucky did not use any of the three methods to set the performance standards for operational use.Rather the results of the three methods were used as input to the synthesis panels that provided the final recommendations to the State Board of Education.Hence, it might be argued that the variability due to method is not relevant to judging the uncertainty of the final performance standards.Thus, a reasonable question is what is the degree of uncertainty for operational standards in Kentucky or any other state?Results recently reported by Hoover (2003) are relevant to this question.Hoover compared of the percentage of students labeled proficient or advanced according to three national test batteries and NAEP.Hoover's results show that performance standards that are finally adopted after much care and expense by national test publishers for their tests or by the National Assessment Governing Board for NAEP also have a great deal of uncertainty.
According to the three national tests the percentage of grade 4 students who are proficient or above in reading is 24% according to one test publisher, 40% according to another publisher, and 55% according to the third publisher.
According to NAEP the figure is 31%.For grade 4 mathematics, the corresponding four numbers are 15%, 34%, 44%, and 26% (Hoover, 2003, p. 11).Hoover's comparisons also show that, whereas there is apparently a substantial decline in the percentage of students who are proficient or above from grade 8 to 9 (33% vs. 11%) according to one publisher, there is no such decline according to the other two publishers.While one publisher's performance standards show a fairly steady decline in percentage of students who are proficient or advanced in mathematics from grades 1 through 12 (from 41% to 5%) another publisher's performance standards show that slightly more than twice as many students (27% vs. 12%) are proficient or advanced at grade 12 than at grade 1.

Conclusion
The variability in the percentage of students who are labeled proficient or above due to the context in which the standards are set, the choice of judges, and the choice of method to set the standards is, in each instance, so large that the term proficient becomes meaningless.Insistence on standards-based reporting of achievement test results where such reporting serves no essential purpose is more harmful than helpful.This is particularly true in the context of NCLB where schools, districts, and states are subject to substantial sanctions based on the progress that is made against arbitrary performance standards that lack any semblance of comparability from state to state.
Several years ago I (Linn, 1995) described four uses of performance standards: exhortation, exemplification of goals, accountability, and the certification of student achievement.Performance standards and associated cut scores are essential only for the fourth use.Although reporting results in terms of performance standards is often done to exhort teachers and students to exert greater effort standard-based reporting is not essential to that use.Nor are performance standards essential for the purpose of exemplifying goals.NCLB and a number of state accountability systems depend on the performance standards, but that would not have to be the case.
One of the purposes of introducing performance standards is to provide a means of reporting results in a way that is more meaningful than a scale score.
Certainly, reporting that a student performed at the proficient level appears more understandable than saying that the student has a scale score of 215.
Parents familiar with student performance in school in terms of grades of, say, A, B, C, D and F might naturally assume that proficient is like a grade of B or C. Given the huge inconsistencies in definitions of proficient achievement and in the associated stringency of cut scores, however, it seems clear that attaching such an interpretation to performance levels cannot mean the same thing across states where standards vary so radically in their stringency.It would be better to find another way of dealing with these non-essential uses of performance standards and cut scores.
There obviously is a legitimate interest in being able to measure progress in student achievement.There are many ways of measuring progress and setting AYP targets that do not depend upon the reporting of results in terms of performance standards.Effect-size statistics that would measure the year-to-year difference in mean achievement scores in terms of the standard deviation of scores in the base year is one obvious way that progress could be measured.Holland's (2002) proposal to measure progress in student achievement by comparing cumulative distribution functions is another approach.Comparisons of cumulative distribution functions provide a means of monitoring changes in student performance throughout the range of performance.Changes in the percentage of student exceeding any selected score level can be readily determined rather than just focusing on one arbitrary cut score that corresponds to the proficient performance standard.
Comparisons might also be made to norms for a base or reference year.Finally, if it is decided that the best way to track progress is in terms of the percentage of students scoring above a fixed cut score, sometimes referred to as PAC for percent above cut, then it would be better to pick the cut score based on norms in a base year than to use an arbitrary definition of "proficient" performance that bears little similarity to the definition of proficient performance in another state.The median or some other percentile rank in a base year might be used as the cut score.This would provide a clear and consistent meaning that does not seem possible to achieve for the proficient performance standard.
Using PAC statistics to track progress would provide a reasonable alternative to tracking progress in achievement for different states in terms of percent proficient or above.PAC statistics based on a cut score at, say, the median achievement level in a base year would also be much more interpretable than percentages proficient or above when comparisons are made across states.
Reports of individual student assessment results in terms of norms have more consistent meaning across different assessments than reports in terms of proficiency levels based on uncertain standards.Furthermore criterion-referenced reports of results can be provided by illustrating the types of items that the student can answer correctly and the types that they cannot.A fixed cut score is not essential to criterion-referenced interpretations of achievement.

Postscript
For the reasons discussed above, I believe it would be desirable to shift away from standards-based reporting for uses where performance standards are not an essential part of the test use.I recognize, however, that existing state and federal laws require the setting of performance standards and the reporting of results in terms of those standards.Thus, at least until the laws are changed, there is no choice but to work to make performance standards as reasonable as possible.Assuring that judges on standard setting panels understand the context in which the standards will be used is a minimal requirement for obtaining reasonable performance standards.Normative information needs to be made part of the process for judges to anchor their absolute judgments with an understanding of current levels of performance of students and likely consequences.As Zieky (2001) has noted, considering both absolute and normative information "in setting a cutscore can help avoid the establishment of unreasonably high or low values" (p.38).In addition to knowing the percentile rank corresponding to particular cut scores, it would also be desirable to have some means of providing judges with information comparative information about the relative stringency of their standards in comparison to standards set in other states before judgments are finalized.Normative information would be one way of making comparisons to standards in other states.
It is critical that the context in which the standards will be used be made as clear as possible to panels of judges who set the standards.The uses of the standards for purposes of NCLB with its expectation that all students will be at the proficient level or higher by 2014 and sanctions for schools that do not meet AYP targets is an important part of the current context that standard setters need to consider.
Finally, while there is no agreed upon best method for setting standards, the literature does provide useful indications of the differences among different methods in their relative stringency and ease of use.Jaeger's (1989) advice that multiple standard setting methods be used and the results of the different methods be "considered together with extrastatistical factors when determining the cutoff score" (p.500) seems as sound now as it in 1989.The experience in Kentucky with the use of multiple methods as input to a synthesis panel provides an excellent example where that advice was taken seriously in practice.
A number of authors have suggested helpful criteria to consider in selecting a method.Hambleton (2001), for example, identifies 20 criteria that he presents in the form of questions to be considered in evaluating a standard-setting process.Raymond and Reid (2001) provide useful advice on the selection and training of judges, Kane (2001) provides a good discussion of considerations in the validation of performance standards and cut scores, and Bond (1995) has identified five principles intended to help ensure fairness.Although such considerations cannot eliminate the arbitrariness, they can help make the standards and cut scores more reasonable and more defensible.

Table 3 Percentage of Students at the Proficient Level or Above on KCCT For High School Grade Tests According to Standard Setting Method
If improvement in performance in State A was large enough that three quarters of the students in 2013-2014 performed above the median level in 2002-2003 that would represent a large improvement in student achievement.It would also be readily understood that that students in state A generally had better achievement than students in state B where 55% of the students in 2013-2014 scored above the 2002-2003 median.Furthermore, the norm-based results for states A and B would be much more interpretable than a statement that three-fourths of the students in State C were proficient or above in 2013-2014 compared to only 55% in State D, because the meaning of proficient might be radically different for States C and D. Indeed, if the stringency of the proficient performance standard were as variable from state to state as it is now, it might well be that the achievement of students in state D was actually better than that in state C.