I. Introduction

Administrators, faced with a glut of data, often find creative ways to reduce it (the SET process) to meaningless numbers. I encountered one who insisted that it sufficed to consider only the question on overall effectiveness, because he had once seen a report that, on average, the average on this question equaled the average of all other questions. He persisted in this policy even in cases for which it was patently false ... Advocates often cite a few superficial studies in support of the reliability of student evaluations. However, other studies give a more complex picture ... Many experienced faculty question the reliability of student evaluations as a measure of teaching effectiveness and worry that they may have counter- productive effects, such as contributing to grade inflation, discouraging innovation, and deterring instructors from challenging students.

Concerning questions about the legal basis of student evaluations of faculty, Lechtreck (1990) points out that, "In the past few decades, courts have struck down numerous tests used for hiring, and/or promotions on the grounds that the tests were discriminatory or allowed the evaluator to discriminate. The question, How would you rate the teaching ability of this instructor, is wide open to abuse" (p. 298). In his column, "Courtside," Zirkel (1996) states, "Courts will not uphold evaluations that are based on subjective criteria or data" (p. 579). Administrative assumptions to the contrary, student evaluations of faculty are not objective, but rather, by their very nature, must be considered subjective. (p. 2) (Note 3)

To know what can be done, and how to do it, is beyond a doubt, the most important species of information. The next to it is, to know what cannot be done, and why we cannot do it. The first enables us to attain a positive good, to increase our powers, and augment our happiness: the second saves us from the evil of fruitless attempts, and the loss and misery occasioned by perpetual failure. (p. 14)

II. The Conventional SET Process

1. The SET survey instrument is comprised of a series of questions about course content and teaching effectiveness. Some questions are open-ended, while others are closed-ended.
2. Those, which are closed-ended, often employ a scale to record a response. The range of possible values, or example, may run from a low of 1 for "poor," to a high of 5 for "outstanding."
3. In the closed-ended section of the SET survey instrument, one question is of central import to the "summative" function. It asks the student: "Overall, how would you rate this instructor as a teacher in this course?" In the main, this question plays a pivotal role on the evaluation process. For ease of reference, I term this question the "single-most-important question" (hereafter, the SMIQ).
4. In the open-ended section of the SET survey instrument, students are invited of offer short critiques of the course content and of the teaching effectiveness of the instructor.
5. The completion of the SET survey instrument comes with a guarantee to students; that is, the anonymity of individual respondents.
6. The SET survey instrument is administered: (i) by a representative of the university administration to those students of a given class who are present on the data- collection day, (ii) in the latter part of the semester, and (iii) in the absence of the instructor.
7. Upon completion of the survey, the analyst then takes the response to each question on each student's questionnaire, and then constructs question-specific and class-specific measures of central tendency, and of dispersion – this in an attempt to determine if the performance of a given instructor in a particular class meets a cardinally- or ordinally- measured minimal level of "teaching effectiveness." (Note 5)
8. It seems that, in such analyses, raw SET data on the SMIQ are used in the main. More likely than not, this situation arises from the fact that the SET survey instrument does not provide for the collection of background data on the student respondent (such as major, GPA, program year, required course?, age, gender, …), and on course characteristics. (Note 6)

III. Fallacies Of A Conceptual Sort Inherent In The SET Process

III.1. Students As A, Or The Only, Source Of Reliable Information on Teaching Effectiveness

• The Public-Good Argument: The advocates of the SET process would argue: The university is a business, and the student its customer. And since the customer is always right, customer opinion must drive the business plan. Mainstream economists would argue that this is a false analogy. Their reason is that these same advocates are assuming that the provision of tertiary education is a "private good." This (economists would argue) is not so: It is a "public good."(Note 7) As such, students are not solely qualified to evaluate course content, and the pedagogical style of a faculty member.

• The Student-Instructor Relationship Is Not One of Customer-Purveyor, And Hence Not A Relationship Between Equals: As Stone (1995) noted,

  Higher education makes a very great mistake if it permits its primary mission to become one of serving student "customers." Treating students as customers means shaping services to their taste.  It also implies that students are entitled to use or waste the services as they see fit.  Thus judging by enrollment patterns, students find trivial courses of study, inflated grades, and mediocre standards quite acceptable.  If this were not the case, surely there would have long ago been a tidal wave of student protest.  Of course, reality is that student protest about such matters is utterly unknown.  Tomorrow, when they are alumni and taxpayers, today's students will be vitally interested in academic standards and efficient use of educational opportunities.  Today, however, the top priority of most students is to get through college with the highest grades and least amount of time, effort, and inconvenience.

  As Michael Platt (1993) noted:

  The questions typical of student evaluations teach the student to value mediocrity in teaching and even perhaps to resent good teachers who, to keep to high purposes, will use unusual words, give difficult questions, and digress from the syllabus, or seem to. Above all, such questions also conceive the relation of student and teacher as a contract between equals instead of a covenant between unequals. Thus, they incline the student, when he learns little, to blame the teacher rather than himself. No one can learn for another person; all learning is one's own .... (p. 31)

Merely by allowing the forms, the teacher loses half or more of the authority to teach. (p. 32)

• Students Are Not Sufficiently Well-Informed To Pronounce On The Success Or Failure of the Academic Mission: Because of age and therefore relative ignorance, students are not sufficiently well-informed about societal needs for educated persons, and employers' needs for skill sets. Therefore, students are not in a position to speak for all vested interests (including their own long- term interests). For example, Michael Platt (1993) noted:

  Pascal says: while a lame man knows he limps, a lame mind does not know it limps, indeed says it is we who limp. Yet these forms invite the limpers to judge the runners; non-readers, the readers; the inarticulate, the articulate; and non-writers, the writers. Naturally, this does not encourage the former to become the latter. In truth, the very asking of such questions teaches students things that do not make them better students. It suggests that mediocre questions are the important questions, that the student already knows what teaching and learning are, and that any student is qualified to judge them. This is flattery. Sincere or insincere, it is not true, and will not improve the student, who needs to know exactly where he or she stands in order to take a single step forward. (p. 32)

  In the same vein, Adams (1997) noted,

  Teaching, as with art, remains largely a matter of individual judgment. Concerning teaching quality, whose judgment counts? In the case of student judgments, the critical question, of course, is whether students are equipped to judge teaching quality. Are students in their first or second semester of college competent to grade their instructors, especially when college teaching is so different from high school? Are students who are doing poorly in their courses able to objectively judge their instructors? And are students, who are almost universally considered as lacking in critical thinking skills, often by the administrators who rely on student evaluations of faculty, able to critically evaluate their instructors? There is substantial evidence that they are not. (p. 31)

• The Anonymity of The Respondent: As noted above, the SET process provides that the identity of the respondent to the SET questionnaire would or could never be disclosed publicly. This fact contains a latent message to students. This is, in the SET process,

  there are not personal consequences for a negligent, false, or even malicious representation. There is no "student responsibility" in student evaluations. It is as if the student was being assured: "We trust you. We do not ask for evidence, or reasons, or authority. We do not ask about your experience or your character. We do not ask your name. We just trust you. Your opinions are your opinions. You are who you are. In you we trust." Most human beings trust very few other human beings that much. The wise do not trust themselves that much. [Platt (1993, p. 34)]

III.2. Opinion Misrepresented As Fact Or Knowledge

I cannot think that the habit of evaluating one's teacher can encourage a young person to long for the truth, to aspire to achievement, to emulate heroes, to become just, or to do good. To have one's opinions trusted utterly, to deliver them anonymously, to have no check on their truth, and no responsibility for their effect on the lives of others are not good for a young person's moral character. To have one's opinions taken as knowledge, accepted without question, inquiry, or conversation is not an experience that encourages self-knowledge. (pp. 33-34)

What they teach is that "Opinion is knowledge." Fortunately, the student may be taught elsewhere in college that opinion is not knowledge. The student of chemistry will be taught that the periodic table is a simple, intelligible account of largely invisible elements that wonderfully explains an enormous variety of visible but heterogeneous features of nature. (p. 32)

1. The first premise contains the uninspected supposition that through introspection, any student can "know" an unobservable metric called "teaching effectiveness," and can then be relied upon to accurately report her measurement of it in the SET document. (Note 8)

2. The literature makes quite clear that within any group of students one can find multiple perceptions of what constitutes "teaching effectiveness" (e.g., Fox (1983)). (Note 9)

3. If a measure is unobservable, its metric cannot be claimed to be also unambiguously unique and immutable. (Note 10) To argue otherwise is to be confronted by a bind: A measure cannot be subjective, and its metric objective.

1. One motivation is the "education-as-an-investment-good" view. This is tantamount to the view that "going to university" will enhance one's prospects of obtaining a high-paying and/or an intellectually-satisfying job upon graduation. Latent in this view is the fact or belief that many employers take education as a signal of the productive capability of a university graduate as a job applicant [Spence (1974), and Molho (1997, part 2)]. (Note 11)

2. The other motivation is the "education-as-a-consumption-good" view. This view is tantamount to some mix of these five views: (a) that education is to be pursued for education's sake, (b) that "going to university" must be above all else enjoyable, (c) higher education is a democracy, and (d) in this democracy, learning must be fun, and (e) to be educated, students must like their professor. (Notes 12, 13)

Attention teachers far and wide: It may not be so much what or how you teach that will reap high student evaluations, but something as simple as an enthusiastic tone of voice and beware, administrators, if you use student ratings to judge teachers: Although student evaluations may be systematic and reliable, a Cornell university study has found that they can be totally invalid. Yet many schools use them to determine tenure, promotion, pay hikes and awards.
These warnings stem from a new study in which a Cornell professor taught the identical course twice with one exception—he used a more enthusiastic tone of voice the second semester—and student ratings soared on every measure that second semester.
Those second-semester students gave much higher ratings not only on how knowledgeable and tolerant the professor was and on how much they say they learned, but even on factors such as the fairness of grading policies, text quality, professor organization, course goals and professor accessibility.
And although the 249 students in the second-semester course said they learned more than the 229 students the previous semester believed they had learned, the two groups performed no differently on exams and other assessment measures.
"This study suggests that factors totally unrelated to actual teaching effectiveness, such as the variation in a professor's voice, can exert a sizable influence on student ratings of that same professor's knowledge, organization, grading fairness, etc.," said Wendy M. Williams, associate professor of human development at Cornell. Her colleague and co-author, Stephen J. Ceci, professor of human development at Cornell, was the teacher evaluated by the students in a course on developmental psychology that he has taught for almost 20 years.

At times, evaluations appear to be the academic analogue to "Rate the Record" on Dick Clark's old "American Bandstand," in which teen-agers said of every new release, "Good beat, great to dance to, I'd give it a 9." Students are becoming more adjectival than analytical, more inclined to take faculty members' wardrobes and hairstyles into account when sizing them up as educators.

IV. Fallacies Of A Statistical Sort Inherent In The SET Process

IV.1. Ascribing A Cardinal Measure of Teaching Effectiveness To An Instructor Based on The SMIQ

IV.2. The Rank Ordering Of Instructors By Teaching Effectiveness Based on The SMIQ

A …virtually universal problem with previous research is that the overall rating is viewed as an effective representation of comparative professor value despite the fact that it typically includes assessments in areas that are beyond the professor's control. The professor is responsible to some extent for course content and characteristics specific to his/her teaching style, but is unable to control for student attitude, reason for being in the course, class size, or any of the rest of those factors categorized as student or course characteristics above. Consequently, faculty members should be evaluated on a comparative basis only in those areas they can affect, or more to the point, only by a methodology that corrects for those influences beyond the faculty member's control.
          By comparing raw student evaluations across faculty members, administrators implicitly assume that none of these potentially mitigating factors has any impact on student evaluation differentials, or that such differentials cancel out in all cases. The literature implies that the former postulate is untrue.

Professor characteristics dominated the determinants of the summary measures of performance, and did so more for those summary variables that were more professor-specific. However, certain course- and student- specific characteristics were very important, skewing the rankings based on the raw results. Students consistently rewarded teachers for using class time wisely, encouraging analytical decision making, knowing when students did not understand, and being well prepared for class. However, those professors who gave at least the impression of lower grades, taught more difficult courses, proceeded at a pace students did not like, or did not stimulate interest in the material, fared worse. (p. 414)

Based on the probit analysis, an alternative ranking scheme was developed for faculty that excluded influences beyond the professor's control. These rankings differed to some extent from the raw rankings for each of the aggregate questions. As a result, the validity of the raw rankings of faculty members for the purposes of promotion, tenure, and raises should be questioned seriously. … Administrators should adjust aggregate measures of teaching performance to reflect only those items within the professors' control, so that aggregates are more likely to be properly comparable and should do so by controlling for types of courses, levels of courses, disciplines, meeting times, etc. … Administrators failing to do this are encouraged to reconsider the appropriateness of aggregate measures from student evaluations in promotion, tenure, and salary decisions, concentrating instead on more personal evaluations such as analysis of pedagogical tools, peer assessments, and administrative visits. (p. 414)

V. Why Has The Conventional SET Process Not Been Discarded?

End of term student evaluations of teaching may be widely used simply because they are inexpensive to administer, especially when done by a student in class, with paid staff involved only in the processing of the results…Less-than-scrupulous administrators and faculty committees may also use them … because they can be dismissed or finessed as needed to achieve desired personnel ends while still mollifying students and giving them a sense of involvement in personnel matters. [Becker (2000, p. 114)]

[W]e moderns, it seems, attempt to measure everything…. We evaluate performance by measurement…. What is not measurable we strive to render measurable, and what we cannot, we dismiss it from our thoughts and justify our neglect by assigning it the status of the "less important." … A moment's reflection, however, is all that is needed to realize that measurement cannot possibly do everything we expect it to do. … by omitting from our considerations what cannot be measured, or what we do not know how to measure, often leads to irrelevance and even error. (p. 18)

1. What ought to be appraised in the philosophy of the sciences is not an isolated individual theory, but a cluster of interconnected theories, or what he terms "scientific research programs" (hereafter SRP).
2. An SRP protects a "hard core" set of unquestioned and untestable statements. These statements are accepted as "fact."
3. Stated differently, the hard core of a SRP is surrounded by a "protective belt" of "auxiliary hypotheses."
4. One or more of the hard core statements cannot be refuted without dismantling the entire cognitive edifice, which happens in practice only very rarely. That said, it follows that any departure from the hard core of a SRP is tantamount to the creation of a new and different SRP.

VI. Conclusion

Administrators should adjust aggregate measures of teaching performance to reflect only those items within the professors' control, so that aggregates are more likely to be properly comparable and should do so by controlling for types of courses, levels of courses, disciplines, meeting times, etc. … Administrators failing to do this are encouraged to reconsider the appropriateness of aggregate measures from student evaluations in promotion, tenure, and salary decisions, concentrating instead on more personal evaluations such as analysis of pedagogical tools, peer assessments, and administrative visits. (p. 414)

Notes

1. For reviews of the literature that are essentially supportive of the SET process, see d'Apollonia and Abrami (1997), Greenwald and Gilmore (1997), Marsh (1987), Marsh and Roche (1997), and McKeachie (1997). And for reviews of the literature that are highly critical of some mix of the conceptual, statistical, and legal foundations of the SET process, see Damron (1995), and Haskell (1997a, 1997b, 1997c, and 1997d).
   
2. The terms "formative" and "summative" are due to Scriven (1967).
   
3. On such matters, the position of the Canadian Association of University Teachers on the admissibility of SET data appears unambiguous in light of statements like these: "Appropriate professional care should be exercised in the development of questionnaires and survey methodologies. Expert advice should be sought, and reviews of the appropriate research and scientific evidence should be carried out. Comments from faculty and students and their associations or unions should be obtained at all stages in the development of the questionnaire. Appropriate trials or pilot studies should be conducted and acceptable levels of reliability and validity should be demonstrated before a particular instrument is used in making personnel decisions" [Canadian Association of University Teachers (1998, p. 3)]. In a footnote to this passage, this document continues, "Most universities require at least this standard of care before investigators are permitted to conduct research on human subjects. It is unacceptable that university administrations would condone a lesser standard in the treatment of faculty, particularly when the consequences of inadequate procedures and methods can be devastating to teachers' careers."
   
4. The present characterization represents an amalgam of three sources: (a) first-hand knowledge of the SET documents used at three Canadian universities; (b) a small, non-random sample of SET documents for four universities taken from the internet [viz., University of Minnesota, University of British Columbia, York University (Toronto), and University of Western Ontario]; and (c) non-institutional-specific comments made in the voluminous literature on the SET process.
   
5. The phrase "a cardinally- or ordinally- measured minimal level of "teaching effectiveness"" requires four comments. One, examples of cardinal measures are: The heights of persons A, B, and C are 6'1", 5'10", and 5'7" respectively. And using the same data, examples of ordinal measures are: A is taller than B, B is taller than C, and A is taller than C. Two, the present measurement terminology is used in economics [Pearce (1992)], and (it can be said) is distinct from that used in other disciplines [e.g., Stevens (1946), Siegel (1956, p. 30), and Hands (1996)]. Three, it is the existence of a unique and an immutable metric (in the above examples, distance or length) that makes both cardinal and ordinal measures meaningful. Four, as the above examples make clear, an ordinal measure can be inferred from a cardinal measure, but not the reverse.
   
6. An example of this statement is the instrument used by York University (Toronto). An exception to this statement is that used by the University of Minnesota.
   
7. The distinction between a "private good" and a "public good" can be rephrased in several, roughly equivalent ways. These are: (i) tertiary education has externalities; (ii) that the net social benefits of tertiary education differ from the net private benefits, (iii) that the benefits of tertiary education do not accrue to, nor are its costs borne by, students solely, and (iv) that students do not pay full freight. Because of this, one could argue that (in the evaluation of "teaching effectiveness") the appropriate populations of opinion to be sampled are all groups who share in the social benefits and social costs. These would include not only students, but also members of the Academy, potential employers, and other members of society (such as taxpayers). In sum, because tertiary education is not a private, but a public good, students are not solely qualified to evaluate course content, and the pedagogical style of a faculty member.
   
8. A personal vignette provides some insight into the potential seriousness of the inaccuracy of self-reported data. In the fall of 1997, I taught an intermediate microeconomics course. The mark for this course was based solely on two mid-term examinations, and a final examination. Each mid-term examination was marked, and then returned to students and discussed in the class following the examination. Now, the course evaluation form has the question, "Work returned reasonably promptly." The response scale ranges from 0 for "seldom," to 5 for "always." Based on the facts, one would expect (in this situation) an average response of 5. This expectation was dashed in that 50% of the sample gave me a 5, 27.7% gave me a 4, and 22.2% gave me a 3. The import of this? If self-reported measures of objective metrics are inaccurate (as this case indicates), how can one be expected to trust the validity of subjective measurements like "teaching effectiveness?"
   
9. Indeed, it appears that students and professors can hold different perceptions as to what constitutes "appropriate learning," and hence "appropriate teaching," in tertiary education. For example, Steven Zucker (1996), professor of Mathematics at Johns Hopkins University, laments the gulf between the expectations of students and instructors. He writes: "The fundamental problem is that most of our current high school graduates don't know how to learn or even what it means to learn (a fortiori to understand) something. In effect, they graduate high school feeling that learning must come down to them from their teachers. That may be suitable for the goals of high school, but it is unacceptable at the university level. That the students must also learn on their own, outside the classroom, is the main feature that distinguishes college from high school." (p. 863).
   
10. Alternatively, Weissberg (1993, p. 8) noted that one cannot measure what one cannot define.
    
11. These assertions have been borne out empirically under the rubric, "sheepskin effect." The interested reader is directed to Belman and Heywood (1991 and 1997), Heywood (1994), Hungerford and Solon (1987), and Jaeger and Page (1996).
    
12. Some of these views contradict the raison d'être and the modus operandi of tertiary education. For example, Frankel (1968) wrote: "Teaching is a professional relationship, not a popularity contest. To invite students to participate in the selection or promotion of their teachers exposes the teacher to intimidation." (pp. 30-31) In fact, the Canadian Association of University Teachers (1986) speaks of the irrelevance of "popularity" as a gauge of professional performance by stating: "The university is not a club; it is dedicated to excellence. The history of universities suggests that its most brilliant members can sometimes be difficult, different from their colleagues, and unlikely to win a popularity contest. 'The university is a community of scholars and it is to be expected that the scholars will hold firm views and wish to follow their convictions. Tension, personality conflicts and arguments may be inevitable by-products.'"
    
13. As Crumbley (1995) noted: "There is another universal assumption that students must like an instructor to learn. Not true. Even if they dislike you and you force them to learn by hard work and low grades, you may be a good educator (but not according to SET scores). SET measures whether or not students like you, and not necessarily whether you are teaching them anything. Instructors should be in the business of educating and teaching students--not SET enhancement. Until administrators learn this simple truth, there is little chance of improving higher education."
14. It seems that some psychologists would argue that latent measures of "teaching effectiveness" can be uncovered by a factor analysis of the SET data [e.g., d'Apollonia and Abrami (1997)]. Also, it seems that the motivation for such a claim is the intellectual appeal and success of studies of a completely different ilk. A case in point is Linden (1977) who uses factor analysis to uncover dimensions, which account for event-specific performances of athletes in the Olympic decathlon. However, the expectation that the success found in studies such as Linden (1977) can be replicated in the factor analysis of SET data is unwarranted in that this expectation ignores the fact that the SET data (unlike Linden's data) are opinion based or subjective, have measurement error, and are in need of statistical controls. In brief, it is my view that the use of factor analysis on SET data to uncover latent measures such as "teaching effectiveness" is analogous to trying to "unscramble an egg" in that it just cannot be done. Besides, as the authors of a popular text on multivariate statistics observe, "When all is said and done, factor analysis remains very subjective" [Johnson and Wichern (1988, p. 422)].
    
15. The terms, ordinal and cardinal measures, are defined in a footnote above. In conjunction with that, it should be noted that the type of a variable governs the statistical manipulations permissible [Hands (1996, pp. 460-62)], and "(T)he use of ordinally calibrated variables as if they were fully quantified .. results in constructions that are without meaning, significance, and explanatory power. Treating ordinal variables as cardinal … can mislead an investigator into thinking the analysis has shed light on the real world" [Katzner (1991, p. 3)]. This latter point captures an important dimension of the present state of research on SET data, and of the present paper.
    
16. For reasons of brevity, I have concentrated on only two of several statistical problems. These are "measurement error" and "omitted variables." By doing so, I have overlooked other statistical problems inherent in the SET data like the unreliability of self- and anonymous-reporting, inadequate sample size, sample-selection bias, reverse causation, and teaching to tests. The reader interested in a more complete treatment of some of these issues may wish to consult readings such as Aiger and Thum (1986), Becker and Power (2000), Gramlich and Greenlee (1993), and Nelson and Lynch (1984).
    
17. As Rundell (1996) noted, in actual practice, this would mean: "…'Jones had a 3.94 mean on her student evaluations, and since this is 0.2 above the average for the Department, we conclude she is an above average instructor as judged by these questionnaires' is a statement that appears increasingly common" (p. 1).
    
18. Statistical controls are needed to the extent that they eliminate "observational equivalence." In this connection, two comments are warranted here. One, observational equivalence is said to exist when "alternative interpretations, with different theoretical or policy implications, are equally consistent with the same data.. No analysis of the data would allow one to decide between the explanations, they are observationally equivalent. Other information is needed to identify which is the correct explanation of the data" [Smith (1999, p. 248)]. Two, Sproule (2000) has identified three distinct forms of observational equivalence in the interpretation of raw data from the SMIQ.
    
19. Cashin (1990) reports, for example, professors of fine arts and music receive high scores on the SMIQ, and professors of chemistry and economics receive lower scores, all things being equal.
    
20. Mason et al. (1995) contend that those variables which fall under the "student-characteristics" rubric include: (i) reason for taking the course, (ii) class level of the respondent, (iii) student effort in the course, (iv) expected grade in the course, and (v) student gender. Those variables which fall under the "instructor- characteristics" rubric include: (i) the professor's use of class time, (ii) the professor's availability outside of class, (iii) how well the professor evaluates student understanding, (iv) the professor's concern for student performance, (v) the professor's emphasis on analytical skills, (vi) the professor's preparedness for class, and (vii) the professor's tolerance of opposing viewpoints and questions. Those variables which fall under the "course- characteristics" rubric include: (i) course difficulty, (ii) class size, (iii) whether the course is required or not, and (iv) when the course was offered.
    
21. For an elementary discussion of the ordered-probit model, see Pindyck and Rubinfeld (1991, pp. 273-274.).
    
22. Katzner (1991) also states that this "blind pursuit of numbers" can lead to unintended, and unjust, outcomes. For example, "(W)hen the state secretly sterilizes individuals only because their 'measured intelligence' on flawed intelligence tests is too low, then bitterly dashed hopes and human suffering becomes the issue." (p. 18). That said, it would not be too difficult to claim that the "blind pursuit of numbers" by those responsible for the "summative" function has also led to unintended, and unjust, outcomes. [In fact, see Haskell (1997d) for details.]
    
23. Three comments seem warranted here. One, the enterprise of science can been seen as a "market process" [Walstad (1999)]. Two, the SRP of this cadre of psychologists and educational administrators could be viewed as barrier to entry (of the epistemological sort) into the marketplace of ideas. Three, that said, perhaps the recommendation of Paul Feyerabend (1975) applies in this instance; that competition between epistemologies, rather than the monopoly of a dominant epistemology, ought to be encouraged.
    
24. While it is clear from the above that the protective belt of the SRP associated with the SET has survived many types of logical appraisals (or epistemological attacks), the question remains: Can this protective belt, and this SRP itself, continue to withstand such repeated attacks? I would hazard the opinion that, no, it cannot.
    

General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-0211. (602-965-9644). The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu .