Application of the M&F framework to the Sandra Rubin Glass study, "Markets and Myths: Autonomy in Public and Private Schools", published in Education Policy Analysis Archives, vol. 5, no. 1, January 6,1997 (the 'Glass' study)
The title describes the article well: Glass undertook a large-scale study of teacher and principal autonomy in public and private secondary schools and used the data to explore claims made by Chubb and Moe (1990). She challenges the latter's assertions: that the organization of private schools offered greater teacher autonomy resulting in higher student achievement and that the bureaucracy of public schools stifles autonomy and limits student achievement. Glass attempted to bring to the surface conditions which constrain teacher and principal autonomy in both public and private schools. Although she did not express it this way, it is reasonable to state that she had a hypothesis: "There are no differences between public and private schools in the amount of autonomy teachers and principals have." She also set out to answer the question, "What conditions impede teacher and principal autonomy in both public and private schools?", implying the hypothesis: "There are no differences between public and private schools in the conditions that foster or inhibit autonomy."
In her comprehensive study, Glass used both data source and methodological triangulation. "The methods employed in this investigation were those of the multi-site qualitative case study: interviews from multiple data sources, observations and field notes from a variety of on-site meetings and visits, and analysis of documents (brochures, teacher handbooks, policy manuals, meeting agendas)." She conducted an intensive study of three public and three private secondary schools, interviewing fourteen private school teachers, fifteen public school teachers; assorted principals, heads and assistants from each school were interviewed at their respective sites.
According to M&F, triangulation is described as a series of strategies that directs both the generation of data and the clarification of findings . They go on to say that the purpose of triangulation in qualitative inquiry is "to provide a rationale for increasing the plausibility of qualitative findings" (M&F, p.28). M&F state that, "if different data sources are used for the study of a particular problem, and if it can be claimed that they are relevant for the problem, the likelihood for the total data set to reflect reliability and validity will be enhanced" (p.27). The different sources listed by Glass have face relevance to the problem and in principle enhance the likelihood of reliability and valididy. In the journal article, however, the only data we can identify by source is from the interviews. A set of hyperlinks allows readers access to them as the findings are being reported. Whenever a quotation is given from an interview, the reader may choose to examine the quoted passage in its full context by clicking with the mouse on an icon at the margin. (Glass has taken the extra step of directing the link to the exact location of each quotation in the interview and highlighting the quotation in bold.) We thus have explicit examples of the "evidence-instances" referred to in M&F's rules and an unprecedented opportunity to assess in detail whether the evidence is "positive for the development of the hypothesis". The thirtyseven interviews are presented in a standardized pattern, clear and accommodating to read. She used many of the interview questions from the Moles (1988) and Blase's (1991) surveys (part of the 'High School and Beyond' survey) used by Chubb and Moe (1991) to develop their index of teacher and administrator autonomy. The use of structured tools that have been previously tested lends credibility to the data set. Participants were selected based on years of teaching experience (at least five ), and years of experience at the present school (at least three). The school sites were chosen so that the constituencies of both the public and private schools were as comparable as possible; in both private and public situations, schools servicing high income families and focusing on academic excellence and college preparation were selected as well as less favoured districts. Glass does not explicitly state how the data will be used in the confirmation process (she did the work before M&F's book was published), but she does share her process of weighing the evidence. For M&F, this consideration is a necessary step in research design that allows for consequent evaluation in relation to qualitative confirmation.
We now assess the Glass study for qualitative confirmation by using the M&F rules.
As noted in the previous section, Rule 1 states that the evidence instances must be positive for the development of the hypothesis. If they constitute a denial of the hypothesis, they disconfirm the hypothesis. In more than thirty interviews with teachers and their principals, from both private and public schools, Glass found that participants from both private and public schools experienced about the same measure of autonomy in their environments or were able to work around conditions that constrained it. The interviews brought out the degree of complexity inherent in the idea of autonomy. Rule 1 is thus satisfied.
Rule 2 states that if the evidence instances constitute a methodologically unique class, they must (minimally) not be contradictory to one another. The interviews constitute just such a class, and the evidence instances Glass presents do not contradict each other. In most journal articles, or even books, we have to be content with the evidence instances provided by the author-always and necessarily a small subset of all possible instances-but here we are able to read all the text of all the interviews. We do not claim to have done so, but each one of us read at least one interview through, explicitly searching for evidence instances that would contradict the hypotheses. We found none. While the researcher does not report explicitly on the data realized from onsite meetings, visits, and analysis of documents, she does refer to the high achievement standards of schools, curriculum content, parent involvement, and how arrangements were made to collect data--information that had to be collected from these sources. Consider, for example, the following statements from Glass's "Findings" (we have put inferences assumed to be made from field notes and the like in bold):
In a private school, new teachers will generally define the curriculum predicated on their own content knowledge and interest. Because of smaller faculty numbers, there may be two or three other teachers with whom to coordinate curriculum; yet each teacher specializes in a particular facet of that content area. While each of the three independent schools in this study have either a middle school or middle and elementary school as part of its organization, students come from a variety of other schools. Consequently, coordination is a matter of interest only within the upper school. Any coordination of curriculum is accomplished within the institution, as described by this private school teacher: (a quotation follows, with link to the interview).
………
This study was conducted in a right-to-work state in which teacher unions are virtually non-existent, but teacher associations are predominant. These associations are seen as variously strong or weak depending on locale. Only one of the three public schools is in a district having a very strong teacher association. Most, if not all, of its teachers are members of the association and quite a few are active in its leadership. The other two schools are in districts that negotiate teachers' contracts with the association, although the faculty are much less active.
It is not possible to confirm or disconfirm these statements from the journal article, because the relevant evidence instances are not supplied. Those that are supplied satisfy Rule 2, but what do we say about the others? We do not know.
The Chubb and Moe (1990) findings are presented, and we could regard them as "background evidence" (Rule 3), but rather than being "adduced for the hypothesis", they are findings to be questioned, and possibly to be disconfirmed. This is an outcome of research that M&F appear not to have foreseen. Glass's opening statement, and her analysis of data is an argument for a disconfirmation of their report. On the other hand, background information from Sedlak (1986) and Ball (1987) are adduced as support for Glass's hypothesis. In our opinion, the evidence of Sedlak and Ball is not sufficient for development of the hypothesis, but neither are they contradictory. Applying Rule 3, we would say the hypothesis is confirmed by Sedlack and Ball but contradicted by Chubb and Moe. The hypothesis would therefore remain as "undetermined" if the Chubb and Moe data were sufficient. This is why Glass argues so strongly that the Chubb and Moe evidence is, to put it mildly, not sufficient. She points out that they present no evidence whatsoever on private schools from the "High School and Beyond" study that is the basis for their arguments. Applying Rule 3, we conclude that the background evidence is itself contradictory and would leave the hypothesis undetermined. It seems as if we have to decide for ourselves how to weight the evidence.
Rule 4 states that, "for a given hypothesis to be confirmed by evidence instances derived from a variety of methodological approaches, the class comprising these statements first should be partitioned into relevant categories" (p.43). In the Glass study, as previously stated, the interview data are internally consistent, and we do not have access to the data from the field notes, onsite meetings and visits or document analysis. Rule 4, therefore, cannot be applied. Glass has woven information in her discussion that makes it reasonable to acknowledge consistency in those data, but the evidence instances are missing that would allow us to make a strong case.
We believe that the Glass study, in employing focused interviews using open ended questions and observations, makes a plausible case for qualitative confirmation of her hypothesis, but applying the M&F rules did not firmly settle the matter. The study explores real life situations with no attempt to manipulate or control conditions. She argues that a high degree of autonomy is experienced by teachers and principals in both private and public schools, and that her findings disconfirm those of Chubb and Moe (1990) that teachers in private schools experience more autonomy than teachers in public schools. Regarding her second hypothesis, she identified six factors associated with autonomy: conflicting and contradictory demands, shared beliefs, layers of protection, a system of laws, funding constraints, and matters of the size of institutions. She concludes that autonomy is a complex process--an issue that does not distinguish the public from the private sector. As her only qualifier, Glass observes that similar organizational effects may not be encountered in schools under the duress of poverty and social dislocation, perhaps seeking to avoid the fallacy of confirming the consequent. It would appear from this example that qualitative confirmation yet eludes us; we revisit this in our appreciation.
Return to the Contents Table.