Reforms , Research and Variability : A Reply to Lois

Lois Weiner (2003) argues that the research reports from High Performance Learning Communities (HPLC) were biased because of the close working relationships between the researchers and the leaders of the Community School District Two (CSD2) reform. Contrary to any claims otherwise, this relationship was quite open and acknowledged. The intent of the HPLC investigation was always to link scholars and practitioners in a new form of research and development in which scholars became problem-solving partners with practitioners. There are important issues about how to profitably conduct such “problem-solving” research. These issues are worth substantial attention from the communities of researchers and practitioners as collaborative research/practice partnerships proliferate. Serious studies of such partnerships are needed, going well beyond the anecdotal attacks offered by Weiner in her article. Dr. Weiner’s (2003) article is at once an analysis of data on demographics and achievement in Community School District Two (CSD2) in New York City and an

attack on the research strategy (and by implication the research ethics) of the High Performance Learning Communities (HPLC) project that I co-directed, along with Richard Elmore and Anthony Alvarado.Her paper begins with what can only be construed as a personal attack on the researchers and practitioners of the HPLC project.The attack is inherent in the title of the paper, in the way quotes are used and in the personal story of Weiner's own interest that threads through the introduction but is never fully documented.Nevertheless, I welcome Dr. Weiner's effort to provide new data and a fresh perspective on work to which we devoted substantial professional effort during the period 1996-2001.
Dr. Weiner points out that CSD2 was not a typical urban district in terms of its demography.She is absolutely right about this.As we have noted in most reports on HPLC research, the district sits in the midst of some of the greatest concentrations of wealth in the nation, and a noticeable (although minority) portion of middle class parents send their children to CSD2 schools.At the same time, the district has large numbers of students of color and families of poverty (as measured by eligibility for free and reduced lunch), as well as immigrant students who are in the process of learning English.
Dr. Weiner's reports of the demographics of the district as a whole and schools within the district roughly match the data we have collected and reported in several papers over the course of our five-year study.Perhaps most important to her argument is that schools in CSD2 did not uniformly represent the demographics of the district as a whole.That is correct.CSD2 contained "rich" schools and "poor" schools, schools with very few children of color and others almost entirely filled with minority students.The variability among schools was never in question.What is important to ask is whether CSD's unique (at the time) system of curriculum leadership and professional development within schools led to learning gains-especially in "high need" schools.
Dr. Weiner addresses this question by comparing CSD2 schools with relatively high need ratings (7 schools in all) with a single high need school in District 25 in a single year on a single test in one subject matter.Overall, the CSD2 schools did not outperform the District 25 school.Dr. Weiner seems to imply that we should therefore conclude that CSD2's program of curriculum and professional development was not effective.Possibly-but it pays to look at more evidence than she provides.HPLC conducted a number of analyses of CSD2 academic performance, both for the district as a whole and school-by-school.Summaries of most of these analyses appear in the 2001 Final Report of the project.(Note 1) One HPLC analysis examined changes over time in reading and mathematics during the period 1992 to 1998-a period in which the CSD2 curriculum and professional program was being put into place and expanded, and during which a stable test in each subject was being used in New York City.In 1993-the first year of New York's renormed math achievement test, just under 70% of CSD2 students were at or above grade level in math; in 1998 , about 82% of CSD2 students were at or above grade level.The story is similar for reading: Scores rose from just under 60% at or above grade level in 1992 to about 72% in 1998.This gradual rise in overall achievement could have resulted from a change in overall district demographics resulting from more middle class students attending CSD schools.But it did not.According to our data, during this period the percentage of students in the district eligible for free or reduced lunch remained stable at about 53%.
The overall improvement also might have resulted mainly from nudging students already near "grade level" over the mark into performance level 3, leaving the students in greatest need behind.To check this, we analyzed achievement quartile-by-quartile wherever such data was available in several successive years on the same test.In the period between 1996 and 2000, the proportion of CSD2 students testing in the bottom quartile in reading fell a bit every year-to a low of just over10% by 2000.Math drops were smaller, perhaps because the CSD2 math curriculum and professional development system was introduced later and might not yet have fully taken root.
Unfortunately, we were not able to obtain detailed enough data on other districts in New York to make comparisons with them.But we were able to use the variability among schools within CSD2 to examine whether the leadership's curriculum and professional development system can be credited with raising achievement, especially for children with the greatest academic need.Using questionnaires and ratings of classroom instructional quality to assess the extent of engagement in the CSD2 program, our studies showed that deep teacher engagement in professional development and faithful implementation of the district's literacy and math programs both raised overall achievement and reduced the connection between achievement and socioeconomic status.It is interesting that a similar finding for mathematics, using a different curriculum but a similar professional development system, was reported for the Pittsburgh Public Schools during the period 1996-98.(Note 2) We are still left with the question of whether there was something special about the mix of students and schools in CSD2 that might have made it easier than elsewhere to effect the kinds of learning changes that the leadership sought.Here Dr. Weiner makes an important contribution in calling to attention the fact that there are important cultural differences among minority and English learning groups.As she points out, several of the high poverty schools in CSD2 were Chinatown schools.She suggests that Chinese immigrants are, in Ogbu's terminology, "voluntary" immigrants, and their children perhaps more likely to participate actively in the opportunities offered by schools.This and other possible cultural difference between the Chinatown schools and other high poverty and minority schools certainly warrant further investigation.Meanwhile, however, Weiner's data make it clear that Asian students and schools were not the only minorities who did well in the CSD2 reform effort.Note, for example that the school with the highest overall academic performance in Dr. Weiner's Table 5 (PS 198 in CSD2), had a population of 52% Hispanic and 26% Black students.Thus, interesting as the "Asian question" is, does not call the overall record of CSD2 into question.This brings me back to what appears to be Dr.Weiner's main point: That the research reports from HPLC were biased because of the close working relationships between the investigators and the leaders of the CSD2 reform.There was never a secret about this relationship.Indeed the intent of the HPLC investigation from the start was to link scholars and practitioners in a (then) new form of research and development in which scholars became problem-solving partners with practitioners and practitioners accepted the responsibility of collecting evidence in as unbiased a manner as possible, using it to refine and-when necessary-alter their theories of action.(Note3) Our goal was to deeply document, analyze and understand the actual practices of the CSD2 reform.We conducted extensive interview and observation studies of professional development and classroom practice as well as the studies discussed here that examined impact on student learning.Variability-among students, teachers and schools-was a central object of investigation and analysis throughout.
HPLC was not a simple undertaking and there were difficulties encountered along the way.Dr. Weiner points to one of them-an unwillingness of many teachers to return a questionnaire when they felt that anonymity could not be guaranteed because of the relationship between the researchers and the district leaders.Problems of that kind are easy to recognize and to address.There are in addition deeper issues about how to profitably conduct such "problem-solving" research, however-for example, the subtle ways in which question are formulated, or avoided, because of the common perspectives that emerge in long collaborations.These issues are worth substantial attention from the research and practice communities as collaborative research/practice partnerships proliferate.Serious studies of such partnerships are needed, going well beyond the anecdotal attacks offered by Dr. Weiner in the opening sections of her paper.
At the same time, it is critically important to conduct more "arm's length" research on reform programs or other interventions that appear to be succeeding.The HPLC project did not claim to be an arm's length investigation, but all parties to it would welcome such investigations.CSD2 as such no longer exists (having been absorbed into a much larger instructional Region in New York's recent school reorganization).But many of the ideas pioneered in CSD2 are now being tried in districts across the country.There is, thus, plenty of opportunity both for collaborative problem-solving research of the HPLC variety and more arm's length evaluations.Such studies will tell us what aspects of the CSD2 effort can "travel" well to other environments, what effects they have on various populations of students and educators, and-of utmost importance-what revised or totally new theories of action are likely to meet the demands for increased academic achievement across a broad spectrum of the school population.The final report also pays closer attention to CSD 2's demographics and resources and its exceptionality as an urban district .Yet, the analysis still falls short.In this response I present newly analyzed census data that suggests why use of achievement scores disaggregated by race and ethnicity is essential to make a compelling case that the rise in test scores was not caused by the influx of different kinds of students.I conclude with a response to the characterization of my article as a "personal attack" that stems from my unexplored "personal interest," suggesting that the disagreement is primarily political and personal.

Demographics And Test Scores
Responding to my suggestion that changes in test scores in District 2 may be due to alterations in its demographic, Dr. Resnick notes: This gradual rise in overall achievement could have resulted from a change in overall district demographics resulting from more middle class students attending CSD schools.But it did not.According to our data, during this period the percentage of students in the district eligible for free or reduced lunch remained stable at about 53%.
Here as in the final report, Dr. Resnick uses data about stable poverty rates in the district AS A WHOLE.During the period of time marked by rising test scores, 1993-1998, two striking demographic changes occurred simultaneously in CSD 2. The median income in Chinatown fell, dramatically, and the population of Lower Manhattan soared and gentrified.A new analysis puts the change this way: TriBeCa and Chinatown are divided by only one street--Broadway.However, the disparity between the rich on the west side of Broadway, and poor on the east side of Broadway, grew in the last decade.According to a newly released scholarly analysis of census data, the median household income in TriBeCa increased $20,000 in the last 10 years to about $90,000.
As the quote above indicates, census data must be re-analyzed to understand changes in neighborhoods in New York City because the units of measurement in the census do not correspond to neighborhoods -or to school districts.Still, the analysis that has now been done, cited above, indicates the reasons researchers' use of an overall figure for CSD 2's level of poverty may be misleading.

School-Wide Data Mask Within-School Segregation
Achievement data disaggregated by race, ethnicity, and poverty district-wide and within each school seems to me essential to make the case that CSD 2 has succeeded in boosting achievement of Black and Hispanic students for another reason.Research on CSD 2 has used school-wide achievement data but this may mask different levels of achievement within schools, due to the presence of gifted and talented programs, programs that enroll more White, middle class students.Employing a method of testing for racial bias used widely to detect discrimination in housing, ACORN, the community advocacy organization, sent African American and White testers to New York City public schools to inquire about placing their children in gifted and talented programs and concluded that African American and Latino parents were steered away from these programs (ACORN, n.d.).The evidence was brought to the attention of the U.S. Department of Education, civil rights division, which entered into a Memorandum of Understanding with the NYC Board of Education in 1998 to assure equal access (Advocates for Children, n.d.) The ACORN report comments on these practices in CSD 2: At PS 116 (CSD 2), the request of the white tester to see some classes 'was granted graciously.'The black tester, who was not shown any classes, was told that classrooms were usually viewed in group tours.At another CSD 2 school, PS 11, an office worker insisted that a black tester could not possibly live at the address she presented (London Terrace), implying that it was not a "black" address; the tester was then given no information about the school or its gifted program.A subsequent white tester had no problem presenting a London Terrace address.(n.p.) At PS 116 (CSD 2), the principal displayed surprise that the white tester was inquiring about the regular program, explained the gifted program and, despite the fact that the program has a waiting list, provided an application form and told her she could have her child tested at a number of locations.The white tester was even given listings of private school bus services that provide transportation to the school from the Upper East Side as well as from the Upper West Side --which is not part of District 2. The black tester at PS 116 was also treated cordially but had to inquire about the gifted program and was told by the office person that registration did not guarantee acceptance into the program.At PS 11 (CSD 2), the white tester was even provided with a list of private testing services whom she could call.(n.p.) My article proposed that PS 11's scores required close examination because the school's scores were reported in the aggregate, a factor that masked the presence of a segregated gifted and talented program.This information is in the section that Resnick refers to as "anecdote" but the ACORN report confirms the information I gathered from my informants.
"Anecdote" Or "Description Of The Context"?
I do not dispute Dr. Resnick's observation that information I gathered from a relatively small number of teachers and administrators in CSD 2 can be characterized as "anecdotal."I agree that it is suggestive rather than conclusive.However, I contend that these interviews constituted the sort of preliminary field work that researchers regularly use to identify external threats to a study's validity.
Another illustration of the reasons "anecdotal" information of the sort I gathered is valuable is the case of PS 198, a school to which Dr. Resnick calls our attention: Note, for example that the school with the highest overall academic performance in Dr. Weiner''s Table 5 (PS 198 in CSD2), had a population of 52% Hispanic and 26% Black students.Thus, interesting as the ""Asian question"" is, does not call the overall record of CSD2 into question.
A commentary piece in Education Week, written by a literacy volunteer at PS 198 who was critical of the "balanced literacy" initiative, and the exchanges that ensued between Anthony Alvarado and Elaine Fink, suggest that use of PS 198 as an exemplar of the success of CSD 2's literacy program is problematic.(Note 2) The literacy volunteer, Louisa Spencer, observes "Soon after coming to PS 198, I had heard that its remarkable recovery in reading scores had owed a great deal to its principal's insisting on the introduction of Open Court in the early grades.Ms. Harwayne's exposition of the district balanced-reading program included no such ingredient, however, and, instead, presented with great artistry the merits of the district's system, denying the reality of any dichotomy between phonics and "whole language" methods and explaining how the district seamlessly combines them. But

Political Critique
My critique is not intended to minimize the difficulties that arise in collaboration between researchers and school officials, or the importance of trying to develop new sorts of cooperation, especially in urban districts.However, I maintain that collaboration demands the stance from university-based researchers of "critical friendship."The insular nature of the relationship between the researchers and CSD 2 officials, including union leaders, as articulated in the "nesting doll" study design, seems to have undercut the aspect of critique that was needed.
I regret that Dr. Resnick considers my analysis of research on District 2 a personal attack.That is certainly not my intent.The discussion in my article is of research formulated and reported by the principal investigators of the studies, including but not limited to Dr. Resnick.Her biography makes clear that as a conscientious scholar she takes personal responsibility for research published under her name.My article criticizes work for which she is a principal investigator.I acknowledge that the article may be unusual in the sharpness of its political criticisms of the research but it leaves the calculation of the moral dimensions of the researcher's roles and decisions to readers.
My initial motivation for examining research on CSD 2 was an interest in determining why a school with high test scores was under "surveillance," a term that I acknowledge is is pointed but describes the intense scrutiny and discipline of school people enforced through the "walk through."I fully described my relationship with the school elsewhere, in a publication cited in the article (Weiner, 2002).I submit that my interest in research on CSD 2 is no more "personal" than Dr. Resnick's in collaborating with CSD 2 officials.However, I do have a political reason for pursuing this research, one unconnected to the curricular content of the model, on which I have offered no opinion beyond suggesting that all "one size fits all" curricular mandates fail to take into account the importance of context (Weiner, 2002).One political concern I have about CSD 2's model is its presumption that professionals should control schooling (Weiner, 2002).CSD 2's assumption is challenged by a new research synthesis about benefits that accrue from schools' respect of knowledge that families bring to children's schooling (Henderson and Mapp, 2002).
Another political issue sparked my study of research on CSD 2, my concern about fear among teachers and administrators in CSD 2 schools who felt inadequately protected by their union.Children who are taught by fearful teachers do not learn how to think critically or to be courageous in defense of unpopular ideas or beliefs, and I believe this is as important to schooling as what is contained in any math or literacy curriculum.An essential role of research, especially in bureaucratic, hierarchical school systems, is to make elbow room for critique by asking very hard questions and pressing school officials to do the same.Research that fails to uncover tacit cultural assumptions, for instance about whose knowledge counts, can miseducate talented, conscientious school officials with whom researchers collaborate because the research deflects attention from very difficult topics, ones essential to reform, like the variability of student achievement associated with race and ethnicity.
I hope that understanding of CSD 2 as a model for urban school reform has been refined in this exchange of sometimes sharp differences.Just as important, I hope we have modeled and made space in urban schools for similar discussions among families, teachers, and administrators.I thank Dr. Resnick for responding to my article and thus joining with me in this project.

asu.edu Abstract Dr
This form of Problem Solving Research and Development was recommended in a National Academy of Education report to OERI as one important means of bringing research and practice into closer interaction See Brown, A.L. & Greeno, J.G. (Eds.)Recommendationsregarding research priorities: An advisory report to the National Educational Research Policy and Priorities Board.National Academy of Education.Spring, 1999.Weiner, L. (2003, August 7).Research or "cheerleading"?Scholarship on Community School District 2, New York City.Education Policy Analysis Archives, 11(27).Retrieved [date] from http://epaa.asu.edu/epaa/v11n27/.Resnick is an internationally known scholar in the cognitive science of learning and instruction.Her recent research has focused on school reform, assessment, effort-based education, the nature and development of thinking abilities, and the relation between school learning and everyday competence.Her current work lies at the intersection of cognitive science and policy for education.Dr.Resnick founded and directs the Institute for Learning, which focuses on professional development based on cognitive learning principles and effort-oriented education.She is co-founder and co-director of the New Standards Project, which has developed standards and assessments that have widely influenced state and school district practice.Resnick was a member of the Commission on the Skills of the American Workforce and served as chair of the assessment committee of the SCANS Commission and of the Resource Group on Student Achievement of the National Education Goals Panel.She has served on the Commission on Behavioral and Social Sciences and Education and on the Mathematical Sciences Education Board at the National Research Council.Her National Academy of Sciences monograph, Education and Learning to Think, has been influential in school reform efforts, and her widely circulated Presidential Address to the American Educational Research Association, "Learning In School and Out," has shaped thinking about youth apprenticeship and school-to-work transition.Dr.Resnick is Professor of Psychology at the University of Pittsburgh, where she directs the Learning Research and Development Center.Educated at Radcliffe and Harvard, she received the 1998 E. L. Thorndike Award from the American Psychological Association and the 1999 Oeuvre Award from the European Association for Research on Learning and Instruction.The World Wide Web address for the Education Policy Analysis Archives is epaa.. Resnick's response to my article and HPLC's final report on CSD 2 shows we concur about two key concerns.One is the difficulty and importance of developing modes of research that allow university-based scholars to assist school officials.Another is attention to variability in achievement levels associated with students' race and ethnicity.However, her response still fall short in taking into account external threats to the validity of the study's conclusions that CSD 2's model of reform has raised achievement and is exportable.I present newly available census data that suggests why use of data disaggregated by race and ethnicity is essential to make a compelling case that the rise in test scores was not caused by the influx of different kinds of students.I suggest that while pointed, the article's critique is political in nature and does not constitute a personal attack on the researchers' whose work has been scrutinized.I propose that airing differences among researchers on difficult questions in urban school systems can model the kind of frank exchange that should occur when researchers and school officials collaborate, so as to make space for dissenting viewpoints in schools.I appreciate Lauren Resnick's thoughtful response to my critique of research done in and on District 2, and I hope that other scholars whose work has proposed that CSD 2 is a model for urban school reform, including Dr. Elmore, will join in this discussion.(Note 1) Dr. Resnick and I concur on two key issues.One is the difficulty and importance of developing modes of research that allow university-based scholars to assist school officials.Another is the importance of taking into consideration the variability in achievement levels associated with students' race and ethnicity.In this second regard, HPLC's final report is considerably more nuanced in its characterization of District 2's success than the earlier reports (HPLC Final Report, 2001).

from 25, 366 to 34,420 but
The area that spans Lower Manhattan (Community District 1), including Tribeca, is part of CSD 2. Census data indicate that from 1994-2001 the total population in this area increased the percentage of the population receiving public assistance declined from 33.5 to 6.8%.A sharp drop in the number of people receiving AFDC also occurred in Community District 2, which encompasses Greenwich Village and Chelsea, while the total population remained about the same ( New York City Department of Planning, Community District Profiles, n.d., online)If one looks only at the CSD 2's overall poverty level, one misses this change.An increased population in Chinatown schools of poor immigrant students would have raised the numbers of CSD 2 students in poverty.But the increase in numbers of poor Chinese students would have been counterbalanced by the influx of students from wealthy and middle class families, for instance in Tribeca and Chelsea.The census data is suggestive that the stable overall poverty rate for CSD 2 may mask very significant demographic changes that account for some part of the increase in test scores over the decade.