On ideology, causal inference and the reification of statistical methods: Reflections on "Examining instruction, achievement and equity with NAEP mathematics data."

Abstract

The purpose of this article is to comment on the prior article entitled "Examining Instruction, Achievement and Equity with NAEP mathematics data," by Sarah Theule Lubienski. That article claims that a prior article by the author suffered from three weaknesses: (1) An attempt to justify No Child Left Behind (NCLB); (2) drawing causal inferences from cross-sectional data; (3) and various statistical quibbles. The author responds to the first claim, by indicating that any mention of NCLB was intended purely to make the article relevant to a policy journal; to the second claim, by noting his own reservations about using cross-sectional data to draw causal inferences; and to the third claim by noting potential issues of quantitative methodology in the Lubienski article. He concludes that studies that use advanced statistical methods are often so opaque as to be difficult to compare, and suggests some advantages to the quantitative transparency that comes from the findings of randomly controlled field trials.