Significance Comparisons and Measurement Error

Leilan Shu and Sara Dada report,

We first use a simple linear regression model of average test score and average household income to first establish a positively correlated relationship. This relationship is further analyzed by differentiating for other community-based factors (race, household type, and educational attainment level) in three multiple variable regression models. For comparison and to evaluate any consistencies these variables may have, the regressions were run on data from both 2007 and 2014. In both cases, the final multiple regressions found that average household income was not statistically significant in impacting the average test scores of the counties studied, while household type and educational attainment level were statistically significant.

Pointer from Tyler Cowen. If this were credible, it would seem to suggest that “schooling inequality” is really ability inequality.

BUT…Whenever somebody says that “X1 does better than X2 at predicting Y,” watch out for the impact of measurement error. A variable that is measured with less error will drive out a variable that is measured with more error.

In this case, suppose that the variable that matters is “parents’ resources.” Income could measure that variable. Educational attainment could predict that variable. Income has many sources of measurement error–if nothing else, one year’s income could be high or low due to volatility. Educational attainment has fewer sources of measurement error. So even if parents’ resources is the true cause of children’s test scores, you could wind up with a zero coefficient on income, particularly if you include another regressor with lower measurement error.

And this is one of many reasons to prefer experimental data to regressions.

2 thoughts on “Significance Comparisons and Measurement Error

  1. If this were credible, it would seem to suggest that “schooling inequality” is really ability inequality.

    It isn’t completely but that’s the major component–at least in the USA where very few schools plumb the depths; no schools where teachers don’t show up half the time or sit in the front and read from a textbook for an hour.

    Perhaps the most depressing book I ever read was Robert Weissberg’s Bad Students, Not Bad Schools(Transaction, 2010), which makes an uncomfortably good case for the title.

  2. ” If this were credible, it would seem to suggest that “schooling inequality” is really ability inequality”

    Perhaps that “ability” conforms to Amartya Sen’s “capacity equality'” which is the focus of his 1979 Tanner Lecture.

    That would bring into focus the nature of capacity on which to “build’ ability.
    The concentration of “education” on a goal of creating or building “ability” without regard to capacity has long been part of the “Great Conceit.”

Comments are closed.