Against the Null Hypothesis?

1. Jo Craven McGinty in the WSJ:

Decades after the end of legalized segregation, and the funding disparities that accompanied it, minority students remain disproportionately concentrated in high-poverty areas. Academically, they trail students in more affluent areas, and they fall increasingly behind as the years pass. The result is an achievement gap that limits the educational and career opportunities of nonwhite children.

She refers to a study by Sean F. Reardon and others. The abstract reads

In this paper we estimate the effects of current-day school segregation on racial achievement gaps. We use 8 years of data from all public school districts in the U.S. We find that racial school segregation is strongly associated with the magnitude of achievement gaps in 3rd grade, and with the rate at which gaps grow from third to eighth grade. The association of racial segregation with achievement gaps is completely accounted for by racial differences in school poverty: racial segregation appears to be harmful because it concentrates minority students in high-poverty schools, which are, on average, less effective than lower-poverty schools. Finally, we conduct exploratory analyses to examine potential mechanisms through which differential enrollment in high-poverty schools leads to inequality. We find that the effects of school poverty do not appear to be explained by differences in the set of measurable teacher or school characteristics available to us.

The last sentence appears to support the Null Hypothesis. But the study and McGinty’s interpretation clearly assume that placing a child in a high-poverty school will worsen that child’s educational outcome. One possibility is that peer effects are strong, as the late Judith Rich Harris claimed. Another possibility, more consistent with the Null Hypothesis, is that the relationship between neighborhood poverty and school outcomes is not causal.

McGinty cites a paper by Raj Chetty and others that finds that moving to a different school affects educational outcomes. If these results are causal, then this would be evidence against the Null Hypothesis. But I am not convinced that moving to a different school district is unrelated to the characteristics of the parents and hence of the children.

2. Angelo Codevilla writes,

During Word War II, only 4 percent of some 18 million draftees were illiterate. Despite (or because?) of massive expenditures on education over the subsequent two decades, 27 percent of the Vietnam war’s draftees were judged functionally illiterate. Between 1955 and 1991, the inflation-adjusted average K-12 per-pupil expenditure in America rose 350 percent. In 1972, 2,817 students scored 750 or better on each half of the SAT. By 1994, only 1,438 made this score though the test had been made easier. Today, U.S. 15 year olds rank 24th out of 71 countries in science, and 38th in math. In 2018, college students spent less than a third of the time their grandparents did studying for their classes.

If you believe the Null Hypothesis, then this must be due to a worsening the innate characteristics of American children. To blame the education system, you have to believe that the Null Hypothesis is not true, and that the education establishment has found ways to achieve worse outcomes.

Either possibility is distressing.

29 thoughts on “Against the Null Hypothesis?

  1. Wasn’t the identification strategy of the Chetty paper very strong? I wouldn’t just dismiss it.

  2. Re: per pupil expenditures rising. Isn’t most of this due to health care costs for teachers increasing? That and the Baumol story could explain how expenditures went up a lot with no improvement in outcomes. It’s not that we are doing anything to improve education for kids- it’s just the inputs becoming more expensive. So I don’t think it’s evidence for the Null hypothesis.

    • It’s also that there are more inputs. Student/teacher ratios are down. And there are often counselors, psychologists, “paraprofessionals” aka aides, etc.

      • Partly in response to the failure of many students to do well in schools, the people who think about them or run them have decided that it is simply not enough for a teacher to “teach” and expect all students to learn. Rather, schools must take care of the whole student; otherwise, they cannot be expected to learn up to their potential. That means counselors and psychologists. It also means smaller classes. And since “inclusion” is now part of educational theology, it also means aides and “accomodations”–and in some cases “out-of-district placement”. All of this costs money.

  3. The problem with poor performing schools is that they have poor quality students and the associated families from which they come. I think everyone really does understand this, but most can’t say so for reasons having to do with political correctness. The best way to fix the poor performing schools is to stop mandating attendence- the poor quality students will just stop attending, and it won’t really affect how they turn out as adults anyway- those things are set by their genetic endowment and their parents, or lack thereof.

  4. Please enlarge upon:
    “the educational establishment.”

    What is it; how does it function; what is its hierarchy; who are its political and operational functionaires (and what are their objectives).

    • There is an immense literature on this.

      For starters I suggest the writing of E.D. Hirsch, more specifically,

      _The schools we need, and why we do not have them_. 1996. It seems to have a new edition I’m not familiar with.

      If I recall off the top of my head, Hirsch discusses something called “The Blob” that may go back to the writings of Bestor 40 or 50 years ago. I believe the Blob was defined as an “interlocking directorate” of (1) teachers colleges, (2) administrators, and (3) state licensing boards.

      https://www.penguinrandomhouse.com/books/80409/the-schools-we-need-by-ed-hirsch-jr/9780385495240/

      I don’t think the establishment has a manifesto. Offhand the objectives seem to be

      (1) maximizing revenue
      (2) minimizing competition
      (3) maintaining barriers to entry by outsiders, and
      (4) stonewalling critics.

      It’s too bad because even without all that our schools might not be great.

  5. Another possibility is our culture is weaker and not giving kids the right tools to succeed.

  6. This reminded me of this abstract Tyler Cowen posted about back on July 9th. The paper, from 2012, looked at the ending of race-based busing with some students reassigned to neighborhood schools.

    We find that both white and minority students score lower on high school exams when they are assigned to schools with more minority students. We also find decreases in high school graduation and four-year college attendance for whites, and large increases in crime for minority males. The impacts on achievement and attainment are smaller in younger cohorts, while the impact on crime remains large and persistent for at least nine years after the re-zoning. We show that compensatory resource allocation policies in CMS likely played an important role in mitigating the impact of segregation on achievement and attainment, but had no impact on crime. We conclude that the end of busing widened racial inequality, despite efforts by CMS to mitigate the impact of increases in segregation.
    That is from Stephen B. Billings, David J. Deming, and Jonah E. Rockoff.

  7. As for the study of draftees, weren’t college kids with high GPAs exempt from the Vietnam draft? I’m guessing the WWII draftees were closer to a random sample than the Vietnam draftees, but someone can correct me if I’m wrong.

  8. I feel like the “strong” null hypothesis in education is that absolutely nothing in education matters at all long term.

    But I think a more plausible “weak” null hypothesis is that there’s nothing a given school system can do to permanently improve outcomes for a given cohort of students. The weak version would permit peer, family, neighborhood and society effects, but still be quite an interesting result.

    It’s also possible that education outcomes vs education quality follows a sort of sigmoid curve. So low quality institutions produce low quality outcomes and high quality institutions produce high quality outcomes. But the curve is also mostly flat at the tails so even large changes in quality for a high or low quality school often makes little difference in outcomes. This kind of thing would create the appearance that the null hypothesis is true (especially if we, say focus on suburban schools) but still leave us with plenty of data showing that interventions can make a difference when schools or students cross from the low quality to the high quality regime.

  9. To be fair, the education establishment did find ways to achieve worse outcomes.

    “Whole word” reading instruction

    “A working paper found that Common Core had significant negative effects in grade 4 reading and grade 8 mathematics based on National Assessment of Educational Progress scores. The size of the negative effects were generally small.” –Wikipedia

    Not to mention the movement to let classes be disrupted due to “modern” ideas about discipline of students. The “sage on the stage” model, which places the teacher paramount, magnifies the impact of disruptions requiring teacher attention and action as the model inculcates “school helplessness” in all students so when the teacher is diverted the other students are left without skills to continue on their own.

  10. I agree with the poster who said that it’s both. Charles Murray’s book on “Coming Apart” about the changes in white America from 1960 to 2010 shows the breakdown of family and community. On the other hand, the USA is a much more racially diverse country that it was in 1960, and that has an impact even when one accepts the null hypothesis.

  11. “During Word War II, only 4 percent of some 18 million draftees were illiterate. Despite (or because?) of massive expenditures on education over the subsequent two decades, 27 percent of the Vietnam war’s draftees were judged functionally illiterate. ”

    I’m sorry, but anyone who believes that the number of functional illiterates increased sevenfold in a quarter of a century shouldn’t be writing about education.

    “In 1972, 2,817 students scored 750 or better on each half of the SAT. By 1994, only 1,438 made this score though the test had been made easier.”

    Same idiot is wrong again. The test was renormed the year after. Yes, scores dropped. We had millions more taking the test, as opposed to just the top students.

    And international comparisons are moronic given that we educate all our students, not just the top ones.

    The harsh truth is that other countries are copying our educational systems, including educating more of their populations, even though it does make stats look less strongl.

    By the way, the whole language/phonics debate isn’t responsible for much. Phonics is generally taught in most schools, and the case for content-based curriculum being the solution is overstated.

    • Same idiot is wrong again. The test was renormed the year after. Yes, scores dropped. We had millions more taking the test, as opposed to just the top students.

      I am not sure the evidence cited is conclusive here- they only described the numbers of scores which topped 750 for both sections in 1972 and then in 1994, and if the numbers cited were correct, there was a drastic drop in the absolute numbers scoring that high in 22 years, even with millions more taking the test.

  12. Everybody wants to move to where there are “good schools”.

    Where is the research about why the schools next-to the good schools aren’t also just about as good?

    Why aren’t the good schools expanding?
    Theory – the parents are getting worse, as parents.
    Both genetically and thru worse nurturing in a schooling-relevant way.

    Probably can’t improve schools much without FIRST improving the parents?

    • To a first approximation good students make good schools. What are good students? They are relatively bright and conscientious, and they buy in to the need to pay attention, not cause disruption, and try to at least pass. They tend to have parents who are the same way and who are willing to tax themselves to pay above average for schools. (Because they are bright, conscientious, etc., they probably also have more money to spend.) Outsiders will see the local culture as pro-education.

      Everyone will then get the direction of causation wrong and think that a community which is pro-education and spends a lot of money on its schools creates good schools.

      If one town has “good schools” and the next town doesn’t, the reason is probably that the first town has a lot more good student material than the other.

  13. Yeah it’s no that the students are more focused on basketball, football, it’s that the schools are bad for academics but they have great basketball and football coaches.

    • Also the phrase “functionally illiterate” can have broad and variable meaning. I doubt that the level of literacey changed that much. In fact I bet people are slightly more literate today than ever.

Comments are closed.