Null hypothesis watch

A report from RAND on a Gates Foundation experiment called the Intensive Partnership for Effective Teaching (IP) to try to help low-income minority (LIM) students.

Overall, the initiative did not achieve its stated goals for students, particularly LIM students. By the end of 2014–2015, student outcomes were not dramatically better than outcomes in similar sites that did not participate in the IP initiative. Furthermore, in the sites where these analyses could be conducted, we did not find improvement in the effectiveness of newly hired teachers relative to experienced teachers; we found very few instances of improvement in the effectiveness of the teaching force overall; we found no evidence that LIM students had greater access than non-LIM students to effective teaching; and we found no increase in the retention of effective teachers, although we did find declines in the retention of ineffective teachers in most sites.

Pointer from Tyler Cowen.

20 thoughts on “Null hypothesis watch

  1. Not surprising. This is a really, really, hard problem so I’d expect more failures than successes. Hopefully, the response to this is “Let’s figure out what went wrong and try something else”.

    But at least people are tackling the problem, rather than using the problem for their own political gain.

    • From the link, I couldn’t tell whether they were dealing with elementary, middle, high school, or all three. If they are dealing with elementary school, with basic literacy and numeracy, I agree “Hopefully, the response to this is “Let’s figure out what went wrong and try something else”.” Being able to read and write and understand numbers is important for just about every citizen, and maybe achievable.

      However, if they are talking about getting everyone to legitimately master the academic skills and knowledge embodied in states’ high school education standards, my response is, “Stop! Maybe half the country’s young people can’t do that and trying to force them to only leads to unhappiness and lies, unhappiness for the students and lies for the school system that has to pretend they have achieved what they haven’t.”

      Rather, the response should be to do things that will be useful for people who are not academic, to stop looking down on people who aren’t academic, and to stop punishing them for their lack of going along with the pre-college that we pretend is all that education can be.

    • Hopefully, the response to this is “Let’s figure out what went wrong and try something else”.

      There is no problem to solve. You can’t change IQ. The “solution” is to accept this and stop throwing time and money at unsolvable problems. Use such time and money in way that has better ROI.

      I would prefer we took the money spent on failed educational ventures and just wrote every American in the country a check. 25k per kid (NY per pupil spending) for glorified daycare is a crime against humanity. Give the kid a downpayment on a house for christ sake. I think that would make the world a better place.

      • There is no problem to solve. You can’t change IQ.

        Consider this statement: IQ is immutable, therefore, all education is a waste of time and money.

        You are conflating IQ with knowledge, asdf. If you believe literacy and numeracy matter to a society then it follows that the efficacy of any given education program is relevant.

        • It sure is. Maybe you two can actually agree. There are important things that just about anyone can learn. Just about everyone has an IQ high enough. We should do what we can to see that such learning takes place.

          However, there are lots of things that lots of people can’t learn–and even the massive amounts of drill and repetition that would be necessary to get them to pass a test–and then forget!–just aren’t feasible (partly because the young people would turn off and then passively resist or “act out”).

          Right now my feeling (based on reading and years as a high school teacher) is that no more than half of the high school population can legitimately master what they are supposed to.

        • Per Roger, you are making a statement “all education is useless” which is purposely meant to engage in misinformation, sensationalism, and drive an agenda (and you should feel guilty about it).

          Let’s get two things straight:

          1) Kids in high school (and much earlier*) have either learned “literacy and numeracy” already, or they are never going to be able to learn it no matter what you do.

          2) Does it take 25k/kid/year to teach “literacy and numeracy” to an adequate degree within the child’s ability level?

          It seems obvious to me that these basic skills, to the extent one is capable of them, can be taught for less money and over less time than you are advocating for.

          Let me be straight. You are advocating for an expensive and ineffective educational racket for evil reasons and using electrified rhetoric that we are trying to keep kids from being able to read and do their multiplication tables as a complete lie to keep that racket going. That is monstrous.

          *
          https://www.greatschools.org/gk/articles/k-5-benchmarks/

        • asdf, I think you could be more charitable 🙂 Maybe I’m naive but I read RAD as saying that literacy and numeracy are important and worth spending money on. I think we all agree on that. Now how much and for how long …

          • I lack charity because I’m exhausted.

            I’m tired of the same mistakes being made over and over again. I’m tired of the people making those mistakes blaming others. I’m tired of having an ideology based on misinformation shoved down my throat.

            It’s not just academic or a game. Big important decisions that affect peoples lives get made based on this stuff.

        • you are making a statement “all education is useless” which is purposely meant to engage in misinformation, sensationalism, and drive an agenda (and you should feel guilty about it).

          Ummmm, no. I was trying to use an extension of your logic “You can’t change IQ”, therefore, “There is no problem to solve” to emphasize that changing IQ is not a relevant factor.

          I obviously failed to communicate my intent. My bad.

          • The context of the Null Hypothesis isn’t “are we going to teach our kids basic reading and writing”. All but the lowest IQ people can learn those things by puberty.

            It’s “can this or that educational policy achieve goal XYZ”. Where goal XYZ is always something like “close the gap” or “make everyone college ready” or “have and ROI on education spending that will solve our societal problems.”

            Since I believe IQ makes all of those goals impossible, it’s very relevant.

          • See, you do agree. 🙂

            As I see it, there are two foundational questions in education:

            1) What CAN young people learn? My heterodox answer is a lot less than most everyone in the business assumes.

            2) What SHOULD young people learn? Just about no one in the business asks this question. Instead, they automatically think, “Whatever will prepare you to enter college 13 years after you enter kindergarten.”

  2. But it did succeed in getting less effective teachers to leave which isn’t nothing. I didn’t read the whole report so I have no idea whether the number leaving was tiny anyway. The notion that this proves teacher quality is irrelevant seems premature. And it seems vaguely like this model put more responsibility in administrators hands than administrators are capable of handling. School administrators don’t seem to lack competence generally as illustrated elsewhere this morning: https://www.joannejacobs.com/2019/12/violent-students-keep-coming-back/

    • If getting rid of “ineffective teachers” didn’t make much difference to student achievement, I might have problems with their definition of “ineffective”. Kind of like “the operation was a success but the patient died.”

  3. Is a good response to this, we do not know how to teach students much more, so maybe we should focus on teaching them the most valuable stuff for them to flourish.

    Auto mechanics might be more valuable for the average student than a foreign language as taught on our high schools. Even if you want a foreign language conversational might be better than teaching conjugation of the verb to be.
    Maybe statistics over factoring quadratic equations.

  4. The district school results don’t surprise me – The three districts in the study are state-run monopolies and taylor-derived management practices won’t work in these monopolies because the accountability/governance models are contrived (compared to more authentic school choice choice models).

    The charter schools on the other hand….need to take a deeper look t the study.

  5. I am not seeing student growth/value added results for the CMOs in the studies so not sure report lends support to the null hypothesis.

    Also, these CMOs seem poorly managed. “Working jointly, [these CMOs] also developed a common observation rubric and observation process and used common stakeholder feedback measures.”

    Not sure why a leader of an organization would use the same (identical!!) human capital system found in another organization. they ceded control of their most important process to a consortia. Seems odd. Like google and amazon evaluating their engineers the same way in spite having different goals, strategies, cultures, and contexts.

    Again, I don’t think this adds to the null hypothesis. But it does show that some CMO leaders let themselves get sucked up into misguided initiatives – perhaps for the foundation funding.

  6. With what has been done to the “education” system over the past twenty years, the problem is NOT IQ. It is instead “culture”. Third world culture, and fourth world culture, to be specific.

  7. This problem has been developing for many decades, at least since the 1950’s. I know this because I was one of the children who had trouble learning to read; I was sent home one summer (after first grade?) with little beginning readers to practice on over the summer. While I did manage to figure out reading, too much was unconscious for me to be able to spell. At one point, my mother askednif she told me a word was spelled, and she gave a string of consonants, would I write that down. I had no conscious idea that there should be a correspondence between the letters and the word. By some fluke, I found the books, Why Johnny Can’t Read and Reading: Chaos and Cure. After reading these, and going thru the exercises in the back of one of them (very boring for an above average reader), I became a poor speller and am now, decades later, a mediocre one.
    Fearing similar problems for my child, I taught him to read by sounding out words in a McGuffy’s Reader when he was in nursery school. After several months of lessons, he could read anything he wanted to.
    So far as I can see, the whole word method of teaching reading is like going to California from Chicago by heading east, a waste of time and effort.

    https://www.americanthinker.com/articles/2019/12/k12_are_low_test_scores_a_problem.html

  8. if gates wants to help minority kid’s education to improve tell the adults to raise the children they have. there is a direct correlation between two parent homes and educational performance. the people with the greatest percentage of two parent families do the best in school. the people with the least number of two parent homes do the worst in school. it is the elephant that no leftist wants anyone to recognize. it would hurt the left’s eternal goal of destroying the family.

  9. Tyler Cowen links to a Vox piece today that appears relevant to this article as well as the issue of scalability as well. The article includes the following:

    2. ”Using the Results from Rigorous Multisite Evaluations to Inform Local Policy Decisions” (2019) by Larry Orr, Robert Olsen, Stephen Bell, Ian Schmid, Azim Shivji, and Elizabeth Stuart

    The Cohen/Dupas paper is in some ways the best possible case for randomized trials being valuable. This paper, published this past spring, is the best counter case I’ve seen.

    Focusing on education, this team of researchers tries to use average results of education policies, as measured by big randomized trials held in different locations, to predict the results in individual locations. They find that this doesn’t work very well at all: you can’t just take average results and expect that the same effect will hold in your specific case. It’s a challenging result for evidence-based policy and one I’m still grappling with.”

    Could the scalability issue reflect limitations of the analytics?

Comments are closed.