succeeding or failing in a single trial doesn’t usually constitute adequate evaluation of a program. Rather, promising ideas need to be subject to iterative evaluation in the relevant contexts.
succeeding or failing in a single trial doesn’t usually constitute adequate evaluation of a program. Rather, promising ideas need to be subject to iterative evaluation in the relevant contexts.
I don’t know how unique Manzi’s thinking is among data scientists, but among policy analysts, it’s like a radical breath of fresh, sane, humble, serious air. He lends great weight to the curious Hayekian task.
I am surprised there aren’t more comments. Manzi is pointing out a real limitation to our knowledge and what can possibly be ascertained. I suspect that the practitioners of statistical analysis haven’t really thought about the big picture and have become captured by the glamor of their software and their data. If they were to recognize the limitations of their analysis, it would also put a brake on a lot of economic policy. I don’t see that happening.