Noah Smith on Natural Experiments

He writes,

With lab experiments you can retest and retest a hypothesis over a wide set of different conditions. This allows you to effectively test whole theories. Of course, at some point your ability to build ever bigger particle colliders will fail, so you can never verify that you have The Final Theory of Everything. But you can get a really good sense of whether a theory is reliable for any practical application.

Not so in econ. You have to take natural experiments as they come. You can test hypotheses locally, but you usually can’t test whole theories. There are exceptions, especially in micro, where for example you can test out auction theories over a huge range of auction situations. But in terms of policy-relevant theories, you’re usually stuck with only a small epsilon-sized ball of knowledge, and no one tells you how large epsilon is.

Pointer from Mark Thoma. Read the whole post.