Good Models Fail in Interesting Ways

Consider three monetarist models.

(1) If the Fed reduces the rate of growth of high-powered money relative to recent trend, this will slow the growth rate of nominal GDP.

(2) If the Fed raises the Fed Funds rate above the interest rate on medium-term bonds, this will slow the growth rate of nominal GDP.

(3) If the Fed lowers the expected future growth rate of nominal GDP, then this will slow the growth rate of nominal GDP.

Model (3) is less likely to fail, in the sense that lower expected future growth of nominal GDP is very likely to be correlated with lower growth in nominal GDP. However, when model (1) or model (2) fails, the result is interesting. Because model (3) is not likely to fail in an interesting way, it is, in my view, an inferior model.

Over the years, economists have developed many criteria for evaluating a model. I think that if you ponder the subject, and you review situations where we learn from models and where we do not, you might agree with me that a good model is a model that is capable of failing in interesting ways. Bad models are models where either (a) failure would be met with indifference, perhaps because we already know that several key assumptions are implausible, or (b) where “success” is so heavily built into the model–as in (3) above–that there is nothing to be learned from confronting the model with evidence.

10 thoughts on “Good Models Fail in Interesting Ways

  1. I object on the basis that you are confusing prediction and control. If the goal is to control the path of NGDP, then your criteria are precisely backwards. Now, I know that you are smart and knowledgeable, so there must be some sort of charitable explanation. My guess is that you don’t think the Fed can control NGDP expectations very well. That PSST is what controls NGDP and thus expectations thereof.

    That is the heart of your issue with Sumner I think. Trying to apply your interpretative frameworks meta model in this instance just obscures the issue. Sumner isn’t ultimately about interpretation, he’s about control.

    Note, that he is actually making a very testable prediction, contra your interpretive frameworks model. He claims that if we intervene in the economy in a particular way, we will get particular results.

    • Try to describe a set of observations that could falsify Sumner’s model. If NGDP hits a desired level, then the model is true. If NGDP misses the desired level, then the model is also true, and we just say that the Fed messed up. As a result, we can never learn anything about the model based on observations. We just make claims that the Fed hit or missed, based on the presumption that they cannot miss if they do things right.

      • Umm, no offense, but that’s clearly not the charitable view. He has a very specific proposal to use level targeting with an NGDP futures market controlling OMOs. This is what I meant about prediction versus control.

        If he’s right, actual NGDP consistently ends up close to the target. If he’s wrong, it doesn’t–model falsified.

        • What you are saying is that before the model can be tested (a) the Fed has to announce a level target and (B) there has to be an NGDP futures market. OK, but until that happens I am not going to treat it as a verifiable model.

          • Again, that doesn’t seem very charitable. Sumner has also described smaller incremental steps such as the Fed sponsoring a modest betting market and comparing the performance of its forecasts to the Fed’s internal forecasts.

            It seems like you’ve set a very high bar of testability. Many (most?) of Einstein’s predictions only became testable decades in the future with the development of better experimental apparatus. That is actually true of much modern physics, e.g., Super Hadron Collider.

            So you’re willing to say General Relativity and Higgs Boson weren’t good models at the time of publication because they required a substantial investment in experimentation? Much more than an NGDP prediction market I might add.

  2. You can’t know if 3 fails at all, absent some betting market for nominal GDP growth. But, it’s more of a truism than a model. It’s true…ish. A truishism.

  3. Here are three other models:

    (1) If the Fed stabilizes the growth rate of high-powered money, then it will stabilize the growth rate of RGDP.

    (2) If the Fed adjusts the Fed Funds rate as an intermediate target so as to stabilize the inflation rate, the nominal price of a consumer basket of goods, then it will stabilize the growth rate of RGDP.

    (3) If the Fed stabilizes the growth rate of NGDP, the nominal price of a GDP-deflator basket of goods, then it will stabilize the growth rate of RGDP.

    Model (3) fails in the interesting way of NGDP being stable, yet RGDP being unstable with offsetting instability of inflation.

    Now, one might have a separate question about whether the Fed can stabilize the nominal price of the GDP-deflator basket of goods or whether there is something special about that basket that is different from a consumer basket of goods or a basket of gold.

  4. Is there a known, pedictable relationship between GDP and Nominal GDP? Regardless of past correlations, what prevents a politically motivated Federal Reserve of doing things which affect NGDP differently from GDP?

    Could I run a successful restaurant where I posted a sign saying that part of the meal would be nominal food, at my discretion? What if I forced people to eat there?

    Basing theories on NGDP seems to be a result of the Streetlight problem. The theories are being adapted to what we can see under the streetlight, what is now measured. This doesn’t help to manage what is in the dark.

    The purpose of economics is to understand the dark, not merely to manipulate the few variables obviously under the light.

  5. What if they do hit the NDGP target but the RGDP doesn’t behave?

    Isn’t the one model also based on assumptions even more tenuous underlying another model?

    What are the assumptions of the underlying model?

    As a method to reduce Fed discretion and failure modes, I get it, but as panaceas go seems underwhelming. I really wish they hadn’t started paying interest on reserves, then at least we’d know something. I wonder if they don’t want us to know, so they dial up the causal density.

Comments are closed.