A recent survey of leading economists, called the IGM forum, asked two questions about CBO forecasts.
Question A: Forecasting the effects of complex legislative actions is hard, so even competent, non-ideological and non-partisan projections could differ substantially from outcomes.
Question B: Adjusting for legal restrictions on what the CBO can assume about future legislation and events, the CBO has historically issued credible forecasts of the effects of both Democratic and Republican legislative proposals.
The answers were overwhelmingly affirmative for both. I have been following the IGM forum for years, and you rarely see such a strong consensus.
What does the term “credible” mean?
Does the affirmative answer to question B mean that the forecasts are accurate enough that policy makers should take them seriously? John Whitehead seems to think so. Pointer from Mark Thoma.
Or does the affirmative answer to question A mean that the forecasts are not accurate enough to reliably guide policy? Russ Roberts and I would tend to think so.
Anyone, including Russ or me, who criticizes economic methods faces the following argument.
1. Policy has to be based on some model and some forecast.
2. A formal model or a statistical forecast is more rigorous than intuition/opinion.
3. Therefore, the best approach is to use formal models and statistical forecasts.
I think that the problem comes in the way that one interprets point (1). Consider two possibilities:
1a. Policy has to be based on a “model” and a “forecast” which rule out any empirical analysis of how policy is formulated and implemented. Also, the “model” and the “forecast” can ignore the possible evolutionary responses of decentralized activity, including possible emergent market solutions to the problem that the policy is intended to solve, as well as innovations responding to the policy that mitigate its effects or that produce unintended adverse consequences.
1b. Policy has to be based on a “model” and a “forecast” which do take into account empirical public policy and dynamic market responses to the original problem and to the proposed solution.
If we interpret point 1 as “1b,” then I accept the logic that a model is better than no model and a forecast is better than no forecast.
If we interpret point 1 as “1a”, then the argument is a swindle. Models that ignore empirical public policy and dynamic market responses are not necessarily better than intuition/opinion, and they should not be regarded as credible.
Taking into account these requirements for credibility, CBO forecasts are not credible. Using them may very well do more harm than good.
The swindle is suggesting 1a is relevant.
I do some forecasting customer cost in business and there is a lot of work and there is always something missing or wrong. Take for instance the CBO Obamacare estimates:
1) In terms of uninsurance Americans, they expected 28M and by 2016 the nation has 27M. So they nailed that one beyond belief.
2) I think most CBO estimates for ACA plans are still 5% higher than the 2016 actuals.
3) But it did miss the ACA signups by ~15M…Because they did not assume the size of the Medicaid expansion and they assumed the employer based healthcare in 2009 would continue to crumble. A reasonable assumption in 2009 at the height of the Great Recession and employee health benefits were cut WAY back with the increase of deductible plans. This benefit decrease masked a lot of the wage cuts of the Great Recession. However, the labor market turned a lot quicker than expected and by 2012 the skilled labor market started getting tight. (Notice since the unemployed claims since 2011 have been relatively low compared to other recesssions.)
“But it did miss the ACA signups by ~15M…Because they did not assume the size of the Medicaid expansion and they assumed the employer based healthcare in 2009 would continue to crumble.”
So if their reasoning was this far off, why should we give them credit for “nail(ing)” the overall estimate in (1)?
If I model sales saying product A will decrease and product B will increase, when the opposite happens, how can i then take credit for lucking into an “accurate” forecast of gross sales?
1. We must use some model
2. This is a model
3. Therefore we must use this model
I’ve heard this somewhere before…