Maria Konnikova surveys some of the literature.
Lord and his colleagues asked people to read a series of studies that seemed to either support or reject the idea that capital punishment deters crime. The participants, it turned out, rated studies confirming their original beliefs as more methodologically rigorous—and those that went against them as shoddy.
This and other studies serve to highlight confirmation bias, which helps to reinforce tribalism in politics.
I would like to point out that this form of confirmation bias is a very important problem in academia. That is why I think that studies should be evaluated methodologically before the results are known. A referee should be asked whether the study is capable of producing results that influence someone to change their minds. Could the results turn out to be against your prior beliefs? If so, would that influence your prior beliefs?
Confirmation bias perhaps has such power because of a key principle often overlooked. Aristotle and Euclid, who introduced the mathematical/logical system we still use, understood the the first premises cannot be deduced. A deductive system, i.e. Geometry, must begin with ‘assumptions’. No other way.
Because of the amazing success of this method, it is forgotten that the initial ideas were not proven. All logic has this weakness. This is mathematics. Economics and politics even more so. Mises understood this and spends many pages on thinking in his writing.
I can see several issues here:
1) In terms of confirmation bias, I assume that already. I don’t read Arnold Kling to get the Housing Bubble was mostly a Private Sector failure Point Of View. (My opinion) I assume your bias is it was the all government fault to lead the private sector to failure.
2) I think of studies feel the need to show a result. I analyze a lot of customer data in my job and often find the results fit standard thoughts on the subject. So when I consulting on a customer’s accident data and they failed to disprove the null hypothesis. I stated that is a very valid result and suggested that current policy is already fine. (Considering they need union agreement to change policy so it made the project quicker to finish!)
3. A lot of times we see slight changes in measurement can make a difference. In terms of police shooting African-Americans, we see all four:
3a. When stopped police do not treat African-Americans different than white suspects.
3b. African-Americans are stopped over 3 times as much.
3c. African-Americans commit more crimes.
3d. Crime is higher for Poor citizens where more African-Americans are.
ALL are true and it is interesting to analyze the mix to get a result. Also these variables can change, so in the state of California we are seeing all the increase in crime from white citizens with decreasing minority crime! I am not sure what that means but it will interesting to watch.
1) so you will only review evidence from people who present your hypothesis?
I literally don’t remember what Arnold’s full spectrum take on it is, but I know it isn’t going to coincide 100% with my own opinions, but I don’t think he is biased and it is certainly more nuanced than your characterization of it.
Btw, I can really think of only one person I know I could never read with an open mind.
Maybe econ and psychology are different, but in my field nobody is qualified to critique anyone’s methods. There is maybe one other guy doing what I do. And maybe in psychology and econ they shouldn’t either since it seems like it is largely a way to delegitimize results and conclusions. What did you do and what did you get? The discussion and conclusions are your reward for doing the rigamarole. Just ignore them if you don’t like them. The underlying problem is politicization of topics.
“A referee should be asked whether the study is capable of producing results that influence someone to change their minds.”
That would sure cut down on the journal proliferation problem!
But seriously, we used to have a nice little thing called The Journal of Medical Hypotheses, but they nipped that in the bud. What if there was a journal of proposed study methodologies?
Well, evaluated methodologically, if you start out asking people to read a series of squishy studies (measuring the deterrence of a single factor is a questionable pursuit) and then ask people for their opinion on those studies, there really isn’t much there except bias to talk about.
Why do we have so many studies to begin with? Is the demand really driven by the questions, or are they driven by academic requirements and lobbying?
In really deep disagreements, the question of which methods and approaches are valid is not settled. As a result, simply evaluating the kind of evidence presented in absence of the results is not likely to alter the outcome. That is, I suppose, the experiments are demonstrations; the outcome being known in advance of the execution.
I’ve had a similar idea for a while now, if you want to close in on truth you should be asking people who disagree with you what would it take for you to change your mind. Of course this only works on issues of disagreement in worldview, not disagreement in values.