In a classic study of peer review, 75 psychologists were asked to referee a paper about the mental health of left-wing student activists. Some referees saw a version of the paper showing that the student activists’ mental health was above normal; others saw different data, showing it to be below normal. Sure enough, the more liberal referees were more likely to recommend publishing the paper favorable to the left-wing activists. When the conclusion went the other way, they quickly found problems with its methodology.
I prefer the term social disciplines for economics, political science studies, sociology, anthropology, psychology, and history.
I believe that it is true in general that one’s instinct is to focus on methodological flaws when results conflict with your priors and to ignore methodology and instead focus on results when a study finds congenial results. Probably this affects natural science as well, but my guess is that in natural science results are often more robust, so that if you do not like the results you cannot just carp about the methods.
In fact, it probably would be a good idea of journal editors would send out papers stripped of their results for peer review. That is, the referee should recommend for of against publication of a paper based on the methods used and the question asked, not on the answer obtained.
Problems with peer review aren’t limited to social sciences.
http://www.slate.com/articles/health_and_science/science/2015/04/fake_peer_review_scientific_journals_publish_fraudulent_plagiarized_or_nonsense.html
I believe it should all be re-absorbed back into the humanities, so we can be more open in admitting it’s the same muddling through and judgment calls as any scholarship.
The natural sciences are far from immune to this sort of thing. I followed the climate debate for many years where folks frequently ignored outrageously bad methodology and invented statistical methods if these yielded the “right” results.
“In fact, it probably would be a good idea of journal editors would send out papers stripped of their results for peer review.”
Yes, that’s a good idea although it’s probably not practical for many journals. Somebody would still need to review the results and conclusion. And whoever did so would have to read the intro and methods to understand the conclusion. This would increase review times. Also, I expect journals that regularly published failures would struggle to attract readers. I think these problems could be surmounted, but I’m not sure how.
I only look at the pictures. Pictures or it didn’t hapoen.
“No results? But how can I know they reached the right conclusion?!”
The problem here is that we don’t place any value on “disciplines”. Science is good because we think it’s the only epistemologically sound thing we have. The scientific method. And of course everybody finds the value in history, which is why we don’t call it a science, it’s just history.
But what’s the value of psychology or sociology or economics if they are not sciences? What are they then? What is a “discipline”? Just some smart guys saying things that favor their team? Why would anyone pay for that?
I like the article in general, but it is clear that race is a social construct- at least the particular races we now recognize…. Despite the fact that Wade identified “five distinguishable races”, anyone involved in classification problems knows that there are lots of ways to draw the circles around the points. The absurdity of race is that anyone thinks there is even any point in doing this, beyond using the race label to make stereotypical judgments about individuals. The oft-cited medical justification is ridiculous, as relevant genetic variations can now be directly observed.
I would agree, except it is the policy of the federal government that:
1) Everyone can be put into a fairly small number of racial boxes.
2) Everyone must be put in one of those boxes and statistics must be kept based on those boxes.
3) If there are any differences between the boxes, they will be assumed to be the result of discrimination, and steps must be taken to make the boxes equal.
So are you calling for the feds to prohibit collecting and publishing statistics that put people in “racial” boxes? Are you calling for “color-blind” treatment of people, no affirmative action, etc.?
The problem is not questioning what you disagree with but accepting without question that which you agree with. Worse though is trying to stop research of areas you disagree with.
Surprising results should be more questioned as they have more policy implications, while questioning what most would consider of little significance is a waste, so in that sense this study is reassuring,
Had the study referenced conservative activists with similar results would you be surprised? Would you have questioned the results or thought of course?