The current peer review process serves as both gatekeeper and evaluator. Post-publication review would separate these functions by letting the author decide when to publish. “Making publication trivial would foster a stronger recognition that study results are tentative and counter the prevalent and often wrong view that whatever is published is true,” Nosek explains. Another big benefit, as Nosek and his colleagues argued in 2012, is that “the priorities in the peer review process would shift from assessing whether the manuscript should be published to whether the ideas should be taken seriously and how they can be improved.” This change would also remove a major barrier to publishing replications, since novelty-seeking journal editors would no longer serve as naysaying gatekeepers. Ultimately, Nosek would like the OSF to evolve into something like a gigantic open-source version of arXive for all scientific research.
Think of letting blogs can do the job of editors and peer reviewers.
I think this is very important stuff, but I don’t like the “broken science” meme. It makes it sound as though everything that’s being published is nonsense. In fact, today’s problems were, for the most part, identified in 1978 by Edward Leamer (his still-underappreciated book Specification Searches)–having to do with p-hacking, false positives, selective reporting of results, etc.
The other big issue is that the current system is asking reviewers and editors to decide *today* whether a paper is a big contribution or not, when it may take years to really know for sure. So we are too skeptical of papers before they are published, and not skeptical enough of papers after they are published.
The American Finance Association recently had a panel discussion on the referee system, whether it is broken. There is a video of (most of) the discussion, here:
https://www.youtube.com/watch?v=vyFhcB1W6Kg
_Medical Hypotheses_, until Bruce Charlton was pushed out, was a rich lode of entertaining notions that wouldn’t pass the average gatekeeping function.
Bruce Charlton continues this tradition elsewhere, at his blog.
There are some bloggers who just have about four hobbyhorse ideas that they parade, rotating between them. Steve Sailer and Bruce Charlton don’t fit that classification. The same could probably be said of Nassim Nicholas Taleb and Paul Graham.