One more thought about (pre-publication) peer review: it increasingly appears to me that peer review is a solution in search of a problem. Do we want to:
- improve papers scientifically?
- improve readibility of papers?
- detect fraud ?
- find and highlight good papers?
- give every scientist access to expert feedback?
- etc.
I am quite sure that for all of these tasks, 21st century technology allows much better solutions than pre-publication peer review.
@kjbinstl One could obviously be more radical. Or focus on other parts (e.g. fraud detection would plausibly work better if we funded dedicated detection teams). More generally I find it extremely implausible that a we could not improve substantially over a system designed decades ago to ease editorial workload in big journals. The saddest part to me is that we are not even trying, not evaluating and experimenting as good scientists should.
@modrak_m @kjbinstl I agree that this would be a great improvement over the current profit-driven publishing system. Two thoughts about peer review specifically:
1. As noted in my other toot, I suspect that reliance on preprint comments replaces more egalitarian access to peer feedback with a brutal attention economy. Big names benefit; the community becomes less permeable. How can this be addressed?
@modrak_m @kjbinstl 2. If only few receive spontaneous preprint comments and journals solicit extra ones, the process seems to change only for those few. Most authors still rely on pre-publication journal peer review. That said, if all journal-side reviews were published on the preprint, the system would become much more efficient for papers that subsequently approach multiple journals.
But perhaps I misunderstand your initial criticism that peer review is a solution in search of a problem?
@FrederikAust The aim of the proposal was to show, that we could achieve something very close to current state with much more efficiency and transparency (one of the weird quirks of the current system is that most review effort is spent on bad-but-not-obviously-garbage papers, especially when the author acts in bad faith and doesn't really try improve the paper between submissions to different journals). I do think we should go further than that 1/2
@FrederikAust I agree that sliding into brutal attention economy would be quite bad, but I don't think that's inevitable. Also, quite often people don't get good peer feedback in the current system anyway. If we focus on peer feedback then an interesting example is Stack Exchange. Some (not all) SE sites allow very good, reasonably egalitarian access to high-quality peer feedback/help. We could (and should) have other goals then feedback and those IMHO deserve their own solution. 2/2
@FrederikAust So IMHO pre-pub peer review is a "solution in search of a problem" in the sense that whenever you pick up a specific goal it supposedly has, PR generally doesn't really achieve it and it is easy to imagine systems that achieve the goal substantially better at lower cost. PR makes no sense to me unless you heavily value both a) having just a single system to do many things and b) not changing anything. I don't think either of those values is really defensible. 3/2 😄
@kjbinstl A relatively tame proposal, mostly following current norms but adding efficiency (details left out for brevity):
- Every paper starts as a preprint and is versioned
- Anybody can review/comment on preprints both under real name (with some verification/trust system) or ano/pseudonymously (similarly to PubPeer)
- Journals work as overlays providing "stamps of approval" (like https://www.bioverlay.org/) - they may use the reviews already present for a preprint or solicit some extra.