I haven't followed the eLife saga closely, and I do admire their openness to experiment (and roll back experiments if they don't work out), but I am confused about everyone saying that the latest experiment gets rid of gatekeeping.
If the rule is that any paper editor sends out to review gets published, along with reviews, surely the more accurate description is that the new model moves all gatekeeping before the review stage (whereas only some of it is in status quo)?
What am I missing.
@erolakcay You are correct. The idea is that if there is going to be gatekeeping, it is better for it to be clearly in the hands of a named individual who can be held accountable.
@ryneches That's not how it's sold, though, is it? The title of this editorial announcing the experiment is literally: "Scientific Publishing: Peer review without gatekeeping" https://elifesciences.org/articles/83889
This has been very inexplicable to me.
@erolakcay @ryneches maybe a succinct way of reconciling the two points is that the elife model can be called peer review without gatekeeping, but not publishing without gatekeeping.
I also really like @ryneches 's point that an editor could choose to send a paper for review precisely to generate highly critical reviews, so the signal from the editor's decision is not as clear-cut as "gatekeeping" naively suggests.
@erolakcay @askennard I don't think the subset is small, though! I read a lot of pre-prints where the main point is solid, but the manuscript has some glaring weakness. You are right that this could be very unkind if people carry on using the same tone and habits they use when writing closed reviews, but... I think habits will change, and we'll all be better for it. When people know that their review will be reviewed, I think there is more of an incentive to be kind and constructive.
@ryneches @askennard I agree on your second point: I think making reviews (but not reviewer identities unless they sign) can be great, and posting reviews on biorxiv is a good step. (As is, btw, the cross-review process; I've enjoyed it in a few journals that do it, incl. eLife.) But that's logically independent of the issue of publication regardless of reviews (although indirectly connected).
@ryneches @askennard I do think you might underestimate (as I once did) how much "closed" reviews get reviewed. Apart from anything else, good reviewers are a valuable resource and editors are keen to know and keep track of who writes thorough and useful reviews.
@erolakcay @askennard The question is, "Valuable to whom?" The fact that they are closed means their value does not spread far and does not last long when compared to the paper (assuming it makes it into press). That value is captured by the narrow interests of the journal, not the community.
@ryneches @askennard I agree there is some loss of value when reviews are kept confidential. I don't agree that "journal interests" are necessarily narrow (some are, but others, mostly society journals, do represent their communities).
Anyway, I don't claim to have the last word at all. Experiments are welcome, and kudos to eLife for doing them.
@askennard @ryneches there is a small subset of papers where this works (i.e., the work is obviously important, data unique etc.) but most of the time this is simply unkind to the authors. And with only indirect incentives to address highly critical reviews, I don't see how this leads to more improvement in those papers relative to a process where the editor makes a judgment call later.