https://www.jneurosci.org/content/43/48/8074
Open Peer Review at J Neurosci!
Welp. There's another journal I won't review for. 😢
Peer review is not commentary. Peer review is an INTERNAL process for the authors to make their work better and editors to make the judgement of whether to gatekeep* the paper in their journal. It should only be of interest to historians of science. I do not write my peer reviews for public consumption.
* If you don't like gatekeeping, then you should do your science by systems that don't gatekeep, such as preprints.
Once a manuscript is sent out for review, there shouldn't be any further gatekeeping: if it's thought to be a bona fide attempt at rigorous science, worth of the time and attention of experts to produce an evaluation, then reviews are merely – greatly – the view of expert, practising scientists in the field, including specific feedback for authors to tighten up their work, arguments, conclusions.
Indeed, gatekeeping has no place in #ScientificPublishing. Let's envision a near future, a present really, where scientific discourse happens constructively and gatekeeping journals are there merely to contextualise and "storify" findings reviewed by others, to de-jargonise them, packaging them for outreach to serve a broader readership.
It depends on the goal of the product.
If your goal is to communicate discussion within a community of experts, then gatekeeping is unnecessary. (I can do my own "gatekeeping" of work by colleagues, in the sense that I can decide whether to believe their work or not based on my own experise.) This works great for a community of experts, and it is one of the reasons that arXiv and its descendants work so well for the scientific community of experts.
However, if your goal is to communicate to non-experts (journalists, medical doctors, politicians, teachers, engineers and even scientists outside of a field), then gatekeeping serves a strong purpose of ensuring that crap isn't disseminated.
I don't know if you were around when the creationists were trying to get stuff published in the gatekept spaces so that they could get it into schools, but one of the most important reasons that we were able to keep creationism out of most schools was that there were no "respectable" papers for them to cite in the courts, which used the gatekeeping process of science journals to differentiate between science and non-science. (Given the noted lack of scientific expertise in the judges making those decisions.)
For non-gatekeeping situations, why would the gatekeeping be done at the pre-review stage? It should definitely be done at the POST-review stage, as that it where it has been assessed for quality. Pre-review, everything should be available. There is no reason to gatekeep pre-review.
@adredish @albertcardona @jerlich the rise of genAI is about to make that second function -gatekeeping for non-experts- more and more important. I suspect it will also increase the need for gatekeeping for experts.
I wish it weren’t so but too much of the reform of peer review and publication was imo predicated on a reasonable assumption of scientists as good faith actors in ways that don’t look sustainable now given the unprecedented ability for machine generated nonsense.
@UlrikeHahn @albertcardona @jerlich
We have had lots of discussion on this mastodon feed about whether we should gatekeep or not, but @UlrikeHahn brings up the important question of #genAI.
But we haven't had much discussion of whether our gatekeeping system actually works in the new genAI world. There's lots of evidence that peer review doesn't do a good job with nefarious actors, and even that peer review itself has lots of limitations, even with good-faith actors. How is peer review going to cope with genAI?
So my question is:
Let's assume (for the sake of the discussion) that we DO want to gatekeep science, at least for non-experts... how can we do that? How can we gatekeep for non-experts in such a way that they are not fooled by genAI?
This leads to followup questions:
Can we gatekeep for non-experts without gatekeeping for experts? Do we need a way to gatekeep out genAI even for experts?
@adredish @UlrikeHahn @albertcardona
I would argue that the rise of GenAI demands more not less transparency. The "old" peer review system hides the process of approval, so once a paper is "peer-reviewed" it somehow has an air of legitimacy.
I share your concerns- but I think the solution involves networks of trust and more active science communication.
@jerlich @adredish @albertcardona I think the transparency issue is not the main one, though - it’s the level of gatekeeping required.