๐ณ๐ฑ
I have been thinking about writing a webapp that would enable QOTO to become the first "open, distributed, scientific journal".. I am still thinking about how I am going to go about it but the idea would be a scientific journal that anyone, regardless of credentials or pre-approval, can submit articles and peer-review and ultimately the acceptance or rejection of an article will happen without a centralized authority.
In all liklihood acceptance will be "fuzzy" in that it will be up to the reader to decide if the groups which approved of the article through peer review have sufficient credentials that you trust them. So the key will be in designing a way to make that work.
Ideas are welcome of course.
@khird The reader wouldnt be verifying the reviewers individually, I agree that would be unworkable.
The solution would have to lie in a network of trust. So, for example, a user might specify they trust people with PhD's from acredited institutions with 20 points of default trust, and then one additional point for every successful publication they have in a third party journal, and an additional 5 points for each article they have published on QOTO that has passed my personal criteria for acceptance.
You set up the categories of "trust" and then those criteria are used to determine what passes for the individual.
Peer-review being anonymous can be a possiblity and optional. The identiy need not be public as long as the system itself knows the person and can evaluate their credentials.
This is what I meant when I said the approval process needs to be "fuzzy".. it has to use some trust network where a person can generalize groups or categories of "properties" as being things that increase or decrease one's trust in the person.
@freemo I think you're right in that it should be a trust network, but not in that it should be an automated system applying rules to determine who's trusted.
I see it working something like your browser's certificate store - you add "editor certificates" to your profile on the qoto-journal webapp in the same way you add "root certificates" to your browser. Each editor forwards submissions to his pool of reviewers and signs the articles they recommend for publication. If an article is accompanied by the signature of an editor you trust, the article shows up in your view of the journal. If an editor includes malicious or incompetent reviewers in his pool, and consequently becomes known for publishing bad papers, people will stop trusting his certificate.
I think an automated system would be prone to people gaming the rules, and the reader wouldn't have the fallback of just revoking an editor's certificate in case things got out of hand. For instance, if I were to try and exploit the rules in your example:
- I might review papers totally outside my competence, because although my experience in fluid dynamics is totally irrelevant to, say, political science, the rules award my review of one equal credit to the other
- I might find another author and set up a tit-for-tat scheme to give each other five free points every iteration, no matter the quality of our papers
What worries me is that if the system initially develops a reputation for being easy to game and accepting of low-quality content, it will be very hard to shed that reputation later on, even if improvements are made. So it needs to be done right the first time.
@khird That would create a centralized system, though, where whoever controls the root certificates controls effectively who has the power to curate the articles.
The only real difference between your proposal and mine is that in mine the “root certificates” are dictated by the reader, they opt-in or out of root certificates and this ultimately determines the credibility rating of articles.
There can even be meta-root authorities that effectively suggest the list of root authorities people should (but are not required) to adopt.
I might review papers totally outside my competence, because although my experience in fluid dynamics is totally irrelevant to, say, political science, the rules award my review of one equal credit to the other
In my system, and what i tried to explain, it is only when you get a paper past peer review you get points. You wouldnt get points simply for submitting a review.
I might find another author and set up a tit-for-tat scheme to give each other five free points every iteration, no matter the quality of our papers
In a system that adds or takes away points as one gains credentials this wouldn’t necessarily work. As other more credible authors review the same work and reject it you will find the tit-for-tat scheme is more likely to lower your credibility than to raise it.
What worries me is that if the system initially develops a reputation for being easy to game and accepting of low-quality content, it will be very hard to shed that reputation later on, even if improvements are made. So it needs to be done right the first time.
On this I generally agree. But since ultimately the trust comes from the user and there is no absolute “score” or acceptance outside of the user defining who they trust or who they dont then this should be easy enough to do.
For example the user might choose to just use the QOTO’s recommended trust (which we hand pick ourselves) and leave the setting at that, in which case the system would be operating with no more risks than a traditional journal. The difference is the user could pick a different trust priovider all toegher, or combine trust providers in a weighted way, or even define their own subset of trust based off trust providers (such as only accepting authors who have posted in a particular set of journals or who have been cited atleast 100 times)
@freemo @khird This all sounds like a contextual โweb of trustโ scheme, thereโs quite a bit of prior art in web of trust, not sure about contextualizing it like this, but it seems like a minor enhancement (one separate web of trust per context, rules for cross-context trust would be the only complicated part).
@freemo Have you seen https://researchers.one/ ?
@freemo Scientific publishing is already decentralized. Publishing decisions are made by journals and for most studies there are many journals to chose from.
Especially if you count journals without reputation, which QOTO would (initially) be, there is more than enough choice. Anyone can start a journal. https://pkp.sfu.ca/ojs/
Coding a new journal system is a fair bit of work.
@GrassrootsReview Many centralized journals does not equate to decentralization. The whole point of a decentralized framework is to remove the authority and thus any need for consideration of a journals integrity (as there are no journals).
@freemo For me decentralized journals do mean that the system is decentralized.
Reputation is important. Otherwise it is just blogging, which is decentralized, but not particularly suitable for scientific progress.
But do write up how your system would work and which problem it would solve.
@GrassrootsReview Reputation is important, but not of the journal, of the people publishing and those doing the peer review.
Decentralized journals dont exist, only a collection of centralized journals, not the same thing. Each journal is centralized and largely independent from each other.
In a proper decentralized systems the article writers and peer reviewers would have a reputation that isnt tied with or verified by any journal acting as an authority but rather in a decentralized way.
Yes I agree a proper writeup on the mechanics would be key to really have a meaningful discussion of my plans.
@freemo In a good system the reputation of the authors is not important. My colleagues will be willing to read my blog posts, but new people and outsiders need the support of the reputation of the journal.
The journal and the editors are named and can build up a reputation.
Reviewers are normally not named. Requesting them to do so makes it a lot harder to find reviewers because that means writing reviews affects your reputation and which would require putting in 10 times as much work.
> In a good system the reputation of the authors is not important. My colleagues will be willing to read my blog posts, but new people and outsiders need the support of the reputation of the journal.
That doesnt add up to me. Typically the journal decides who can or can't get the final say on peer-review. So yes, the people submitting and authoring the peer review absolutely matters. The journal just acts asa a proxy to decide who can or can not make the call on peer-review.
> Reviewers are normally not named. Requesting them to do so makes it a lot harder to find reviewers because that means writing reviews affects your reputation and which would require putting in 10 times as much work.
Right, my system would allow reviewers to remain anonymous while still allowing them to gain a reputation. You might know two reviewers are the same person, but you wont know who that person is.
The problem here is that while I agree reviewers should have some level of anonymity, at the same time the general public should be able to review the peer-review process itself and judge it on an article by article basis, as well as judge the reputation of the reviewer (without knowing their identity, so based solely on their past quality of work reviewing).
@freemo Peer review helping outsiders is explained here in more detail. It adds up. http://variable-variability.blogspot.com/2014/01/peer-review-helps-fringe-ideas-gain.html
Normally an editor, not the journal decides on which manuscripts are published and who becomes reviewer. The editor-in-chief ("journal"?) typically decides which editor does which manuscript. The peer reviewers are advisors of the editor, but do not decide.
Publishing the peer review reports is an idea I like and only the arguments should count. The editor as the decider is known.
@GrassrootsReview Right, the point here is there is a centralized group (in this case the editors and peer reviewers collectively) who decide reputations and what passes and what doesnt, and generally obscures the peer-review process publicly. Thus not just centralizing the process to a select few at the journal, but ultimately obscuring it from public scrutiny as well.
The point of decentralization is openness, to allow the consumer of the article to have the power to judge the reputation by opening up the processes and giving them the information needed to do so.
@freemo Just start you own journal if you prefer editors and reviewers who have less expertise. The consumer can decide to read it.
Climate "skeptics" have been there before and have started something that looks like a scientific journal to fool the public and non-science journalists. http://variable-variability.blogspot.com/2014/02/global-warming-solved-in-open-peer.html
Interestingly I just got a follow by some Social Darwinian COVID deception account. The same MO as the climate "skeptics", just another way to kill people.
@GrassrootsReview I never said anything about prefering a journal where edirors or reviewers have less expertise. I am talking about a system that is open and transparent, hopefully this should result in **more** expertise, not less.
The last thing an open and transparent process would do would be to create climate "skeptics" articles that are considered credible, since it the system would require a scientist to prove their expertise openly. In fact quite the opposite, such an open and transparent, but high reputation/expertise, decentralized system ideally should gain trust not just from traditional scientists but also people like climate change deniers, and this hopefully should build trust with the scientific community once again.
@freemo You wrote: "Right, the point here is there is a centralized group (in this case the editors and peer reviewers collectively) who decide reputations and what passes and what doesnt, and generally obscures the peer-review process publicly."
That problem is solved by starting you own journal in the decentralized system we have.
If quality of science mattered we would not have climate "skeptics", anti-vaxxers or COVID deniers. Quality is not a good predictor of their memes.
> That problem is solved by starting you own journal in the decentralized system we have.
It is not, starting my own journal would put me in charge, and the journal would not be "decentralized", like all journals it would be centralized (in this case me being the central point of authority for my journal).
The problem with centralized systems, such as journals, is that I am in control and can obscure whatever parts of the process I wish. If I happen to take a bribe (as rarely but occasionally happens) there are no good checks and balances to flush that out. I might get caught, I might not. Generally its not a huge issue with journals, but it has happened.
The distrust in science, which seems to come from across the political spectrum, doesnt come from no where. These centralized and closed processes create suspicion and ruin the communities trust in journals. The climate deniers are wrong in their denial of climate, but they are right regarding the lack of transparency and the power distribution that ultimately fuels their suspicion.
It isnt about quality of science so much as the transparency and the ability for individuals to have the means to judge that quality, which only comes when the system is open, both in terms of open access, and open processes.
@freemo Everyone can start a journal, not just you. So it is perfectly decentralized.
If you start your own journal or in some decentralized system contribute to the quality of the scientific literature less frequently, it is (initially) hard to judge whether you do it well. Bribes would thus be more effective.
Trust in science is high & science is becoming more open. The world is increasingly unequal and authoritarian and authoritarians do not like their power being constrained by reality.
@GrassrootsReview decentralization is not “everyone can start a centralized and walled off community”.. that isnt in any sense what decentralization is. For something to be decentralized there has to be some sense of interoperability and organization.
I really dont know why you insist that having more than one centralized entity is somehow equivalent to decentralization.
or in some decentralized system contribute to the quality of the scientific literature less frequently, it is (initially) hard to judge whether you do it well. Bribes would thus be more effective.
Bribes are more likely, of course, if you dont have a reputation yet.. thats the whole point of a decentralized system specifically building reputations through webs of trust. It isnt until I’ve proven that i am not susceptible to bribes than my opinion and publications hold a high trust value in the first case.
@GrassrootsReview I just realized your accounts description is almost exactly what im advocating for with an open decentralized peer review process. After reading it and how it is almost word for word what im arguing for here I am rather surprised you are arguing against it.
To quote you:
Let’s bring the quality control of scientific articles back to the scientific community with open post-publication peer review independent of scientific journals
@freemo If that was what you were proposing, then I like it. ๐
But it is not just another journal (it does not publish, it only reviews). It is a journal-independent review. Starting a new journal (system) will not break the power of the publishers.
It has editors, who moderate the debate. Named journals and named editors who build up a reputation and make it possible to have anonymous comments without becoming YouTube comments.
It is better for science by being post-publication peer review.
Thats a big part of it.. people should be able to know
1) what peer-reviewers have either liked, or criticized in the paper.
2) what peer reviews were acted on and what changes were made
3) What the credentials, accomplishments, and past publications or peer-review the person has done
navigating this as an aggregated web of trust so users dont need to be knee deep in it would also be nice.
Also, never deleting a paper but at worst labeling it as debunked and showing the above reasons why.
The key is opening up the process both to reduce the friction for people who want to contribute, and to make the process transparent so any end user can judge the credibility from all the facts.
I don't know hat you mean here but I don't like the sound of it. Trust is the first precondition to betrayal.
It means you as the user can choose to trust someone directly (as a reviewer or publisher or whatever).. or you can also choose groups of people to "meta-trust"... for example as a user I might decide I want to add 5 points of trust to anyone who has a PhD from MIT, or anyone that that Bill Nye thinks is cool.
Basically there can be curated lists of people and you can choose if you wish to trust those lists or not, even as they change. If you dont like it, then you dont have to trust lists at all and can just evaluate each person yourself.
@freemo I think you need one more layer of indirection. Generally speaking, peer reviewers are supposed to be anonymous, and in any case it sounds unworkable to expect each reader to verify each reviewer's identity and credentials. But if your journal has editors who each verify and evaluate credentials of their own reviewer pool, the reader's decision becomes to trust the judgement of each editor, of whom there are fewer and whose identities are publicly known. Having an editor also means there's someone who can issue retractions, which could be quite important for the journal's credibility if there are bad actors who get something past the reviewers.