🇳🇱
I have been thinking about writing a webapp that would enable QOTO to become the first "open, distributed, scientific journal".. I am still thinking about how I am going to go about it but the idea would be a scientific journal that anyone, regardless of credentials or pre-approval, can submit articles and peer-review and ultimately the acceptance or rejection of an article will happen without a centralized authority.
In all liklihood acceptance will be "fuzzy" in that it will be up to the reader to decide if the groups which approved of the article through peer review have sufficient credentials that you trust them. So the key will be in designing a way to make that work.
Ideas are welcome of course.
@freemo I think you need one more layer of indirection. Generally speaking, peer reviewers are supposed to be anonymous, and in any case it sounds unworkable to expect each reader to verify each reviewer's identity and credentials. But if your journal has editors who each verify and evaluate credentials of their own reviewer pool, the reader's decision becomes to trust the judgement of each editor, of whom there are fewer and whose identities are publicly known. Having an editor also means there's someone who can issue retractions, which could be quite important for the journal's credibility if there are bad actors who get something past the reviewers.
@freemo I think you're right in that it should be a trust network, but not in that it should be an automated system applying rules to determine who's trusted.
I see it working something like your browser's certificate store - you add "editor certificates" to your profile on the qoto-journal webapp in the same way you add "root certificates" to your browser. Each editor forwards submissions to his pool of reviewers and signs the articles they recommend for publication. If an article is accompanied by the signature of an editor you trust, the article shows up in your view of the journal. If an editor includes malicious or incompetent reviewers in his pool, and consequently becomes known for publishing bad papers, people will stop trusting his certificate.
I think an automated system would be prone to people gaming the rules, and the reader wouldn't have the fallback of just revoking an editor's certificate in case things got out of hand. For instance, if I were to try and exploit the rules in your example:
- I might review papers totally outside my competence, because although my experience in fluid dynamics is totally irrelevant to, say, political science, the rules award my review of one equal credit to the other
- I might find another author and set up a tit-for-tat scheme to give each other five free points every iteration, no matter the quality of our papers
What worries me is that if the system initially develops a reputation for being easy to game and accepting of low-quality content, it will be very hard to shed that reputation later on, even if improvements are made. So it needs to be done right the first time.
@khird That would create a centralized system, though, where whoever controls the root certificates controls effectively who has the power to curate the articles.
The only real difference between your proposal and mine is that in mine the “root certificates” are dictated by the reader, they opt-in or out of root certificates and this ultimately determines the credibility rating of articles.
There can even be meta-root authorities that effectively suggest the list of root authorities people should (but are not required) to adopt.
I might review papers totally outside my competence, because although my experience in fluid dynamics is totally irrelevant to, say, political science, the rules award my review of one equal credit to the other
In my system, and what i tried to explain, it is only when you get a paper past peer review you get points. You wouldnt get points simply for submitting a review.
I might find another author and set up a tit-for-tat scheme to give each other five free points every iteration, no matter the quality of our papers
In a system that adds or takes away points as one gains credentials this wouldn’t necessarily work. As other more credible authors review the same work and reject it you will find the tit-for-tat scheme is more likely to lower your credibility than to raise it.
What worries me is that if the system initially develops a reputation for being easy to game and accepting of low-quality content, it will be very hard to shed that reputation later on, even if improvements are made. So it needs to be done right the first time.
On this I generally agree. But since ultimately the trust comes from the user and there is no absolute “score” or acceptance outside of the user defining who they trust or who they dont then this should be easy enough to do.
For example the user might choose to just use the QOTO’s recommended trust (which we hand pick ourselves) and leave the setting at that, in which case the system would be operating with no more risks than a traditional journal. The difference is the user could pick a different trust priovider all toegher, or combine trust providers in a weighted way, or even define their own subset of trust based off trust providers (such as only accepting authors who have posted in a particular set of journals or who have been cited atleast 100 times)
@freemo @khird This all sounds like a contextual “web of trust” scheme, there’s quite a bit of prior art in web of trust, not sure about contextualizing it like this, but it seems like a minor enhancement (one separate web of trust per context, rules for cross-context trust would be the only complicated part).
@khird The reader wouldnt be verifying the reviewers individually, I agree that would be unworkable.
The solution would have to lie in a network of trust. So, for example, a user might specify they trust people with PhD's from acredited institutions with 20 points of default trust, and then one additional point for every successful publication they have in a third party journal, and an additional 5 points for each article they have published on QOTO that has passed my personal criteria for acceptance.
You set up the categories of "trust" and then those criteria are used to determine what passes for the individual.
Peer-review being anonymous can be a possiblity and optional. The identiy need not be public as long as the system itself knows the person and can evaluate their credentials.
This is what I meant when I said the approval process needs to be "fuzzy".. it has to use some trust network where a person can generalize groups or categories of "properties" as being things that increase or decrease one's trust in the person.