News for hackers. Stuff that matters.
Re:The Absurdity of Peer Review: What the Pandemic Revealed by xg15 on Sunday June 06, 2021 @06:00PM (#27416754)
Ok, so what's the alternative to peer review then? We'll "just know" when that paper/blog entry/tweet/4chan post is written by the next Einstein and we can take it for a fact?
Re:The Absurdity of Peer Review: What the Pandemic Revealed by SiVal on Sunday June 06, 2021 @07:17PM (#27417265)
"Fact" in science needs to be based as much as possible on reproducibility by anyone and as little as possible on the support of those with institutional authority.

Alternatives to peer review are seen everywhere these days: open review and discussion and even the open source software model.

A scientist does some work, writes it up in rough form, puts data files online, and tells some friends/colleagues about it. After some feedback and corrections, the scientist puts the writeup online and notifies discussion forums. Then, anyone interested can look it over and provide feedback of all sorts: good, bad, and ugly.

The scientist then incrementally upgrades the paper, including edits and rewording and donated graphics, etc., along the lines of open source software and pull requests. I'm not claiming that many eyes make all bugs shallow, but if someone points out that your algorithm omits the first line of each data table, or your evidence of warming only measured temperature during the day while comparing to temperatures measured day and night, it's true or not regardless of the credentials of the person making the claim.

The paper doesn't have to be fossilized but can be versioned, because "paper" doesn't mean paper anymore, and your institutional worth (as opposed to scientific worth) can be based on citations of all versions rather than the prestige of the journal.

You might fear that authors will be overwhelmed by mostly low-quality feedback, but most papers are read by so few people that this isn't much of a risk, in my opinion. The biggest risk is deliberate lies, but we already have a reproducibility crisis, and only reproducibility--not the filters of authority--can ultimately solve it.

Re:The Absurdity of Peer Review: What the Pandemic Revealed by toomim on Sunday June 06, 2021 @07:18PM (#27417269)
A group of us are working on a solution. Here's a reference to a prior comment thread: https://news.ycombinator.com/item?id=27297013

The solution is to make peer-review better, rather than remove it entirely. Imagine that you can choose a set of people who you trust on any particular topic, and rank them with how much you trust them. Then you, and those people, can all rank any paper (or in general any URL) you read, and each of you will have a feed of incoming papers (or URLs) on any topic computed by a function that you control, in your personal reader app, which crawls your network of trusted people, and the people they trust, and so on, and aggregates all rankings and reviews. After about 7 degrees of hops across that network, you'll cover most of the world.

The key new idea here is that peer review will no longer try to be a single, objective, determination of truth. Each person can have their own slightly different feed, computed by slightly different criteria, using slightly different sets of trust metrics of people over the community, all competing for other people's trust, by writing good reviews, and good papers, in order to gain reputation and influence within the distributed network.

This won't be a single app. Rather, it'll be a distributed use of the WWW. Anyone can host their ratings at any web URL, on any website, and subscribe to the ratings from anyone else, at any other URL, at any other website. We can make this all realtime using the web extensions defined by https://braid.org, so that you can have a continuously-updating feed.

In effect, we'll be building a distributed, subjective hacker news (or equivalently, a distributed subjective reddit). We don't need to fight over objectivity anymore. Instead, we'll all be entitled to our own view of the web, and be able to share our views with others. And we'll find that we are best off ourselves when we import the best views from the best people we know, and that way we will see the world through each other's eyes, and even be able to — with the flip of a dial — see the world through our enemies' eyes.

Please reach out if you are interested in building anything similar! I am [email protected], and am coordinating a group to make this happen.

Re:The Absurdity of Peer Review: What the Pandemic Revealed by leephillips on Sunday June 06, 2021 @08:05PM (#27417549)
As the article points out, the previous Einstein’s papers were not peer reviewed.

Peer review is not what the pubic and journalists think it is.

An example:

The Physical Review journals publish experimental high energy physics papers from large collaborations without review, because there is no point to it. Anyone competent to review these papers is already on the list of 400 authors.

Science is strongly specialized. Results are largely disseminated by preprint. You know if a preprint makes sense if you’re working in the field. You probably know better than whomever the editor can get to agree to review the thing.

By the time something gets published, a year has gone by and research is already building on work reported in the preprint.

Re:The Absurdity of Peer Review: What the Pandemic Revealed by sharken on Sunday June 06, 2021 @06:19PM (#27416887)
There really isn't any good alternative, having one or more qualified peers go over the results is a lot more trustworthy.

Of course it requires that there isn't any hidden agenda or that a specific result of the review is dictated.