News for hackers. Stuff that matters.
Re:Why isn't preprint review being adopted? by bumby on Sunday March 24, 2024 @04:05PM (#39810007)
I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarity in the writing. But it's also extremely frustrating when it's obvious the reviewer hasn't even bothered to carefully read the manuscript.

Under a payment paradigm, we need mechanisms to limit the incentive to maximize throughput as a means of getting the most pay and instead maximize the review quality. I assume there'd be good ways to do that, but I don't know what those would be.

Re:Why isn't preprint review being adopted? by heikkilevanto on Sunday March 24, 2024 @06:13PM (#39810994)
So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...
Re:Why isn't preprint review being adopted? by bumby on Sunday March 24, 2024 @07:30PM (#39811482)
Or possibly just a way to review the reviewers. This opens itself up to competitor bias, though, so it would need to be thought out in a way to minimize that.
Re:Why isn't preprint review being adopted? by currymj on Sunday March 24, 2024 @06:40PM (#39811177)
many CS conferences have something literally called a "meta-review" and then there are further senior people who read and oversee the meta-reviews. it stops there though.
Re:Why isn't preprint review being adopted? by abhgh on Monday March 25, 2024 @12:53AM (#39812958)
Unfortunately the state of meta-review is similar to that of reviews. Rarely delves deeper, mostly acts as a summarizer for the independent reviews.
Re:Why isn't preprint review being adopted? by MichaelZuo on Monday March 25, 2024 @08:25AM (#39815436)
Who picks the meta reviewers?
Re:Why isn't preprint review being adopted? by notatoad on Sunday March 24, 2024 @06:24PM (#39811058)
we need twitter community notes for science
Re:Why isn't preprint review being adopted? by rhelz on Sunday March 24, 2024 @10:33PM (#39812404)
chuckle recall, a paper is given to 3 or 4 reviewers. No need for hierarchies of reviewers; it’s more like a jury; if all the reviewers more or less come to the same conclusion we can have a high confidence that the decision is the correct one.

Under the proposed plan, if one of the reviewers gave a review which was radically different, or otherwise obviously slap-dash job, payment could be withheld and another reviewer commissioned.