Yes, and that's what I understood you to mean and a reddit-like voting system for papers is what I was talking about. (I even wrote "reddit-like voting system" as my last sentence in the post you replied to.)
>You're wrong. Firstly, it's often impossible to judge the significance of a result even soon after publishing.
I'm not talking about "significance" in relation to the whole of human history. I mean "significance" in the sense of a threshold for requesting other people to invest their time & effort to study the paper and also the subsequent subscribers to read.
>, reviewers already donate their time to review papers, so the human effort is already there.
Yes, but the key you're missing is that they donate their time because a respectable _somebody_ requested them to do so. The triggers for motivating that donated effort matters.
If I'm a working scientist working 50 hours a week, I'm not logging into a reddit-like voting system and investing hours to read (mostly bad) papers that have 0 votes so I can submit my 1 vote. No, somebody I respect (e.g. a journal editor) needs to ping me and say, "hey, this paper looks interesting, can you review it?" After I peer review it, I may get some quid-pro-quo such as free access to the journal's other papers and/or when it's my turn to submit a paper, the editor will consider forwarding it. So yes, I'm not "paid in cash" but I'm paid "in kind" with some intangibles.
That social dynamic is not replicable in a reddit-like voting system.
Therefore your following scenario of:
>whereby some random selection of scientists obtain access to early drafts
... is not realistic because scientists do not want to be part of random pool to be assigned random papers. That's contrary to basic human nature! Scientists want papers referred to them by another human and not an algorithm. It's not a technology problem, it's a human problem.
All the software suggestions to solve this from programmers including ideas such as a reddit-like system, or a github-like system, or a blockchain type system, etc ... they all misunderstand the culture and psychological motivations of publishing.
>What is scientific consensus if not a vote among some random sampling of scientists studying a field.
Yes, I can see that if you reduce the problem to the simple abstraction of "votes", it seems like building a software platform to formally capture votes is The Solution.
What your simplification leaves out is all the human signaling required before people render their votes. That's the human dynamic that collaboration software doesn't solve.
(In any case, an accepted published paper isn't "consensus" per se; it's just another piece of evidence for people to weigh. The paper just passed through enough human curation filters that may be worthwhile to read.)
I agree with you that it is difficult to improve on the peer review process. At the same time, several researchers have pointed out that publishers aren't really involved in the peer review process. It should be possible to coordinate journal publication through a software platform, replacing the publishers without affecting the peer reviewers or editors.
Yes, I agree employees at Elsevier aren't doing the actual peer review. However, they do critical filtering work before the any journal editors see the paper. See example from Elsevier employee Angela Kerr.[1]
The programmers who think they can revolutionize academic publishing by developing a new peer-review collaboration platform don't even realize that people like Angela Kerr exist. (See repeated disgust in this thread about "peer reviewers are unpaid". That's true but it also makes Ms Kerr's paid work invisible.)
In addition to basic copy-editing, she prevents crackpot articles such as "Darwin's Theory of Evolution is proven wrong by Trump" from reaching the journal editors and wasting their time. (She may inadvertently forward some bad articles but she has enough education to reject many of them outright to minimize the effort by the journal's scientists.)
Administrative people like her cost money. One would have to replicate her duties somehow in the new software platform. That's because the journal editors (scientists) are busy and they absolutely will not want to log into a new collaboration platform and stare at a long queue of unfiltered papers that everybody uploaded. Even though Angela doesn't do the "peer review" per se, the journals still want her to make a first pass cut. None of the ideas I've seen from programmers integrate her function into their model of what peer-review software should do.
With something like Stackoverflow, you have volunteers scrolling through the queue of questions and deleting/editing bad ones so their highly valued experts don't have to waste time with them. That's workable for "programming questions" because they are typically less than 100 words. Even with smaller workload of not reading 20000-word research papers, volunteers still get burned out and quit.
For highly prestigious papers like Nature and Cell that would attract a high volume of uploaded papers, it's very doubtful you'd have volunteers like Stackoverflow to slog through all the junk. Therefore, you'd have to staff up some copy editors and pay them money to do unpleasant work.
The problem is that Elsevier charges an insane amount of money for subscriptions (~$800 million in profits with 36% profit margin). Yes, a new system can charge much less money and be more efficient but you still have to do the work that Elsevier does. The journal editors don't want to do it. The potential software model of synthesizing ideas from reddit+github+dropbox+bittorrent+Sharepoint+blockchain ... don't account for Angela Kerr. Programmers think only in terms of distributing files like pdf and LaTeX. That's why they think academic publishing is a solved software problem and not a human effort problem.
>It should be possible to coordinate journal publication through a software platform, replacing the publishers without affecting the peer reviewers or editors.
The software platform can replace Elsevier in particular, but it can't replace the fundamental "work activity of a publisher". Angela Kerr (or whoever employs her) wants to be paid to do the unpleasant work that journal scientists don't want to do. And she doesn't want to be paid in reddit-like "karma votes" or github-like "stars". The new system can pay Ms Kerr less money than Elsevier (and hopefully pass those savings on to subscribing libraries) but let's be clear that the journal scientists absolutely do not want a new software platform without her.
[1] https://www.elsevier.com/connect/confessions-of-a-managing-e...
There is great need for a software nonprofit to enable scientists and editors to collaborate with minimal overhead, because even $2000/paper is too painful. The true marginal cost should be under $200/paper for scientific editing,[1] and at that price point, open access will overwhelmingly win. If we can build such a platform, I believe that Elsevier will find it very difficult to retain their journal editors.
You're attacking a straw man. I said reddit-like, not reddit-clone. There are plenty of obvious problems with reddit's voting system. So what?
What is scientific consensus if not a vote among some random sampling of scientists studying a field. What's left is to formalize this process in an open system so we're not beholden to the publishers and other parties with perverse incentives.
> Evaluating dense papers is not an "information technology workflow problem" that's waiting for collaboration software to solve. The bottleneck is human effort to properly read papers to judge its correctness and significance.
You're wrong. Firstly, it's often impossible to judge the significance of a result even soon after publishing.
Secondly, reviewers already donate their time to review papers, so the human effort is already there. What's needed is a well-recognized system encoding a publishing process, whereby some random selection of scientists obtain access to early drafts with a way to provide commentary and ratings on various metrics, with final publication after some quality consensus is reached among the random selection of reviewers (as a rough idea).
This is exactly an information technology problem.