arxiv was about distribution. It didn't replace peer review - articles were still submitted to journals and published there too.

If an article was posted to arxiv and not a journal, the odds of a citation went down massively. And the journal it was submitted to was a factor in whether or not we read it. When articles were eventually published, most authors also updated the preprint with the post peer review version.

Basically it meant that (1) it was easy to keep up to date with what everyone was working on, and pick up interesting new stuff (2) most citations, post 80s, you saw in whatever paper you were reading, you could look up on arxiv and be reading it in seconds.

In machine learning, for the most part, arxiv is used to avoid peer-review. Or a way to "publish" work that has been rejected by a peer-reviewed publication, of course.

And to be more cynical, it's also a convenient source of references to pad up a Related Work section and make it look like incremental work is part of a growing body of groundbreaking new work. /jaded

Edit: well, I'm not just being cynical. The fact that everyone can put their half-baked papers on arxiv means that the 90% of work that is crap, per Sturgeon's Law, is now a much bigger quantity than ever before and one must sift through reams and reams of crap before finding work that has any meaningful results to report. Again, that's the case in machine learning specifically. I don't know about other fields.

Arxiv only lacks the initial quality filter by peer review.

I'm also working in the field of machine learning. In those niche fields I work more specifically (speech recognition), I can usually still get a lot out of Arxiv-only papers. I can pretty easily see the main idea and see if there is some usefulness in the paper or not w.r.t. my own research e.g. by good experimental analysis. In don't really feel overwhelmed in the amount of papers. I don't really see the problem.

I'm surprised that they no longer use the term "preprint" at all, at least it's nowhere to be found on the homepage or "about" section.

The consequences of this amnesia are hilarious: https://twitter.com/gustavnilsonne/status/138948729731431219...

> Why do we call it "preprints"? The term seems to imply that work is preliminary or unfinished. As far as I can tell, the term introduced by @arxiv , the first online repository for scientific manuscripts, is "e-print". Is "preprint" a marketing device invented by publishers?

In theory you should read arXiv and cite the conference version, but often people cite arXiv and nobody cares because google scholar mostly combines things properly.

Or, instead, you get a much larger audience reviewing your article, instead of the 2 to 4 reviewers involved in the paper publishing process :-)

You can get comments from critical and interested readers, you upload a new version of the paper to arXiv, and repeat the process until the article is ready for submission/publishing.

In other words: I think that an article on arXiv (on average and as a whole, individual articles may be exceptional) is subject to much wider and more extensive scrutiny and verification than the average article during the paper publication process.

So, yeah, I disagree. The level of scrutiny one gets from putting their work on arxiv doesn't compare with peer review by experts in one's field.

Discussion starts in blog posts, on Twitter, in conversations.

Then longer blog posts and code demos.

Then pre-prints and more blog posts about that.

Then finally journal.

You get incredibly less review at the publication stage — and your idea “experts in one’s field” don’t also use the internet is hilariously wrong.

I'm glad to hear you're amused by my comment. Where did I express the idea that '“experts in one’s field” don’t also use the internet'? Can you please show me?

That's overly optimistic. What you mean is "with peer-review you know that someone will at least skim your work and maybe catch errors."

I agree with the rest of your points though. Zero peer review is probably not a great solution.

It's very possible to just keep submitting papers to journals until you get someone who shrugs and lets it through. Reviewers aren't paid, and while reviewing is encouraged it basically does bupkis to further your career (and academia is very cut-throat).

Is that a failure of peer-review though? Or is it more of a consequence that even if you do everything right you'll still get some false positives some of the time, and negative results tend not to get published at all?

(i.e. the problem that pre-registration for medical trials hopes to solve - https://en.wikipedia.org/wiki/Preregistration_(science) )

It raises questions around using peer-review as a mark of scientific quality. If it is, why does peer reviewed science time and time again turn out to have enormous glaring quality problems?

In fact, peer review is often more about novelty and importance rather than rigor. Very well structured research will get rejected if it isn't seen as contributing to the field in a nontrivial way.

A few fields (psych is the big one) are funding replication grants. That'd be the true mark of quality.

This is actually deeply problematic, as a part of what's driving the replication problems is specifically publication bias. If the only thing that gets published are unexpected or noteworthy results, you're selecting for statistical aberrations.

Peer review is not infallible, neither is it a mark of scientific quality, as you say. I appreciate that people tend to use it like that- you read, in articles in the lay press, that such-and-such study was "not published in a peer-reviewed journal" as if to say that we can't be sure of the quality of the study and that, conversely, if it was peer-reviewed, we could. That's not right. Peer review is not a sufficient condition to guarantee correct results. It's not even a necessary condition. Like I say in my comment above it is one mechanism of modern research practices that helps improve the quality of published papers.

Wrong, sorry. Disclaimer: I'm not a homeopath and don't go to any. Most of them are, in fact, quacks.

There are, unfortunately, chronic diseases which can be treated but not cured. Mainstream doctors and Big Pharma make a living off of those. If you have one of those, "well, why NOT try a homeopath?" is a perfectly rationale response.

"Sinus rinsing" is something that might be termed "homeopathy." It IS medically respectable, unlike most of their "treatments" (like magnets).

I wonder if academic papers could benefit from a similar process of communal development (though it flies in the face of the dead-tree format of academic papers, but we're talking about Arxiv here, not Elsevier). Even if suggestions don't get merged, having a repository that captures discussion around the central artifact is also useful (such as when a maintainer decides that a feature doesn't belong in a project).

https://openreview.net/group?id=ICLR.cc/2022/Conference&refe...

This is a remarkably naive (and charmingly optimistic) view of peer-review.

For the vast majority of the history of academic research peer review did not play a major part. It wasn't until the 1970s that the modern day peer review system emerged and it did so as a reaction to a funding crisis in academia to produce the illusion of legitimacy.

It of course has done no such thing. It has done nothing to limit the reproducibility crisis which has shown it's ugly head in nearly all scientific fields of study. One could even argue that peer review coupled with a publish or perish culture is the cause of such a failure of science.

This also means just about any great scientific discovery you can think of prior to 1970 did not go through the peer review process as we know it today. If science seems less exciting today, peer review is certainly one of the reasons for this. I'll leave you with Geoffrey Hinton's words on the subject:

> Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

Laking peering review is a feature, and not just because it speeds up time to results.

Great way to join a convesation. Well done.

I didn't say anything about "the vast majority of history of academic research". I talked about my experience of peer review of my work. Are you responding to someone else's comment but quoting mine?

I think sometimes reviewers hide behind their anonymity to give snarky responses, and using their power over you, knowing you will try your very best to implement all of their suggestions in order to publish faster. I just think the whole process should be more transparent.

https://www.youtube.com/watch?v=gv5mI6ClPGc

(Discussion w Noam Chomsky)

Peer reviewing is essential, of course. But note that in most journals reviewers are not getting paid. They have even less incentives to find mistakes in your work than you are. I have no idea how such an utterly broken system emerged, where journals siphon billions of tax money exploiting the work of scientists.

This is quite separate than the exploitative aspects of the big scientific publishers. Reviewers are certainly exploited by publishers, because publishers make money from reviewers' free work. But reviewers would do the work for free anyway because that's what they expect others to do.

The situation for conferences in CS is basically how it works in journals in other fields. There may be more low-quality submissions to journals, but at the same time, the extreme competitiveness means most high-quality submissions are rejected also. The "solution" has been the appearance of open-access journals which are less concerned with subjective reasons for rejection like "impact", and scale to accept more papers. But they cost thousands of dollars. Open archives are a more democratic solution.