Representative quotes:

"Is there any idea so outlandish that it won’t be published in a Critical/PoMo/Identity/‘Theory’ journal?" tweeted the Harvard psychologist Steven Pinker.

"Three intrepid academics," wrote Yascha Mounk, an author and lecturer on government at Harvard, "just perpetrated a giant version of the Sokal Hoax, placing … fake papers in major academic journals. Call it Sokal Squared. The result is hilarious and delightful. It also showcases a serious problem with big parts of academia."

Mounk didn't read the hoaxers writeup carefully enough to be a reliable judge; he gleefully tweeted two papers from the hoax that were rejected everywhere they were submitted, as evidence of the important problem the hoax uncovered --- this despite the fact that the hoaxers did in fact provide an accounting of what got accepted where (even though their writeup at the beginning of the piece misrepresented that accounting).

From what Pinker wrote, who knows if he even read it?

> can be fooled some of the time

7⁄20 or 7⁄17, depending on how you want to count is concerning.

Shouldn't the default presupposition of any research be a lack of novelty?

I think generally people tend to assume that colleagues aren't dishonest, and mostly peer-review is supposed to catch honest mistakes.

You could try working without any faith in your colleagues of course, but you are probably better of switching workplace if that sounds like reasonable behaviour to you.

What your describing sounds nothing like rigor.

The rigor comes from the experiment and the future analysis by other researchers, not from the review. You are expecting something from peer review that never exists.

Alternative take: A system designed to detect bad research failed to detect blatant falsehoods, algorithmically generated noise, rabid propaganda and other seemingly easy things to detect. That PhDs worked on this doesn’t mean the system didn’t fail spectacularly.

> doesn’t mean the system didn’t fail spectacularly.

You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science.

We have to stop pretending like the peer review process is some kind of magical filter of true science, or that it was ever intended to be as such. That notion leads to the faulty idea that if it's published, it's true. Junk science gets published, even if it's not disingenuous, and that's probably true no matter how great you make the peer review system.

What is the point of journals if quite literally anything can be published in them? Isn't the point that they serve as a basic filter for obvious noise? Shouldn't we be critical of a peer review process that doesn't function at even the most basic level?

A filter that does filter out the vast majority of obvious noise is very useful even if it lets something through. And the current peer-review does that, it throws out almost all of the spam; it pushes useful work of lower quality/impact down to less selective venues (generally with more specific subfields, so that I can read impactful work of domain X and less impactful work of subdomain X.2 but not X.1 and X.3); and even for accepted papers the process often results in significant improvements to them as a result of the reviews.

Explicit fraud isn't that common, since it does have risks to the submitter that far outweigh the (limited) benefits of getting an extra publication on your CV.

> quite literally anything can be published

I've got a stack of rejected papers that beg to differ.

> Isn't the point that they serve as a basic filter for obvious noise?

Yeah, and obvious noise is usually rejected. But sometimes it is not so obvious what the noise is, and that's why the scientific process is bigger than the peer review system.

> Isn't the point that they serve as a basic filter for obvious noise?

Totally, there is a lot to fix about it. But if you really want to put your effort into changing the system, I don't think you should focus on peer review. Destroying the business model of journals is where I would start.

Why is this being downvoted? This is a description of the "invisible boot" as Broad & Wade call it.

I'm not saying that peer review is the end all be all of truth, but it does serve some purpose. And in this case it didn't provide any value at all.

Peer Review is not set up to detect fraudulent research and attempts to do so are fraught with risk, since established researchers often have a hard time believing in results from new paradigms. What the heck is an author supposed to do if a reviewer says "I think these numbers are made up"?

The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed. But we always approach reviews with a charitable lens, even when reviewing for extremely low-acceptance outlets.

The process that detects the bad research comes afterwards, as it should.

> The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed.

It clearly failed to do that in this case.

Ok? That somebody with a year of free time and a pile of fraudulent attempts can get through sometimes is not exactly a surprise.

A system that prevents what the authors did here would necessarily reject a huge amount of useful research.

The whole point of a peer review system is to detect fraud. If these bullshit papers got published, who knows how many of the other papers are of equal quality.

The point of peer review is to stop bad research, not fake research. The reviewers are supposed to make sure that the methods used to acquire the data are clearly documented and that the conclusions drawn from the data follow from the data that was collected.

They can't verify that the data collected actually was collected without literally watching over the shoulder of the researcher while the researcher was collecting it, which is obviously not feasible. A researcher actively trying to deceive the reviewer with fake data but with a good method for collecting that data and analysis of the data that would be correct if the data were real is not something that should be caught by a reviewer. That should be caught by someone reproducing the research after publication.

>The point of peer review is to stop bad research, not fake research

I would submit that "fake research" is, in fact, bad research.

To a reviewer with three hours to spare, fake research is indistinguishable from good research. Unless you want to give me months to review a paper, I will not be able to detect fraud.

Instead I rely on the fact that the community will attempt to integrate these ideas after publication and arrive at the conclusion that the paper was garbage.

If you only have three hours, can't detect fraud, can't tell if the the paper is even real. Then what's the point of even doing the peer review? Again, peer review offered no value at all in this case.

Peer review can still inspect whether the evidence presented in a paper would—if not fabricated—truly be enough to support the paper's conclusion. Mistaken analysis of honestly collected data, improper experimental methods, flawed proofs, etc. can all show up at review time.

You can certainly come to that conclusion if you'd like. Arxiv exists. But "peer review isn't especially usrful" and "these entire fields are nonsense" are very different conclusions.

Where did you learn that the whole point of peer review is to detect fraud?

The Wikipedia article on peer review says "Peer review...occasionally detects fraud, but is not designed to do so."[1]

The article cites a Nature article that says "Peer review is not currently designed to detect deception, nor does it guarantee the validity of research findings. It should, however, identify flaws in the design, presentation, analysis and interpretation of science and provide prompt, detailed, constructive criticism to improve research."

[1] https://en.wikipedia.org/wiki/Scholarly_peer_review

[2] https://www.nature.com/nature/peerreview/debate/nature05007....