I'm not saying you're wrong — I've got my name on exactly one paper so what would I know — but that being true would suggest that peer review is fundamentally broken.

No, it would suggest that every paper has something in it that can be considered a fatal flaw by a person biased against it. Peer review is precisely supposed to admit this and minimize flaws in conclusions given inevitable flaws in methodology.

Sort of a research paper equivalent of:

"If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." by Cardinal Richelieu

Peer review is not a certificate of truthfulness, peer review just says that a grad student reading the paper did not found anything too suspicious there and it looks like a typical paper in that field.

Peer review, at its core, is a social consensus process. It can work, but it is structurally inclined to propagate agreement.

I wonder if it serves to reinforce this problem, as those more motivated to reinforce certain literatures or perspectives will surface those repeatedly in reviews.

There's a popular work that comes to this conclusion in its beginning chapters [0]. The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted. Typically those who initially disrupt a scientific dogma are not treated well. Eventually the old guard literally dies off and the new ideas can begin to take hold.

[0] The Structure of Scientific Revolutions, T. Kuhn

> The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted.

This can happen of course, but it's not really what Kuhn is talking about. Kuhnian paradigm shifts are to science what massive refactorings are to a codebase: the problem isn't that the old code was wrong (though it might also be wrong), it's that it can't be extended to meet new requirements. And while some of us are better at writing extensible code than others, no one gets it right every single time.

A huge part of the problem is that the layperson is taught otherwise.

2nd grade we learned the scientific method. I don't remember peer reviewed being a thing.

Replication was critical.

Also, I've seen peer reviewed papers with atrocious stuff in them, unless I know the peer, it doesn't mean much to me. Give me 100 independent groups coming to the same conclusions. That has far more weight than a few of your friends/professors reading your paper.

Not even that. I was a grad student that reviewed a paper submitted to a well-known journal from a heavyweight in my field. The paper's own data showed that the authors were reporting 95% noise. My advisor rejected the paper only to see it published a few months later (in another famous journal) after the authors removed the data that allowed us to detect the noise throughout the data.

Not exactly. Control for that in a future experiment?

To be fair, as you start to leave chemistry, true knowledge(the goal of science) is impossible to find. The best we can get is little glimpses of the truth.

To be fair, I have seen studies that were pretty bullet proof. It just seems that most are lazy or simply impossible to prevent variables.

I mean I think peer review kind of is fundamentally broken, but at the same time this phenomenon isn't necessarily a sign of it.

Even within technical papers, let alone pop writing or internet discussions, many of the claims citations are provided for are broad e.g. "X can increase people's anxiety". A very rigorous study might exist showing that when X happens a specific way it does increase some form of anxiety in some specific subset of the population, but using it as a citation for the broad statement without further context can be misleading at times.

It's in fact possible that by considering different subsets of the same statement you'd get an opposite directional effect. That can certainly be used to confuse people, especially in a situation where it is unlikely most readers will dive deeply into the cited works.

As far as nit picking - the same general principle applies. There will always be some tradeoff between scope, rigor, and available resources to do the study. If we waited for papers to be perfect in both correctness and interestingness/utility there would be almost nothing to ever publish.

So I think the systemic problem here is moreso an undervaluing of review article and text book type resources (both reading them and writing them) in favor of vomiting out random individual paper citations for whatever claim. Science needs more heterogeneity in the roles different PIs fill for the system.

Improving peer review process would be great (and might indirectly help), but I don't feel it's the root.