Healthcare pro also. You are right. Academic publishing has become extremely toxic to both science and the healthcare system in general.

A major issue with this is the political and hierarchical benefits of publishing. It encourages professionals to focus on that, to the point that clinical work is now looked down upon and producing loads of shitty papers will propell you to the forefront of the academic star system and make you rich. It's truly cancer for our healthcare system.

But I keep hearing that the definition of good science is "peer reviewed" science. So you are right that it is not the peer review that makes the science, but what about the science us laypeople can rely on? How else can we tell a serious claim from a fantasist one?

The scientific method is hard to explain, and scientists frequently mention peer reviewed papers when talking to the public. I think that this is mainly to dismiss things like health anecdotes that spread around the internet. We don't have time to explain in detail why your uncle's implausible and untested miracle cure for whatever disease is unlikely to work.

There's no easy answer to which sources are reliable, but it helps to look at what reputable organizations say. They typically base their advice on the majority opinions of relevant experts. Scientists are human, so they are bound to make mistakes, but in the whole they have made demonstrable progress in a variety of fields.

Peer reviewing is mostly orthogonal to journal publishing. Publishing in a journal generally requires a peer review, but there's no reason why you need a journal publication to get a peer review. Journals largely exist to manage prestige and career advancement in academia.

As for what science lay people can rely on? I'm not sure there's a simple answer to that question.

I think you either have to find scientists you can trust (which presents a chicken & egg process) or study the field yourself.

That said, there are some good heuristics. Good scientists are able to show their reasoning process and explain why X is bad/good science. Bad ones handwave everything and push credentialism or rely on fallacious reasoning.

For example, I remember long ago when Ars Technica posted a detailed explanation of both what homeopathy is and how we know that it's completely bogus, why "water memory" doesn't and cannot exist, etc.

Meanwhile, in another failure of peer review, Nature published nonsense on "water memory" back in 1988:

https://en.wikipedia.org/wiki/Water_memory

How can you criticize The Lancet for a small number of faked studies that have gotten through, but then praise pre-print servers, which are rife with complete nonsense?

Pre-print servers are great for getting results out quickly, but the level of quality control is extremely low. Journals are slower, but have a higher level of quality control.

There is a trade-off between speed and quality control, and both pre-print servers and journals sit at useful points in that trade-off.

Given the pressure to get information that might help fight CoVID-19 out quickly, journals have probably shifted towards the "quick/low-quality" end of the spectrum in the past few months. They'll move back when things calm down.

Journals have been hype/politically-driven since as long as one can remember. And peer-review quality is a joke.

Preprints are not perfect, but I'd wager that anything free and truly open to review is better than what we have now.

Specifically I am praising pre-print servers in the current crisis, not in general. I don't propose they can replace journals wholesale in the near future.

The harm done by bad research in the Lancet is colossal because the results are widely read and disseminated, that is obvious isn't it? Pre-print servers are still quite obscure. Anyway, arxiv has been running for a long time with much success in certain fields, and hasn't been overwhelmed with nonsense, so your characterisation of pre-print servers is poorly calibrated and hyperbolic.

Also the Lancet charges money for its products, and can therefore expect to be held to greater account.

So the pre-print servers are better because not a lot of people read pre-prints?

> arxiv has been running for a long time with much success in certain fields, and hasn't been overwhelmed with nonsense, your characterisation of pre-print servers is poorly calibrated and hyperbolic

I just said that the level of quality control on pre-print servers is extremely low - which it is. They check for very little beyond blatant plagiarism and obvious junk (determined within minutes).

I didn't say that pre-print servers are useless. They're very useful, but it's undeniable that prestigious journals apply an additional filter, which is much more exacting.

If you were to pick a random study from a pre-print server, and a random study from the Lancet, which do you think is more likely to be reliable? Which do you think has undergone more thorough peer-review?

You don't seem to be reading my comments. I agree with you that pre-print servers are not useless, and that prestigious journals are also not useless. Pre-print servers obviously have a lot of questionable content, this is immediately obvious. As I said, they are not a replacement for peer-reviewed journals (except maybe in the early stages of a pandemic, which was my long departed original point). It is also true that good research gets published in prestigious journals. Lots of good research also doesn't get published in prestigious journals because the editor of the Lancet decides it isn't interesting enough, or they chose clueless or callous reviewers who kill the paper in peer-review, or they just accepted a paper with the opposite result last week or for many other arbitrary reasons unrelated to the quality of the research or utility to humanity.