No, you don't understand this.

The Lancet is one of the most impactful journals in medicine, and has played an important role in shaping how medicine is practiced. It is rigorous in its process, but it's not foolproof. So yes, there are some bad studies that come through, but that is why healthcare professionals are TRAINED to interpret studies--we separate all the bullshit from the legitimately useful stuff.

You should see some of the crap that comes through in some of the lower-impact journals.

> The combination of preprint servers + twitter has proven far more effective in looking after humanity’s best interests.

This view is incredibly false and dangerous for so many reasons.

Well if healthcare professionals are trained to interpret study, and separate the wrong from the useful stuff I guess it won't be so hard for The Lancet to hire some of them to review papers... I tend to believe Didier Raoult when he says in his last interview that anybody who works in medecine could easily tell this paper smelled really fishy, and I am sure it would be less costly for The Lancet than the reputation loss they are going to suffer.

You act as if highly-trained professionals don't ever disagree.

Most professionals found Raoult's own published claims of the 100% cure rate of hydrochloroquine combined with azithromycin based on small groups without proper controls fishy, for a start.

I tend to agree with you in whole, with caveat that past contributions don't speak to current value, necessarily. I'm always reminded of the ruthless statement, "God cares not what you have done, but what you are doing". I say it tongue-in-cheek, but there's wisdom to it.

Also, can we all stop describing any view or information as "dangerous"? It sounds like my mom, or religious people from the 80's.

False is one judgement and objectively proven.

"Dangerous" smacks of a censorious nature, which is subjective and, sorry, often based on a conflict of interest/agenda.

Encourage everyone to read Matt Taibbi's recent article about the recent rise of "virtuous" censorship as food for thought: https://taibbi.substack.com/p/temporary-coronavirus-censorsh...

You are confusing impact with quality. They are not the same thing.

> So yes, there are some bad studies that come through, but that is why healthcare professionals are TRAINED to interpret studies

So healthcare professionals are trained to separate the wheat from the chaff, but peer reviewers are not?

> You should see some of the crap that comes through in some of the lower-impact journals.

Thank goodness that "healthcare professionals are TRAINED to interpret studies."

I am a healthcare professional, and I've published in one of the Lancet Journals. I understand perfectly fine.

Here's the thing - The Lancet is not impactful. Researchers choose to publish their impactful research in The Lancet. Other researchers donate their time to peer review this research. It is the researchers that have shaped how medicine is practiced, not The Lancet. It is the researchers and the peer reviewers that are rigorous. The Lancet itself is none of these things, it is a business run by people who don't do research. Without the people who actually do the work, or the patients that volunteer for the research, it is nothing. The Lancet, like all top tier journals, has long forgotten this distinction.

The problem isn't healthcare professionals lacking training. Take the Wakefield study. Did any doctor decide to stop offering vaccines because of that study? No. But it had a large impact on a the anti-vax narrative. These top journals have influence far beyond the professional sphere. This is why your suggestion we should give them a free pass is dangerous.

I think it is clearly true that rapid publication and out in the open discussion and peer review is very healthy. I don't see how you could argue otherwise? Why wouldn't I want to be able to read the opinion of someone I respect intellectually eg Andrew Gelman, on a study they have decided to comment on? How is that 'dangerous' if I am a healthcare professional who should be able to critically review a published paper as you suggest? Why do I need to rely on the reviewers the journal has chosen? The Journals want to keep things as they are to maintain their importance and their bottom line. We suffer as a result.

Healthcare pro also. You are right. Academic publishing has become extremely toxic to both science and the healthcare system in general.

A major issue with this is the political and hierarchical benefits of publishing. It encourages professionals to focus on that, to the point that clinical work is now looked down upon and producing loads of shitty papers will propell you to the forefront of the academic star system and make you rich. It's truly cancer for our healthcare system.

But I keep hearing that the definition of good science is "peer reviewed" science. So you are right that it is not the peer review that makes the science, but what about the science us laypeople can rely on? How else can we tell a serious claim from a fantasist one?

The scientific method is hard to explain, and scientists frequently mention peer reviewed papers when talking to the public. I think that this is mainly to dismiss things like health anecdotes that spread around the internet. We don't have time to explain in detail why your uncle's implausible and untested miracle cure for whatever disease is unlikely to work.

There's no easy answer to which sources are reliable, but it helps to look at what reputable organizations say. They typically base their advice on the majority opinions of relevant experts. Scientists are human, so they are bound to make mistakes, but in the whole they have made demonstrable progress in a variety of fields.

Peer reviewing is mostly orthogonal to journal publishing. Publishing in a journal generally requires a peer review, but there's no reason why you need a journal publication to get a peer review. Journals largely exist to manage prestige and career advancement in academia.

As for what science lay people can rely on? I'm not sure there's a simple answer to that question.

I think you either have to find scientists you can trust (which presents a chicken & egg process) or study the field yourself.

That said, there are some good heuristics. Good scientists are able to show their reasoning process and explain why X is bad/good science. Bad ones handwave everything and push credentialism or rely on fallacious reasoning.

For example, I remember long ago when Ars Technica posted a detailed explanation of both what homeopathy is and how we know that it's completely bogus, why "water memory" doesn't and cannot exist, etc.

Meanwhile, in another failure of peer review, Nature published nonsense on "water memory" back in 1988:

https://en.wikipedia.org/wiki/Water_memory

How can you criticize The Lancet for a small number of faked studies that have gotten through, but then praise pre-print servers, which are rife with complete nonsense?

Pre-print servers are great for getting results out quickly, but the level of quality control is extremely low. Journals are slower, but have a higher level of quality control.

There is a trade-off between speed and quality control, and both pre-print servers and journals sit at useful points in that trade-off.

Given the pressure to get information that might help fight CoVID-19 out quickly, journals have probably shifted towards the "quick/low-quality" end of the spectrum in the past few months. They'll move back when things calm down.

Journals have been hype/politically-driven since as long as one can remember. And peer-review quality is a joke.

Preprints are not perfect, but I'd wager that anything free and truly open to review is better than what we have now.

Specifically I am praising pre-print servers in the current crisis, not in general. I don't propose they can replace journals wholesale in the near future.

The harm done by bad research in the Lancet is colossal because the results are widely read and disseminated, that is obvious isn't it? Pre-print servers are still quite obscure. Anyway, arxiv has been running for a long time with much success in certain fields, and hasn't been overwhelmed with nonsense, so your characterisation of pre-print servers is poorly calibrated and hyperbolic.

Also the Lancet charges money for its products, and can therefore expect to be held to greater account.

So the pre-print servers are better because not a lot of people read pre-prints?

> arxiv has been running for a long time with much success in certain fields, and hasn't been overwhelmed with nonsense, your characterisation of pre-print servers is poorly calibrated and hyperbolic

I just said that the level of quality control on pre-print servers is extremely low - which it is. They check for very little beyond blatant plagiarism and obvious junk (determined within minutes).

I didn't say that pre-print servers are useless. They're very useful, but it's undeniable that prestigious journals apply an additional filter, which is much more exacting.

If you were to pick a random study from a pre-print server, and a random study from the Lancet, which do you think is more likely to be reliable? Which do you think has undergone more thorough peer-review?

You don't seem to be reading my comments. I agree with you that pre-print servers are not useless, and that prestigious journals are also not useless. Pre-print servers obviously have a lot of questionable content, this is immediately obvious. As I said, they are not a replacement for peer-reviewed journals (except maybe in the early stages of a pandemic, which was my long departed original point). It is also true that good research gets published in prestigious journals. Lots of good research also doesn't get published in prestigious journals because the editor of the Lancet decides it isn't interesting enough, or they chose clueless or callous reviewers who kill the paper in peer-review, or they just accepted a paper with the opposite result last week or for many other arbitrary reasons unrelated to the quality of the research or utility to humanity.

That they have that level of impact would appear to be problematic, given this news, rather than the other way around.

It is exactly such kinds of ignorants of basic statistics, the faulty generalizations of the effectiveness of established journals, have failed the humanity.

Is there a statistical demonstration of the effectiveness of established journals? It seems like you would have to do some form of counterfactual study to prove/disprove this. As far as I know, the argument of the effectiveness of journals only stems from the survivorship bias (e.g. all things considered research until recently has been published by journals).

No, you can look at the stuff people put up on the preprints to see what would happen if there's basically no screening at all.

The preprint model has actually been wildly successful for many communities (https://arxiv.org/, https://www.biorxiv.org/). I don't think anyone is arguing against peer review, but the idea of peer review and an established journal is being conflated. If anything, publishing established journals can create a level of implied authority that reduces much need scrutiny.

> They have a poor record frankly with this kind of thing, Andrew Wakefield et al chief among them.

Would you care to name some others?

Of the top 10 most highly cited papers that have been retracted, 3 of them are in the Lancet, more than any other journal [1].

[1] https://retractionwatch.com/the-retraction-watch-leaderboard...

That number can't be interpreted without context — how many of the top 100 most highly cited papers are in the Lancet? If it's ~30 then your data point doesn't mean anything.

Papers in Lancet are generally more highly cited than average, since it's the most prominent and respected medical journal. Is it any surprise that even the retracted papers in Lancet are highly cited? Surely that's a function of their visibility.

I'm not sure what kind of conclusion we can take from that dataset. It is only 10 papers and it doesn't take into account the average number of citations for the non-retracted articles in each journal.

I was asked to provide examples of the Lancet's poor track record, which I have done. I am not trying to prove it is better or worse than any other journal.

How do you know that it is a poor track record if you do not know how it is ranked against others?

Even if the Lancet had published all top 10 highly cited papers, wouldn’t 3 of them being retracted be concerning to you?

It's a list of "Top 10 most highly cited retracted papers", not "Top 10 most highly cited papers". Without more information about highly cited papers in general (e.g. what proportion are in the Lancet?), we can't tell exactly how concerning 3 being from the Lancet should be.

A top tier journal has 3 papers with >500 citations that turned out to be bullshit in the last 10 years. One of them is probably the premiere example of academic misconduct in the last 50 years, catalysing the anti-vax movement and causing untold harm. What's your definition of poor?

> I am not trying to prove it is better or worse than any other journal.

In that case it's not really clear why you decide to stop reading the Lancet, rather than any other journal? Or have you stopped reading all journals

EAT-Lancet: https://www.thelancet.com/journals/lancet/article/PIIS0140-6...

Autism and vaccine study https://www.npr.org/sections/health-shots/2010/02/lancet_wak...

That /is/ the Wakefield study.

> The combination of preprint servers + twitter

Filtering noise with noise?

The great thing is that there's such a range of noise to choose from, you're sure to be able to find some you like.

I imagine it'd be hard to bring preprint studies + twitter threads to head of your department and use that as the basis for institutional policy change? Although it could change individual physician's practice which I guess is where things start?

Well there were people that thought books were the work of the devil when they first came out. It takes a 2 or 3 generations to get the most important ideas completely accepted. The Flynn effect is helpful here.

The Wakefield argument is weaker than it should be. Wasn't his paper retracted because of its funding source, not because of specific errors in the paper itself? At least that's what Wikipedia's article on Wakefield claims. And of the co-authors, they asked for retraction because of the stink in the press over the funding, again, not because of actual errors in the science. Those don't actually seem like valid reasons for retraction, on their face.

It had 12 co-authors so that's a lot of peer review to start with, before whatever the Lancet applied.

Uh, secret meetings and confidential cover-ups? This reads more like a spy novel than a trustworthy document.

Sorry.