I'll go one stronger and say "You Shouldn't Trust Any One Scientific Study You See".

Individual studies can be really interesting. They're important for researchers to know about to inform their future work. But any one study - even ones that are done honestly, with good methodology and sound foundations - can be just totally wrong. There could be confounding factors you couldn't have known about that completely invalidate the result. Your test subjects could be unusual in some way, your animal models could be a poor analogue for humans in this particular case, you could have just had really aberrant statistical flukes in your statistic sampling.

It's the body of scientific research, the dozens, hundreds, thousands of studies stacked on top of each other that bring certainty.

^ this. Especially on highly sensitive topics like medicine, I tend to be cautious when reading about the findings of any single study; whenever someone tells me "there's a study that says [flavour of the day]" my immediate reaction is to go see if there's a Cochrane meta-analysis :)

Yes. Science can be an adversarial process as people converge on the truth. Like a legal trial, you shouldn't take out of context any single statement from a defence or proceution lawyer as representative of the whole truth

Or it’s like seeing one great play in the middle of the game. It may be a beautiful play, it might have scored the team you’re rooting for a point, but until you know the whole game you don’t know who won.

That is a great analogy, just like scientific studies a defence or prosecution can argue to win, but in the end it's usually the evidence that swings the case...

I think you meant “shouldn’t”.

> Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder?

The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before?

> However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at.

Is it possible that your own preconceived notions about the author and the publication may have caused you to judge this way?

In any case, don't trust every comment posted on Hacker News (including my own) :)

> The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before?

Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.

>Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.

In my experience working with/contracting for neuro labs, a lot of researchers don't really know how to fully leverage the technology that's available, and often rely upon proprietary tools they have limited knowledge of, which doesn't bode well for being able explore for themselves.

The few that I have met that can push the limits of current technology are working in labs ran by the above…

I'm not sure how it is in other fields, but in convos from some other commentators on HN over the past years, makes me think this is not just in neuroscience.

Maybe the problem is that the skills needed to explore the solution space and communicate it effectively have gone up because the complexity it has added to the process without research labs/academia addressing the gap sufficiently? I don't think this is a problem with just labs or academia though, not many people in general have the skills to be able to leverage technology to it's fullest for even the most banal tasks.

I think it's one of those things where a few point shift in the average of the distribution really changes some numbers near the edges of it, though.

I don't see that chart supporting a sudden change. The time point could be moved quite a bit and tell the same story.

Also I had not heard of Kevin Drum before this, and had a positive view of Mother Jones. I'm left with a poor impression of Kevin Drum and a hit to Mother Jones' reputation after reading this.

You could go read the original paper and examine the actual data and scientific conclusions drawn from the underlying data as represented on that chart.

You could look for confirmation or disconfirmation of the hypothesis that preregistration leads to an increase in null results in other data sets.

Both of those seem like more useful ways to dispute this finding than squinting at a graph and finding fault with a blog post headline.

I'm not sure if you meant to reply to me, because every aspect of your post is wrong.

The "squinting at the graph" was assuming that there's a sudden change. I read the paper, and looked up several studies and came to the conclusion that the paper was being misrepresented by the blog text. And I agreed with the headline, but not with the blog text.

I would say that not all research needs to be held to the pre-registering standard of clinical trials. Some research is more exploratory, and should be interpreted that way.

Exploratory work is extremely valuable, but less trustworthy ipso facto.

That is right, And wrong.

There are explicit exploratory methods you can use to carry out research and that is very fine, but most research today is done using an inference model, where you formulate a hypothesis and use data to corroborate it.

That is one of the premises of big data that most people do not get: It enables you to do research explicitly exploratory again, while still being rigorous and somehow falsifiable..

What specific exploratory methods are you referring to?

Agreed, further:

>every significant clinical study of drugs and dietary supplements for the treatment or prevention of cardiovascular disease between 1974 and 2012

There is room for a lot of bias when selecting which studies are 'significant' and on topic. Not to mention deciding which metric to report from the study.

Their procedure was carefully documented, including the inclusion criteria and how exactly they categorized the studies. They even had two teams independently search the literature to find studies meeting the criteria: https://journals.plos.org/plosone/article?id=10.1371/journal...

Literature reviews in meta-analyses are usually conducted like this, with specified lists of search keywords and flow charts of inclusion criteria.

Problems getting harder is not really a valid explanation for the current state of science, or any particular field. At any given glimpse of society, their current state of the art is the best they've managed with all they know. Further progress is always harder, even when we now think of it as something that should have been easy. For instance the basic concept that things happen for a reason is something everybody understands now, even only a completely intuitive level - yet it was one of the breakthroughs of the Greeks. And Newtonian mechanics is also something people tend to understand quite intuitively, yet following the Greeks it would take the better part of two thousand years for people to consider that e.g. objects in motion stay in motion unless acted on by something else. Artistotlean mechanics held that continued motion required continued force. Artistotle seemingly did not consider friction, air resistance, and so on that cause objects to slow. Or just consider the tens to hundreds of thousands of years to develop writing, or to enter into the iron age, and so on.

There's some argument to be said that in times past there were fewer "scientists" working on these topics, explaining the delays. But I don't think that's fair. The reason I put "scientist" in quotes is because many of these things have minimal prerequisites and the average home would have all that's needed to experimentally test and discover these concepts. And their immense value, far beyond the academic, meant their discovery likely would have been able to rapidly spread regardless of the origin - so that, at least to some degree, precludes authority as a necessity.

Many things that now constantly elude our ability to understand, perhaps dark matter is a great example, will likely one day be child's play.

> With the advances in treatments for cardiovascular disease couldn't the problem just be harder?

I'm pretty sure we have no significant advances in the treatment of cardiovascular disease.

And given the alarmingly climbing rates of chronic diseases, e.g. cardiovascular disease, diabetes, obesity, cancer, I'm also pretty sure that the health care industry has done more harm than good when it comes to chronic diseases.

For everything else, for problems they can measure and treat with a pill, sure, I'm no anti-vaxxer, but they failed hard at chronic diseases, promoting cures and guidelines that did more harm than good.

Definitely not accusing you of being an anti-vaxxer, but I would love to hear your sources for some of your statements, because they contradict the data that I've seen. Searching now again, everything I'm finding is showing really big improvements in mortality from cardiovascular disease.

Recent US outcomes: https://jamanetwork.com/journals/jamacardiology/fullarticle/...

US & New York State outcomes 1980-2007, Figure 2: https://www.health.ny.gov/diseases/cardiovascular/heart_dise...

I haven't hit the major mortality sources, but these bits match everything else I've heard.

I'd also like to know the basis for asserting that the health care industry doing more harm than good.

>And given the alarmingly climbing rates of chronic diseases, e.g. cardiovascular disease, diabetes, obesity, cancer, I'm also pretty sure that the health care industry has done more harm than good when it comes to chronic diseases.

I mean, can't this be attributed to other things too? (The way people live has changed in the past 20 years...)

I'd rather see some studies to show if/why those things have spiked rather than disavowing attempts to help it.

What else are we supposed to do?

>> In any case, don't trust every popular news article you read about science, particularly if it's written by Kevin Drum and posted on Mother Jones.

Very good advice. Over 95% of the crap posted on Mother Jones should be ignored.