Maybe not, but these studies get swept up into meta-studies, books, think tanks and special interest groups write policy papers based on that secondary layer, which then become position papers that inform policy makers about Big Decisions. There's a lot at stake when only scientists are aware of the fallacy.

The reason why meta studies get done is precisely because individual studies aren't reliable.

> Our focus shouldn't be on exact replication, and should be on how generalizable such results are.

This is how I normally look at things. If I can't easily replicate an experiment, then it's very likely to be wrong.

Sadly, it's pretty rare (and exciting) when you can easily do something based on the methods in someone's paper

Well that shouldn't necessarily be the standard. There are lots of people I know that can't do a miniprep or run a gel. It doesn't mean that those things don't work or aren't replicable.

> No scientist treats published studies as gospel.

Unfortunately that is a problem. Imagine yourself trying to create a simulation of a biological system, which has to rely on experiments. You may come up with a plan, but every little line on the plan will be either very doubtful or outright false. The problem is that many of these doubts could be dispelled if the experiment was a lot more stringent (much larger sample size, much more controlled conditions etc). That would cost a hell lot more, but it would give you one answer you can rely on.

I don't think making experiments more stringent is the answer here (btw, I've spent a lot of time trying to build simulations of biological systems). Usually, we aren't doing the right experiment in the first place; this is hard to figure out ahead of time. Again, read the ACSB that I linked to below... it's probably the most nuanced and interesting discussion on the subject

Yes, thanks that was a good read. It seems we need a "Map" of biological sciences, in which every study could be a 'pin' in a particular location , signifying that 'this study studies this very particular problem here'. Maybe that would help figure out where are the biggest gaps. Unfortunately, most studies broaden the impact of their results too much to the point that reading the paper abstract can be misleading. Maybe people should just publish their results, but not be allowed to make any claims about them; let others and the community make those claims.

Also, you should read the Peter Walter piece that I think is the best discussion of the topic: https://www.ascb.org/newsletter/2016-marchapril-newsletter/o...

> No scientist treats published studies as gospel.

Why then do so many scientists in so many different fields insist "the science is settled"?

Which scientists? About what issues?