From the horse's mouth: only 6 out of 53 "landmark" oncology studies were significantly reproducible [1]. Bayer did a similar study, found that they couldn't reproduce the results of about 2/3s of the papers they tried to use for drug discovery R&D [2].

This is a near universal consequence of academia's incentives and extends into many of the sciences (although biology is most susceptible due to its complexity).

[1] http://www.nature.com/nature/journal/v483/n7391/full/483531a...

[2] http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.htm...

>This is a near universal consequence of academia's incentives and extends into many of the sciences (although biology is most susceptible due to its complexity).

It is not simply a matter of incentives (although they play a part). Another issue is pure publication bias. If you get 'landmark' results, a paper is much more likely to publish then if you got the results everyone would expect. In many ways this is one of the necessary functions of the journals, as it allows people to look at the new stuff without sifting through many articles of expected stuff, but it does create a bias.

Add to that the incentives of needing to get published, and constrained resources, and a willingness to publish what could be sub-par methodologys (of course they are par), and it is clearly not the best system.

From a Bayesian perspective, papers that are demonstrating phenomena that are both (1) novel and (2) significant are the ones that are most likely to be false positives. If you have a journal built around publishing (1) and (2), you're bound to have an excess of studies that can't be reproduced.

That's a very insightful way of looking at it. If Nature wanted to get out of this unfortunate position, it would have to counteract this with stricter requirements on the articles that it publishes. It could require that papers be replicated before publication; it could require the release of any relevant code and data; it could be really diligent about publishing failure-to-reproduce papers. This would be a hassle, but would probably make Nature more prestigious, so I'm sure they could get a lot of researchers to go along with whatever requirements they want.

Nature's and Science's shenanigans are not publishing papers that turn out to be wrong, it's that (1) when contrary data comes up they refuse to publish it, as in this case, and (2) really questionable editorial policies, where articles get peer-reviewed by the wrong audience, and on-the-face obviously-wrong to anybody in the field papers still get published.

Every journal publishes novel and statistically significant findings, and probably encounter similar rates of type 1 error as Nature and Science. It's just that Nature and Science are more susceptible to human politicking and don't seem to face up to it.

> This would be a hassle, but would probably make Nature more prestigious...

Prestige is a relative thing, and there's nothing more prestigious than Nature (or Science). Your suggestions are good, but I think they would be more likely to be adopted first by a journal that isn't Nature or Science, that possibly wanted to compete with them.

You're probably right. Nature and Science are Nature and Science; they don't need to improve on the status quo because they're already sitting on top.

I hope some lesser journal becomes greater, and soon.

This is true. I join AAAS in order to get access to Science and some of the related journals and have become quite disillusioned at their practice of either hiding, or outright omitting, papers or studies that contrast something that appeared 'first' in their pages. I was always taught that a scientist should never fear being wrong, they should only fear being not right. The discourse and disagreements, the questions and reproduction things. make things more real.