News for hackers. Stuff that matters.
Re:Just 11% of 53 cancer research papers were reproducible by JDDunn9 on Thursday May 09, 2013 @05:53PM (#5683217)
Perhaps part of this is statistical errors and selection bias. Most studies are done with a 95% confidence interval, but only studies that reject the null hypothesis are published. If we assume only 20% of all experiments reject the null hypothesis, then we should expect ~25% of those to be fasle positives. Journals might also be biased to include more sensationalistic results, which are more likely to be statistical errors.

My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%.

Re:Just 11% of 53 cancer research papers were reproducible by gwern on Thursday May 09, 2013 @11:23PM (#5684402)
> My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%.

No; you need to bring in power, not just assume an alpha of 0.05 and a base-rate of real results of 20%. (Consider the recent neuroscience paper estimating the experiments average a power of like 0.3...)