News for hackers. Stuff that matters.
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by Osmium on Wednesday October 16, 2013 @04:05PM (#6561825)
Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by bmahmood on Wednesday October 16, 2013 @04:35PM (#6562018)
Hi this is Bilal from Science Exchange, and I help with the Reproducibility Initiative.

You are right that the per study costs are cheaper than normal as the original protocol is already developed, and there is less trial and error involved. In certain cases, we may not need to validate every aspect of a study as well, only those specific experiments that warrant replication. And depending on the study, those experiments may vary in cost (i.e. a mouse model component would be expensive, but RNA extraction and western blot testing relatively inexpensive).

Additionally, we will leverage expert labs on Science Exchange which perform experiments regularly on a contract basis, allowing for bulk savings on reagents and supplies.

Re:Reproducibility Initiative gets $1.3M grant to validate 5... by loceng on Wednesday October 16, 2013 @05:26PM (#6562250)
Bulk Science. I like it.
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by mathattack on Wednesday October 16, 2013 @04:19PM (#6561909)
If $1.3M can cover 50, they should fund reproducing 500, or even 5000.

I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.

Re:Reproducibility Initiative gets $1.3M grant to validate 5... by Osmium on Wednesday October 16, 2013 @05:57PM (#6562368)
> I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties.

Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result when studied (usually due to minute trace impurities, or marginally different temperatures or flow rates during growth, or a host of other things–all of which need to be analysed and vetted–and this happens even if they're using growth reactors made by the same manufacturer!). It makes replication very difficult and often results in slow progress. And it's expensive too. So it's not just medicine that has that problem.

Edit: I should add that obviously in the end it works out because these materials eventually make it into industry and into millions of devices. So it's ultimately reproducible. But that gap between first discovering a process or a mechanism or just something interesting and making it reproducible consistently enough and at a large enough scale for industry is just huge.

Re:Reproducibility Initiative gets $1.3M grant to validate 5... by mathattack on Wednesday October 16, 2013 @06:35PM (#6562515)
Indeed - thank you for clarifying. This does sound expensive, though all the more necessary.
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by Fomite on Wednesday October 16, 2013 @06:24PM (#6562474)
Beyond it being expensive to replicate once, for observational studies and clinical trials, you need many repetitions because of statistical variability. That gets expensive in a hurry.
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by bmahmood on Wednesday October 16, 2013 @07:12PM (#6562671)
You are right that clinical trial studies would be much more time intensive and costly. The studies we will be replicating through the Reproducibility Initiative are more preclinical cancer studies, rather than later-stage clinical trials.

We felt the preclinical stage was more important to validate, as much of the research is based on academic studies with over 60% failure rate for replication (see http://www.nature.com/nature/journal/v483/n7391/full/483531a...)

By clinical trials, a lot of the pharma companies have already done initial validation, so extra validation is redundant. We hope a reproducibility system to validate preclinical research can help patient groups, foundations, and industry better identify reproducible oncology targets, and hence the focus on preclinical studies (which in turn are less expensive).

Re:Reproducibility Initiative gets $1.3M grant to validate 5... by Fomite on Wednesday October 16, 2013 @08:23PM (#6562936)
Agreed - and this is a hugely important effort. This is more me musing about replication vs. reproducibility (on the computational/statistical end) vs. repeatability.
Re:Reproducibility Initiative gets $1.3M grant to validate 5... by 001sky on Wednesday October 16, 2013 @05:12PM (#6562189)
I came here to say the same thing. For important research, it would be interesting to see more cost/benefit analysis being put to bear in this way. The cost of replication should be so much lower than original work, in many cases. And it would be interesting to evaluate the "science" of academic "science", especially given the controversy around peer review quality controls in general on academic publishing.