Plan to replicate 50 high-impact cancer papers shrinks to just 18

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by wyattpeak on Tuesday August 07, 2018 @04:28AM (#17704729)

I don't disagree with any of that, although I'd stick a big citation needed on the implicit suggestion that there's a large group of scientists who aren't making a good-faith effort to ensure that their successors will have the information they need to reproduce (that is, after all, what a paper is).

My issue is the flippant and silly claim that "[i]t's possible to imagine a version of academia where results that can be attributed to noise don't get published".

[ Parent | on HN ]

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by saalweachter on Tuesday August 07, 2018 @10:56AM (#17707192)

I think this is actually something that can be experimentally examined.

Take a sampling of a large number of papers, give them some sort of rating based on whether they provide enough information to reproduce, how clear their experimental and analytical methodology was, whether their primary data and scripts are available, etc, and then look at that rating versus their citations.

Hopefully, better papers get more attention and more citations.

(And yeah, "peer review" as it is done before a paper is published is not supposed to establish a paper as correct, it is supposed to validate it as interesting. Poor peer review ultimately makes a journal uninteresting, which means it might as well not exist.)

[ Parent | on HN ]

Re:Plan to replicate 50 high-impact cancer papers shrinks to... by wyattpeak on Tuesday August 07, 2018 @11:38AM (#17707585)

That sounds like a very interesting idea. At the least, it would be interesting see the major classes of reproducibility problems. And there may well be a lot of low-hanging fruit, as the comments on this page suggest about data corpuses in computational fields.

[ Parent | on HN ]