Re:Troubling trends in machine learning scholarship (2018)
by godelski on Sunday December 01, 2019 @09:51PM
(#21679481)
> You know, the whole publication + review + reproduction thing really helped science become a more solid process,
I think the model is still good. The examples you stated really do fail at the reproduction part. I think one of the difficulties here is that reproduction is difficult and costly, and there is no one willing to pay for it. A common metric for a paper is "can a smart PhD student replicate the experiment from the contents of the paper?" How many advisors ask their students to replicate a paper? How many students can?
I think the model is still good. The examples you stated really do fail at the reproduction part. I think one of the difficulties here is that reproduction is difficult and costly, and there is no one willing to pay for it. A common metric for a paper is "can a smart PhD student replicate the experiment from the contents of the paper?" How many advisors ask their students to replicate a paper? How many students can?