Re:OpenAI Baselines
by Houshalter on Thursday May 25, 2017 @05:35PM
(#14420308)
Yeah the SARM paper was an extreme example, but that's why it got caught so quickly. How many papers have less extreme but still serious flaws, and don't get caught?
As an extreme example it actually brought me a bit of hope. No source code was released but the "peer review" via arXiv, Twitter, and other various channels ended up bringing the story to a close.
I'd like to imagine it's how effective peer review could be if given sufficient motivation ;)
As you note, though, most papers don't get anywhere in the same magnitude of focus, and others which do may still be entirely unreproducible anyway :(
Your broader point is spot on however. My general hope is that people are wary when their results are strong^, ensuring that you don't get a good result via "cheating", so the majority of bugs are likely to harm performance. If a result is also not reproducible (i.e. "cheating" bug) it won't be used and built on - but if a result is bugged but reproducible (i.e. bug where performance was lower than it should be) then we can still move the field forward even in spite of these issues.
^ When I achieved state of the art for a task - especially given it was a huge jump in accuracy for a relatively small model compared to the previous state of the art - I spent many days sitting there double checking I hadn't accidentally cheated ;)