News for hackers. Stuff that matters.
Re:A new replication crisis: Research that is less likely to... by seoaeu on Saturday May 22, 2021 @03:25PM (#27249548)
Providing source code is a good thing, but a lot of people confuse re-running experiments with replicating them. If you take the authors' source code and re-run it, then any bugs are going to invalidate your results too. The only way to actually have confidence in the paper's results are to rewrite the software from scratch.

In fact, I'd actually go further and question what kinds of errors could possibly be caught be running the same software that the authors did? Any accidental bugs will remain, and any malicious tampering with the experiment data is exceedingly unlikely to be caught even with a careful audit of the code.

There are no comments on this comment yet.