Re:Challenge to scientists: does your ten-year-old code stil...
by kkylin on Tuesday August 25, 2020 @05:21PM
(#24276278)
I agree with the case analysis, but disagree with the implication that the code needs to be runnable (which is seems to be the point of the discussion at hand). In many cases having source code, even if it no longer runs, should be sufficient.
Re:Challenge to scientists: does your ten-year-old code stil...
by kkylin on Tuesday August 25, 2020 @05:33PM
(#24276413)
I complete agree with your case analysis, but disagree with the conclusion that the code needs to be runnable for it to be useful -- I thought this was the point of the discussion at hand? In most situations, having source code, even if it no longer runs, would be sufficient to conduct the analysis you describe.
I'm all for more transparency, and this includes making codes and data public as much as is reasonable. But the real test is if someone can independently replicate the result, and how to incentivize replication studies (in both computational and experimental science) is also important, and in my view should not be divorced from discussions of reproducibility.
Edit: rewrote to clarify my position.
On the contrary; in that case there are four possibilities:
a: your algorithm doesn't work at all, and your observations are a artifact of convenient inputs or inept measurements.
b: your algorithm works, but the description in the paper is wrong or incomplete
c: your algorithm works as described, but the replicater implemented it incorrectly
d: other
Having the original implementation code is necessary to distinguish between cases a and b versus case c, and if the former, the code for the test harness is likely to help distinguish a versus b. (Case d is of course still a problem, but that doesn't mean it's reasonable to just give up.)