Re:A Worrying Analysis of Recent Neural Recommendation Appro...
by vidarh on Tuesday July 23, 2019 @03:19AM
(#20505057)
I did my thesis on OCR, and even simple binary thresholding algorithms (to decide a cutoff point for "background" vs "text") that might boil down to a dozen lines of code if that in some cases, would often be presented as a formula (great; at least something) or imprecise prose, but where you then when translating it into code would find that their results depended on setting certain parameters to certain values that they'd not specified anywhere.
In some cases I basically had to run a test harness and brute force the parameters they'd used, because the algorithm was basically useless without picking parameters in the right range.
It was hard to accept then (~2008-ish), but now we're at a point where frankly at least for the more basic CS papers we ought to be able to expect source and some runnable format (e.g. a VM image) with test data.
I handed in a DVD with gigabytes of test data, full source, built binaries, and a test harness for the work I did for my thesis. All it took was to follow practices that I needed to follow to satisfy myself that my results were valid anyway.
I realize that for some work that relies on much larger and/or proprietary datasets making everything available would be hard, but a small test case illustrating the code run with the right parameters etc. ought to be a minimum unless there are substantial barriers, in which case the paper ought to signpost clearly what is missing and why.
In some cases I basically had to run a test harness and brute force the parameters they'd used, because the algorithm was basically useless without picking parameters in the right range.
It was hard to accept then (~2008-ish), but now we're at a point where frankly at least for the more basic CS papers we ought to be able to expect source and some runnable format (e.g. a VM image) with test data.
I handed in a DVD with gigabytes of test data, full source, built binaries, and a test harness for the work I did for my thesis. All it took was to follow practices that I needed to follow to satisfy myself that my results were valid anyway.
I realize that for some work that relies on much larger and/or proprietary datasets making everything available would be hard, but a small test case illustrating the code run with the right parameters etc. ought to be a minimum unless there are substantial barriers, in which case the paper ought to signpost clearly what is missing and why.