News for hackers. Stuff that matters.
Re:Psychology’s Replication Crisis Is Real by lumost on Saturday December 01, 2018 @02:05PM (#18578281)
how many papers important results are simply bugs?

Numerical code is already bug prone due to subtle and hard to test errors, research code that's not code reviewed or necessarily even tested for correctness can easily generate important results erroneously.

It's also counter-productive not to publish the underlying source code for these papers, as it adds a barrier to other researchers applying the algorithm in new situations. I'd be interested in seeing if those 6% of papers which include the code get more citations than the population of papers which do not include code.

Re:Psychology’s Replication Crisis Is Real by MAXPOOL on Saturday December 01, 2018 @02:18PM (#18578355)
> important results are simply bugs

Probably none. If the paper is important and collects citations, the algorithm is in use. Computer science != working code.

Code is required when you produce something where the scientific importance is less clear. There is need to provide more evidence. Many papers are just "Hey I made some some tweaks and it works in this particular case." Those papers should have working code.

Re:Psychology’s Replication Crisis Is Real by lumost on Saturday December 01, 2018 @03:35PM (#18578767)
Most papers leverage a results table which compares the newly proposed approach with existing approaches. This section is baselines the results of the new approach with prior work and helps determine whether a new result is actually important or yet another way to achieve the same results as previous work. e.g. the tables on page 7 of this paper https://www.semanticscholar.org/paper/Automatic-Acquisition-...

These tables are generated using real implementations that may or may not be correct, and should be subject to review when the paper is published.