News for hackers. Stuff that matters.
Re:Psychology’s Replication Crisis Is Real by __s on Saturday December 01, 2018 @12:14PM (#18577671)
So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implementing the code & collecting a dataset. The exact code & dataset aren't suppose to be required for the results: you should be able to replicate the results by substituting your own code & data. When you have a rat maze experiment, it isn't expected that they include a 3d printer blueprint of the maze & genomes of the rats involved

Or, more succinctly, code & data are left as an exercise for the reader

Re:Psychology’s Replication Crisis Is Real by djsumdog on Saturday December 01, 2018 @01:26PM (#18578041)
But if more journals archive artifices (code + data) and there was a bigger push to keep these around, when replication fails, they can at least go back to the exact code and say, "huh .. they implemented this differently. I think this is the problem with their algo" or "oh, we forgot to account for this." .. ideally after you've written your own, looking only at the methodology and without looking at the code for the original.
Re:Psychology’s Replication Crisis Is Real by secabeen on Saturday December 01, 2018 @01:37PM (#18578115)
Having the journals archive artifices seems like a good idea, but the librarians hate it. The last thing they want to do is to give the for-profit journals another thing to monetize against the academy. What makes more sense is to have the universities and the library systems archive that stuff.
Re:Psychology’s Replication Crisis Is Real by lumost on Saturday December 01, 2018 @02:07PM (#18578290)
why not use a public artifact repository? e.g. github, bitbucket, or a new academic collaboration.
Re:Psychology’s Replication Crisis Is Real by porphyrogene on Saturday December 01, 2018 @12:21PM (#18577696)
It is common to publish the layouts and dimensions of mazes in such studies. The mazes[1] themselves often reflect the nature of the study (e.g. a simple fork to test desicion-making or a complex maze with dead ends to test navigation and/or memory.)

1. http://www.ratbehavior.org/RatsAndMazes.htm

Re:Psychology’s Replication Crisis Is Real by androidgirl on Saturday December 01, 2018 @02:03PM (#18578267)
I see. That does make sense, like making a MIT version of GPL reference code.

Having more researchers avaliable to audit code seems like it would help prevent flaws from slipping through, too, to prevent false conclusions.

Thank you for explaining a bit more.