https://en.wikipedia.org/wiki/Provenance#Science
There is a W3C group that is all about setting things up so that the use of big data repositories and complex scientific code can be used so that others can replicate the findings:
https://www.w3.org/2005/Incubator/prov/XGR-prov-20101214/
It's actually fairly terrible that in Environmental Science and quite a bit of Economics and other areas the code and data is meant to be around but really isn't.
There is a decent short introductory page here:
http://www.ands.org.au/partners-and-communities/ands-communi...
The little experience I have is in computer graphics publications and (less) distributed systems publications. Graphics papers very often have data sets and code that go along with the paper. These are nice, and it would be awesome to have a common and well-known way of publishing all that together with the papers, but they are just examples. Mike Hoye's comparison to the loss of greek fire doesn't seem applicable to the idea of reproducibility by publishing code and dependencies. The meat is in the paper!
Same with my experience in distributed systems papers. I can implement Paxos at any time using only the Paxos PDFs I've saved. Companion code is nice, and again I'd be fully supportive of a system which gave permanence to not ony the papers but companion artifacts... but the paper is the important part, isn't it?
They may be not reproducible if going from pseudocode to working implementation and from there to the pretty graphs published by author is nontrivial. From parent post:
The Sprite-LFS affair I mention in that tweet is an infamous case of a famous, influential systems paper by a famous, influential academic (Ousterhout) that badly failed reproduction when a different famous, influential academic (Seltzer) tried to reproduce its results by writing a new log-structured filesystem. Nobody really knows what went wrong[...]
> I can implement Paxos at any time using only the Paxos PDFs I've saved. Companion code is nice, and again I'd be fully supportive of a system which gave permanence to not ony the papers but companion artifacts... but the paper is the important part, isn't it?
Paxos is closer to the "theory" than the "systems" end of parent's spectrum. Real problem is with things that take man-months to implement and validate, not things that take man-months to invent.
Hackademia has a couple of synergistic efforts going on, directed mostly at securing the free-software distribution infrastructure and somewhat at simplifying system administration: on one hand you have reproducible builds being done by Tor, Mozilla, and Debian, and on the other hand you have Nix, Guix, and the like pushing purely-functional content-addressable package management.
The upshot of all of this is that, in theory, in the near future, it will be practical to publish machine-executable instructions for reproducing the bit-identical executables and other output products you used in your research, at a cost of only a few gigabytes, so they can be exactly reproduced at any time in the future, and then modified to do new experiments; and it will be straightforward to make archival copies of this entire set of dependencies so that it won't be lost because some random guy died or decided to pull his modules from npm.
Mike Hoye famously complained about how far we are from this in a Tweetstorm the other day: https://twitter.com/mhoye/status/725010388547330048
The Sprite-LFS affair I mention in that tweet is an infamous case of a famous, influential systems paper by a famous, influential academic (Ousterhout) that badly failed reproduction when a different famous, influential academic (Seltzer) tried to reproduce its results by writing a new log-structured filesystem. Nobody really knows what went wrong, but our best guess at this point is that the Sprite operating system had unknown bugs in it that gave a performance advantage to log-structured filesystems.
(ChuckMcM may want to weigh in, since his company at the time (NetApp) was getting record-setting performance with WAFL, which is very similar to a LFS in some ways, but not others.)
Jan Vitek, Jean Yang, Sam Tobin-Hochstadt, and Shriram Krishnamurthi are four prominent researchers who are advocating this change; they and others have created a new institution called "Artifact Evaluation Committees", which review the artifacts submitted by authors to see if they are able to reproduce the results the authors report. They've written an overview of it here, and they seem to already be getting significant results: http://jxyzabc.blogspot.com.ar/2016/05/myth-cs-researchers-d...
Hopefully over time this will create a bubble of real, reproducible science inside of the systems part of Computer "Science" research.
[Earlier versions of this comment did not mention Krishnamurthi or Vitek and the synergistic work in hackademia, which was a combination of cluelessness and carelessness on my part. I regret the omissions.]