==========
EDIT to reply to BadInformatics' reply below (I'm being rate-limited): I fully agree that a lot of ML code releases could be better about this, and it's even reasonable to expect them to do some of these more basic things like you mention. I don't agree that bit-for-bit reproducibility is a realistic standard that will get us there.
Because I have and I pity anyone who tries and builds a kernel that can run it.
A lot of open sourced CS research is not reproducible.
"the code still runs and gives the same output" is not the same as reproducibility.
I'm not sure if this was meant to be a counter-argument to me, but I completely agree!
> "the code still runs and gives the same output" is not the same as reproducibility.
Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.
It wasn't :)
> Plenty of good science got done before modern devops came to be
This isn't as strong of an argument as you think. This is more-or-less the underlying foundation behind the social sciences, which argues that no social sampling can ever be entirely reproduced since no two people are alike, and even the same person cannot be reliably sampled twice as people change with time.
Has there been "good science" done in the social sciences? Sure. I don't think that you're going to find anybody arguing that the state of the social sciences today is about the same as it was in the Dark Ages.
With that said, one of the reasons why so many laypeople look at the social sciences as a kind of joke is because so many contradictory studies come out of these peer-reviewed journals that their trustworthiness is quite low. One of the reasons why there's so much confusion surrounding what constitutes a healthy diet and how people should best attempt to lose weight is precisely because diet-and-exercise studies are more-or-less impossible to reproduce.
> If you can achieve that, great -- it's certainly a useful property to have for debugging
If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice.
I don't understand why it is so hard to see that you need either a pretty big collaboration where somebody else has isolated the specifications so you don't need to know anything about the problem your code solves really, or becoming a physics graduate student yourself for this line of work.
0: specifically, for results claiming a 95% confidence level (p<0.05), if nineteen replications are attempted, you should encounter a replication failure exactly once. I would accept perhaps four or five out of nineteen (or one out of two or three) under the reasoning that the law of large numbers hasn't kicked in yet, but anything with zero successful replications is not science, it's evidence (in this, against the entire field of study).
Contra your assertion, many people do some sort of regression testing like this but it's isn't terribly useful for verification or validation - but it is good at catching bad patches.
I used to be a scientist. I get it, getting scientists to do this is like pulling teeth, but it's the least you could do to give other people confidence in your results.
Ok, that's a reasonable ask :) But yeah as you implied, good luck getting the average scientist, who in the best case begrudgingly uses version control, to care enough to do this.
Scientific reproducibility does not mean, and has never meant, you rerun the code and the output perfectly matches bit-for-bit every time. If you can achieve that, great -- it's certainly a useful property to have for debugging. But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold.
To paraphrase a comment I saw from another thread on HN: "Plenty of good science got done before modern devops came to be."
[0] https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/18096
[1] https://pytorch.org/docs/stable/notes/randomness.html
==========
EDIT to reply to solatic's replies below (I'm being rate-limited):
The social science arguments are probably fair (or at least I'll leave it to someone more knowledgeable to defend them if they want) -- perhaps I shouldn't have led with the example of "different groups of participants".
> If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice.
This might be a reasonable thing to enforce if everyone in the field were using the same computing platform. Given that they're not (and that telling everyone that all published results have to be done using AWS with this particular machine configuration is not a tenable solution) I don't see how this could ever be a realistic requirement. Or if you don't want to enforce that the results remain identical across different platforms, what's the point of the requirement in the first place? How would it be enforced if nobody else has the exact combination of hardware/software to do so? And then even if someone does, almost inevitably there'll be some detail of the setup that the researcher didn't think to report and results will differ slightly anyway.
Besides, if you're allowing for exemptions, just about every paper in machine learning studying datasets larger than MNIST (where asynchronous prefetching of data is pretty much required to achieve decent speeds) would have a good reason to be exempt. It's possible that there are other fields where this sort of requirement would be both useful and feasible for a large amount of the research in that field, but I don't know what they are.
> Also, reading through the issues you linked points to: https://github.com/NVIDIA/framework-determinism which is a relatively recent attempt by nVidia to support deterministic computation for TensorFlow. Not perfect yet, but the effort is going there.
(From your other comment.) Yes, there exists a $300B company with an ongoing-but-incomplete funded effort of so far >6 months' work (and that's just the part they've done in public) to make one of its own APIs optionally deterministic when it's being used through a single downstream client framework. If this isn't a perfect illustration that it's not realistic to expect exact determinism from software written by individual grad students studying chemistry, I'm not sure what to say.