That brings up a separate issue that I didn't comment on above: the expectation that the code runs in a completely different development/execution environment (e.g. the one the reviewer is using vs. the one that the researcher used). That means making it run regardless of the OS (Windows/OSX/Linux/...) and hardware (CPU/GPU/TPU, and even within those, which one) the reviewer is using. This would be an extremely difficult if not impossible thing for even a professional software engineer to achieve. It could easily be a full time job. There are daily issues on even the most well-funded projects in machine learning by huge companies (ex: TF, PyTorch) that the latest update doesn't work on GPU X or CUDA version Y or OS Z. It's not a realistic expectation for a researcher even in computer science, let alone researchers in other fields, most of whom are already at the top of the game programming-wise if they would even think to reach for a "script" to automate repetitive data entry tasks etc.

==========

EDIT to reply to BadInformatics' reply below (I'm being rate-limited): I fully agree that a lot of ML code releases could be better about this, and it's even reasonable to expect them to do some of these more basic things like you mention. I don't agree that bit-for-bit reproducibility is a realistic standard that will get us there.

I don't think that removes the need to provide enough detail to replicate the original environment though. We write one-off scripts with no expectation that they will see outside usage, whereas research publications are meant for just that! The bar isn't terribly high either: for ML, a requirements.txt + OS version + CUDA version would go a long way, no need to learn docker just for this.

Have you tried running a specific Cuda version from 10 years ago?

Because I have and I pity anyone who tries and builds a kernel that can run it.

It does seem like a valid response to OP's objection to the imperial college's COVID model, though. Doesn't it?

Reviewing the original comment, I think so (that the original comment is overcritical). For purpose of reproducibility, it's enough that you can validate that you can run the model with different random data and see that their results aren't due to pathological choices of initial conditions. If the race conditions and non-determinism just transform the random data into another set of valid random data, that doesn't compromise reproducibility.