News for hackers. Stuff that matters.
Re:Challenge to scientists: does your ten-year-old code stil... by PeterisP on Monday August 24, 2020 @01:32PM (#24263056)
Your requirements seem to push 'Science ready' far into what I'd consider "worthless waste of time", coming from the perspective of code that's used for data analysis for a particular paper.

The key aspect of that code is that it's going to be run once or twice, ever, and it's only ever going to be run on a particular known set of input data. It's a tool (though complex) that we used (once) to get from A to B. It does not need to get refactored, because the expectation is that it's only ever going to be used as-is (as it was used once, and will be used only for reproducing results), it's not intended to be built upon or maintained. It's not the basis of the research, it's not the point of research, it's not a deliverable in that research, it's just a scaffold that was temporarily neccessary to do some task - one which might have been done manually earlier through great effort, but that's automated now. It's expected that the vast majority of the readers of that paper won't ever need to touch that code, they care only about the results and a few key aspects of the methodology, which are (or should be) all mentioned in the paper.

It should be reproducible to ensure that we (or someone else) can obtain the same B from A in future, but that's it, it does not need to be robust to input that's not in the input datafile - noone in the world has another set of real data that could/should be processed with that code. If after a few years we or someone else will obtain another dataset, then (after those few years, if that dataset happens) there would be a need to ensure that it works on that dataset before writing a paper about that dataset, but it's overwhelmingly likely that you'd want to modify that code anyway both because that new dataset would not be 'compatible' (because the code will be tightly coupled to all the assumptions in the methodology you used to get that data, and because it's likely to be richer in ways you can't predict right now) and you'd want to extend the analysis in some way.

It should have a 'toy example' - what you call 'a schema-correct facsimile of input data' that's used for testing and validation before you run it on the actual dataset, and it should have test scenarios and/or unit tests that are preferably manually verifiable for correctness.

But the key thing here is that no matter what you do, that's still in most cases going to be "write once, run once, read never" code, as long as we're talking about the auxiliary code that supports some experimental conclusions, not the "here's a slightly better method for doing the same thing" CS papers. We are striving for reproducible code, but actual reproductions are quite rare, the incentives are just not there. We publish the code as a matter of principle, knowing all well that most likely noone will download and read it. The community needs the possibility for reproduction for the cases where the results are suspect (which is the main scenario where someone is likely to attempt reproducing that code), it's there to ensure that if we later suspect that the code is flawed in a way where the flaws affect the conclusions then we can go back to the code and review it - which is plausible, but not that likely. Also, if someone does not trust our code, they can (and possibly should) simply ignore it and perform a 'from scratch' analysis of the data based what's said in the paper. With a reimplementation, some nuances in the results might be slightly different, but all the conclusions in the paper should still be valid, if the paper is actually meaningful - if a reimplementation breaks the conclusions, that would be a successful, valuable non-reproduction of the results.

This is a big change from industry practice where you have mantras like "a line of code is written once but read ten times", in a scientific environment that ratio is the other way around, so the tradeoffs are different - it's not worth investing refactoring time to improve readability, if it's expected that most likely noone will ever read that code; it makes sense to spend that effort only if and when you need it.

Re:Challenge to scientists: does your ten-year-old code stil... by lemmsjid on Monday August 24, 2020 @02:27PM (#24263748)
Yep! I don't disagree with anything you're saying when I think from a particular context. It's really hard to generalize about the needs of 'science code', and my stab at doing so was certain to be off the mark for a lot of cases.
Re:Challenge to scientists: does your ten-year-old code stil... by PeterisP on Monday August 24, 2020 @02:43PM (#24263931)
Yes, there are huge differences between the needs of various fields. For example, some fields have a lot of papers where the authors are presenting a superior method for doing something, and if code is a key part of that new "method and apparatus", then it's a key deliverable of that paper and its accessibility and (re-)usability is very important; and if a core claim of their paper is that "we coded A and B, and experimentally demonstrated that A is better than B" then any flaws in that code may invalidate the whole experiment.

But I seem to get the vibe that this original Nature article is mostly about the auxiliary data analysis code for "non-simulated" experiments, while Hacker News seems biased towards fields like computer science, machine learning, etc.