I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code.

As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result.

By the way, yes I tested my ten year old code and it does still work. What I'm saying is that scientific code doesn't need to handle every special case or be easily usable by non-experts. In fact the time spent making it that way is time that a scientist spends doing software engineering instead of science, which isn't very efficient.

Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same instance of the program running on a central machine.

Does scientific-grade code need to publish telemetry? Eh, usually no. Set up alerting so that on-call engineers can be paged when (not if) it falls over? Nope.

Does scientific-grade code need to handle the authorization and authentication of users? Nope.

Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. The reproducibility of results is core to the scientific method. Yes, that includes Monte Carlo code, when there is no such thing as truly random number generation on contemporary computers, only pseudorandom number generation, and what matters for cryptographic purposes is that the seed numbers for the pseudorandom generation are sufficiently hidden / unknown. For scientific purposes, the seed numbers should be published on purpose, so that a) the exact results you found, sufficiently random as they are for the purpose of your experiment, can still be independently verified by a peer reviewer, b) a peer reviewer can intentionally decide to pick a different seed value, which will lead to different results but should still lead to the same conclusion if your decision to reject / refuse to reject the null hypothesis was correct.

I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication.

Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction between experiment and human brain meats that's missing when code is simply reused (whether we consider it apparatus or procedure).

Once we have multiple implementations, if there is a meaningful difference between them, at that point replayability is of tremendous value in identifying why they differ.

But it is not reproducibility, as we want that term to be used in science.

The software was "reproducible" in that the same starting conditions produced the same output, but that didn't mean the _science_ was reproducible, as every study used the same software.

I repeatedly brought it up, but I wasn't advanced enough in my studies to be able to do anything about it. By the time I felt comfortable with that, I was on my way out of the field and into an non-academic career.

I have kept up with the field to a certain extent, and there is now a project in progress to create a fully independent replacement for that original code that should help shed some light (in progress for a few years now, and still going strong).

This is the difference between reproducibility and replicability [1]. Reproducibility is the ability to run the same software on the same input data to get the same output; replication would be analyzing the same input data (or new, replicated data following the original collection protocol) with new software and getting the same result.

I've experienced the same lack of interest with established researchers in my field, but I can at least ensure that all my studies are both reproducible and replicable by sharing my code and data.

[1] Plesser HE. Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Front Neuroinform. 2018;11:76.

Something like this is the norm in some more mathematical fields, where only the polished final version is published, as if done by pure thought. To build that, first you have to reproduce it, invariably by building your own code -- perhaps equally awful, but independent.

The fact that we can have that option is an amazing opportunity that a confluence of attributes of software (specificity, replayability, easy of copying) afford us. Where we are not exploiting this like we could be, it is a failure of our institutions! But it is different-in-kind from traditional reproducibility.

More concretely, it is 100% fair (and I might argue necessary) to demand more of our institutions and work to improve their failures. I'm sure many researchers have encountered publications of the form "we applied <proprietary model (TM)> (not explained) to <proprietary data> (partially explained) after <two sentence description of preprocessing> and obtained SOTA results!" in a reputable venue. Sure, this might be even less reproducible 200 years ago than now, but the authors would also be less likely to be competing with you for limited funding! Debating about the traditional definition of reproducibility has its place, but we should also be doing as much as possible to give reviewers and replicators a leg up. This is often flies in the face of many incentives the research community faces, but shifting blame to institutions by default (not saying you're doing this, but I've seen many who do) is taking the easy road out and does little to help the imbalanced ratio of discussion:progress.

But repeating the simulation a large number of times, with different random seeds, should produce statistically similar output if the code is rigorous. So even if each simulation is not reproducible, as long as the statistical distribution of outputs is reproducible, that should be sufficient.

There are some kinds of code (a script that controls a sensor or an actuator) where I think that doesn't match up well at all. There are plenty of kinds of code where they are, in fact, simply crunching numbers produced earlier. For the latter, I'm honestly not sure the best way to treat it, except to say that we should be sure that enough information is included in some form that replication should be possible, and that we keep in mind the idea that replication should involve human interaction.

Edit: more careful phrasing.

On the contrary; in that case there are four possibilities:

a: your algorithm doesn't work at all, and your observations are a artifact of convenient inputs or inept measurements.

b: your algorithm works, but the description in the paper is wrong or incomplete

c: your algorithm works as described, but the replicater implemented it incorrectly

d: other

Having the original implementation code is necessary to distinguish between cases a and b versus case c, and if the former, the code for the test harness is likely to help distinguish a versus b. (Case d is of course still a problem, but that doesn't mean it's reasonable to just give up.)

I'm all for more transparency, and this includes making codes and data public as much as is reasonable. But the real test is if someone can independently replicate the result, and how to incentivize replication studies (in both computational and experimental science) is also important, and in my view should not be divorced from discussions of reproducibility.

Edit: rewrote to clarify my position.

We've certainly done a lot, scientifically speaking (in terms of post-validated studies), without that level of reproducibility.

well that's just not true. there's no shortage of noise we can sample to get true random numbers. we just often stretch the random numbers for performance purposes.

This reminds me of being in gradschool and the comp-sci people complaining that we don't get bit-wise equal floats when we solve DEs.

Having to re-implement a library from scratch for a project is much more valuable than running the same code in two places. The same way that getting the same results from two different machines is a lot more significant than getting the same result from two cloned machines.

In short: code does not need to be reproducible because scientists know how to average.

This is definitely not correct. The experiment as a whole needs to be reproducible independently. This is very different, and more robust, from requiring that a particular portion of a previous version of the experiment to be reproducible in isolation.

Reproducibility in the scientific sense is different from running the same program with the same input, and getting exactly the same result. Repreducibility means that if you repeat the measurements in another environment, getting somewhat different data, and apply the same theory and methods, you get to the same conclusion.

The property of a computer program that when you run it again with the same input, you get the same output, is nice and very helpful for debugging. But the fact that you can run the same program does not mean that it is bug-free, as much as the fact that you can copy a paper with a mathematical proof does not mean that the proof is correct.

Also, multi-threaded and parallel code is inherently non-deterministic.

> when there is no such thing as truly random number generation on contemporary computers, only pseudorandom number generation,

That is wrong. Linux for example uses latency measurements from drivers such as HDD drive seek latencies or keyboards to generate entropy. While it might not the best thing to rely on for purposes of cryptography, it is surely not deterministic. If it would matter, you could download real-time astronomical noise measurements and use them to seed your Mersenne Twister generator.

Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome.

That an identical experiment leads to an identical outcome fails to validate the hypothesis, because the casual factors may have been misidentified.

Precise reproducibility still matters if generalized reproduction fails, however, because the differences in the experimental implementation may lead to new and more accurate hypothesis about causality.

The problem here is a step before this: The results of these "identical" experiments are so wildly different there is nothing valid to propose, let alone for a recreation to compare against.

So what happens if/when an identical experiment fails to lead to an identical outcome?

Scientific reproducibility does not mean, and has never meant, you rerun the code and the output perfectly matches bit-for-bit every time. If you can achieve that, great -- it's certainly a useful property to have for debugging. But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold.

To paraphrase a comment I saw from another thread on HN: "Plenty of good science got done before modern devops came to be."

[0] https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/18096

[1] https://pytorch.org/docs/stable/notes/randomness.html

==========

EDIT to reply to solatic's replies below (I'm being rate-limited):

The social science arguments are probably fair (or at least I'll leave it to someone more knowledgeable to defend them if they want) -- perhaps I shouldn't have led with the example of "different groups of participants".

> If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice.

This might be a reasonable thing to enforce if everyone in the field were using the same computing platform. Given that they're not (and that telling everyone that all published results have to be done using AWS with this particular machine configuration is not a tenable solution) I don't see how this could ever be a realistic requirement. Or if you don't want to enforce that the results remain identical across different platforms, what's the point of the requirement in the first place? How would it be enforced if nobody else has the exact combination of hardware/software to do so? And then even if someone does, almost inevitably there'll be some detail of the setup that the researcher didn't think to report and results will differ slightly anyway.

Besides, if you're allowing for exemptions, just about every paper in machine learning studying datasets larger than MNIST (where asynchronous prefetching of data is pretty much required to achieve decent speeds) would have a good reason to be exempt. It's possible that there are other fields where this sort of requirement would be both useful and feasible for a large amount of the research in that field, but I don't know what they are.

> Also, reading through the issues you linked points to: https://github.com/NVIDIA/framework-determinism which is a relatively recent attempt by nVidia to support deterministic computation for TensorFlow. Not perfect yet, but the effort is going there.

(From your other comment.) Yes, there exists a $300B company with an ongoing-but-incomplete funded effort of so far >6 months' work (and that's just the part they've done in public) to make one of its own APIs optionally deterministic when it's being used through a single downstream client framework. If this isn't a perfect illustration that it's not realistic to expect exact determinism from software written by individual grad students studying chemistry, I'm not sure what to say.

==========

EDIT to reply to BadInformatics' reply below (I'm being rate-limited): I fully agree that a lot of ML code releases could be better about this, and it's even reasonable to expect them to do some of these more basic things like you mention. I don't agree that bit-for-bit reproducibility is a realistic standard that will get us there.

Because I have and I pity anyone who tries and builds a kernel that can run it.

A lot of open sourced CS research is not reproducible.

"the code still runs and gives the same output" is not the same as reproducibility.

I'm not sure if this was meant to be a counter-argument to me, but I completely agree!

> "the code still runs and gives the same output" is not the same as reproducibility.

Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.

It wasn't :)

> Plenty of good science got done before modern devops came to be

This isn't as strong of an argument as you think. This is more-or-less the underlying foundation behind the social sciences, which argues that no social sampling can ever be entirely reproduced since no two people are alike, and even the same person cannot be reliably sampled twice as people change with time.

Has there been "good science" done in the social sciences? Sure. I don't think that you're going to find anybody arguing that the state of the social sciences today is about the same as it was in the Dark Ages.

With that said, one of the reasons why so many laypeople look at the social sciences as a kind of joke is because so many contradictory studies come out of these peer-reviewed journals that their trustworthiness is quite low. One of the reasons why there's so much confusion surrounding what constitutes a healthy diet and how people should best attempt to lose weight is precisely because diet-and-exercise studies are more-or-less impossible to reproduce.

> If you can achieve that, great -- it's certainly a useful property to have for debugging

If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice.

I don't understand why it is so hard to see that you need either a pretty big collaboration where somebody else has isolated the specifications so you don't need to know anything about the problem your code solves really, or becoming a physics graduate student yourself for this line of work.

0: specifically, for results claiming a 95% confidence level (p<0.05), if nineteen replications are attempted, you should encounter a replication failure exactly once. I would accept perhaps four or five out of nineteen (or one out of two or three) under the reasoning that the law of large numbers hasn't kicked in yet, but anything with zero successful replications is not science, it's evidence (in this, against the entire field of study).

Contra your assertion, many people do some sort of regression testing like this but it's isn't terribly useful for verification or validation - but it is good at catching bad patches.

I used to be a scientist. I get it, getting scientists to do this is like pulling teeth, but it's the least you could do to give other people confidence in your results.

Ok, that's a reasonable ask :) But yeah as you implied, good luck getting the average scientist, who in the best case begrudgingly uses version control, to care enough to do this.

Scientists need to get over this fear about their code. They need to produce better code and need to actually start educating their students on how to write and produce code. For too long many in the physics community have trivialized programming and seen it as assumed knowledge.

Having open code will allow you to become better and you’ll produce better results.

Side note: 25 years ago I worked in accelerator science too.

Yes I understand how seeding PRNGs work and I personally do that for my own code for debugging purposes. My point was that not using a fixed seed doesn't invalidate their result. It's just a cheap shot and, to me, demonstrates that the lockdownskeptics author doesn't have a real understanding of the methods being used.

Also, to be clear, I support open science and have some of my own open-source projects out in the wild (which is not the norm in my own field yet). I'm not arguing against releasing code, I'm arguing against OP arguing against this particular piece of code.

The main issue is if it used sensible inputs, but that's entirely different from code quality and requires subject matter expertise, so programmers don't bother with such details -_-

I guess it can be made so, but not necessarily easy / fast (if it's parallel, and sensitive to floating point rounding). And sounds like the kind of engineering effort GP is saying isn't worth it. Re-running exactly the same monte-carlo chain does tell you something, but is perhaps the wrong level to be checking. Re-running from a different seed, and getting results that are within error, might be much more useful.

But still, for code running on different machines, especially for numeric-heavy code that might be running on a particular GPU setup, distributed big data source (where you pull the first available data rather than read in a fixed order), or even on some special supercomputer, it's hard to ask that it be totally reproducible down to the smallest rounding error.

That's a nitpick, but if the computation is executed in parallel threads (e.g. on multicore, or on a multicomputer), and individual terms are, for example, summed in a random order, caused by the non-determinism introduced by the parallel computation, then the result is not strictly deterministic. This is a property of floating-point computation, more specifically, the finite accuracy of real floating-point implementations.

So, it is not deterministic, but that should not cause large qualitative differences.

Perhaps if you use only single-threaded computation, you are interested in averages, and the processes you are interested in behave well and mostly linear.

But

- running code in parallel easily introduces non-determinism, even if your result computation is as simple as summing up results from different threads

- the processes one is examining might be highly non-linear - like lightning, weather forecasts, simulation of wildfires, and also epidemic simulations

- especially for all kind of safety research, you might actually be interested not only in averages, but in freak events, like "what is the likelihood that you have two or three hurricanes at the same time in the Gulf of Mexico", or "what happens if your nuclear plant gets struck by freak lightning in the first second of a power failure".

What should be reproducible are the conclusions you come to, not the hashed bits of program output.

> If you aren’t doing that, you aren’t running your Monte- Carlo correctly. That’s a huge red flag.

No, it does not follow from that.

Given that such software forms the very foundation of the results of such papers, why shouldn't it fall under scrutiny, even for "minor" points? If you are unable to produce good technical content, why are you qualified to declare what is or isn't minor? Isn't the whole point that scrutiny is best left to technical experts (and not subject experts)?

A non-native English speaker may make grammatical mistakes when communicating their research in English—it does not in any way invalidate their results or hint that there is anything amiss. It is simply what happens when you are a non-native speaker.

Some (many?) code critiques by people unfamiliar with the field of study the research will be about superficial mistakes that do not invalidate the results. They are the code equivalents of grammatical mistakes. That's what the OP is talking about.

First, it's not no different--it's completely different. Third parties have always constructed their own apparatus to reproduce an experiment. They don't go to the original author's lab to perform the experiment!

Second, a lot of scientific code won't run at all outside the environment it was developed in.

If it's HPC code, it's very likely that the code makes assumptions about the HPC cluster that will cause it to break on a different cluster. If it's experiment control / data-acquisition code, you'll almost certainly need the exact same peripherals for the program to do anything at all sensible.

I see a lot of people here on HN vastly over-estimating the value of bit-for-bit reproducibility of one implementation, and vastly underestimating the value of having a diversity of implementations to test an idea.

Same with grammatical or spelling errors. I don't review research but I do review resumes, and I've seen atrocious spelling on resumes. Here's the candidate's first chance to make an impression. They have all the time in the world to proofread, hone, and have other eyes edit it. Yet, they still miss obvious mistakes. If hired, will their work product also be sloppy?

If we're supposed to accept scientific results on faith, why bother with science at all?

If code is what is substantiating a scientific claim, then code needs to stand up to scientific scrutiny. This is how science is done.

I came from physics, but systems and computer engineering was always an interest of mine, even before physics, I thought it was kooky-dooks that CS people can release papers w/o code, fine if the paper contains all the proofs but otherwise it shouldn't even be looked at. PoS (proof-of-science) or GTFO.

We are the point in human and scientific civilization that knowledge needs to prove itself correct. Papers should be self contained execution environments that generate PDFs and resulting datasets. The code doesn't need to be pretty, or robust, but it needs to be sealed inside of a container so that it can be re-run, re-validated and someone else can confirm the result X years from now. And it isn't about trusting or not trusting the researcher, we need to fundamentally trust the results.

I doubt any physicist believes we need to get the Tevatron running again just to check decade-old measurements of the top quark. I don't understand why decade-old scientific software code must meet that bar.

Anyways, in physics they always require several experimental proves for our theory. They also have several "software experiments" for e.g. predicting the same observables. Therefore, researchers need to be able to compile and run the code of their competitors in order to compare and verify the results in detail. In this place, bug-hunting/fixing is sometimes also taking place - of course. So applying the articles suggestions would have the potential to accelerate scientific collaboration.

btw; I know some people who do still work with the data taken at the LEP experiment which was shut down almost 20 (!) years ago and they have a hard time in combining old detector-simulations, monte-carlos etc. with new data-analysis techniques for the exact same reasons mentioned in the article. For large-scale experiments it is a serious problem which nowadays has much more attention than at LEP ages, since LHC has anyways obvious big-data problems to solve before their next upgrade, including also software-solutions.

I suspect that software really is different in this way, and treating it like it's complex, one off hardware is cultural inertia that's going to fade away.

A container can be a .tar.gz, a zip or a disk image of artifacts, code, data and downstream deps. The generic word has been co-opted to mean a specific thing which is very unfortunate.

But the "implementation detail" of which container format you use, and whether it will still be supported in 10 years, is not an implementation detail at all -- since this will determine whether containerization actually solves the problem of helping your code run a decade later. This gets worse as the number, complexity and of container formats expands.

Of course if what you mean is that researchers should provide perpetual maintenance for their older code packages, moving them from one obsolete platform to a more recent one, then you're making a totally different and very expensive suggestion.

The first step is just having your code, data and deps in an archive. Depending on the project and the age, more stuff makes it into the archive. I have been on projects where the source to the compiler toolchain was checked into the source repo and the first step was to boostrap the tooling (from a compiler binary checked into the repo).

We aren't even to the .tar.gz stage yet.

Specifically, to that point, I want to cite the saying:

"The dogs bark, but the caravan passes."

(There is a more colorful German variant which is, translated: "What does it bother the mighty old oak tree if a dog takes a piss...").

Of course, if you publish your code, you expose it to critics. Some of this will be unqualified. And as we have seen in the case e.g. of climate scientists, some might be even nasty. But who cares? What matters is open discussion which is a core value of science.

It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas.

> how do you expect anyone with the right expertise to assess your findings

I would expect other experts in the field to write their own implementation of the scientific ideas expressed in a paper. If the idea has any merit, their implementations should produce similar results. Which is exactly what they would do if it were a physical experiment.

If I'm given bad information and I act on that information, then problems can occur.

Similarly, if the software is giving the scientist bad information, problems can occur.

How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software portion of the research as well?

There was a story on HN a while back about a professor who had written software, had come to some conclusions, and even had a Ph.D. student working on research based on that work. Only to find out that a software flaw meant the conclusions weren't useful to anyone and that student ended up wasting years of their life.

---

This stuff matters. This isn't a model of reality, it's an exploration of reality. It would be like telling a hiker that terrain doesn't matter. They would, rightfully, disagree with you.

We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of software engineers makes little sense to me.

Of course, a bug that was introduced or kept with the counscious intention of fooling the reviewers and the readers is another story.

This is not what is being asked, shame on you for the strawman.

Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".

The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousness".

Of course this is not a blank forgiveness for every bad scientist out there. Of course we should aim at getting better. But we should make the difference between the ideal science process and the science as performed by a human, prone to errors, misunderstandings and mistakes, and realize that these things will always happen, however many times we call for "more rigor because bugs have consequences".

And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.

How this "rigor" you are calling for should manifest, then? Put bluntly, my point was that every software has bug, so how "more rigor" would help? What should we do, what should we ask for in _practical_ terms?

Also, please do not rephrase this last sentence as "oh so since every software has bugs, then you obviously say that we shouldn't fix bugs, anyway other bugs will remain!".

That's exactly what I'm going to do. Point out that we can demand better even in the face of a lack of perfection.

There are two problems here with your stance.

1. The assumption that all bugs are created equal, and 2. The assumption that the truth isn't the overriding concern of science.

It's real easy to define the set of bugs that are unacceptable in science. Any bug that would render the results inaccurate is unacceptable.

The fact that some jackass web developer wrote a bug that deleted an entire database in no way obviates that responsibility of the scientists.

I think that, while we could use a bit more training in software engineering best-practices in the science, the thesis is still that science is hard and we need real replication of everything before reaching important conclusions, and over-focusing on one specific type of errors isn't all that helpful.

It's not clear to me why you think I would argue that inaccuracies should be avoided in software but accept that they're ok for electrical systems.

And if you're a map maker, it's a bit rich to start claiming that the accuracy of your maps is unimportant. If code is "a way to test scientific ideas", then it kinda needs to work if you want meaningful results. Would you run an experiment with thermometers that were accurate to +-30° and reactants from a source known for contamination?

Of course, it is a difference whether you make a clinical study on drugs, and use a pocket calculator to compute a mean, or whether you research in numerical analysis, or are presenting a paper in how to use Coq to more efficiently prove the four-color theorem or Fermat's last theorem.

In short, much of science is not computer science, and for it, computation is just a tool.

Sure, some people might point out spurious bugs and "design issues" or whatever, boo hoo. But others might actually find flaws in the code that meaningfully affect science itself: true bugs.

Sure, they could do this by doing a full replication in a lab and then custom coding everything from scratch. But even then, all you have is two conflicting results, with no good way yet to determine which one is more right or why they disagree. Technically, you can use the scientific progress to eventually find bugs in the scientific process, but why waste so much time when publishing the code will allow for reviews to find bugs so much faster. Its pure benefit to science to not obscure its proofs and rigor.

How many actually try to reproduce the results by writing corresponding code themselves? Apparently lot of papers with slightly wrong findings because code errors have passed the peer review (all of us in the SWE bubble know how often bugs occur), at least in less prestigious journals.

There is nothing wrong with mandating the code to be supplied with the paper. Because, many time code is somewhere between the experimental setup and proof / result.

I myself had very bad experience with extending the undocumented Fortran 77 code (lots of gotos and common blocks) of my supervisor. Finally, I decided to rewrite the whole thing including my new results instead of just somehow embedding my results into the old code for two reasons: (1) I'm presumably faster in rewriting the whole thing including my new research rather than struggling with the old code and (2) I simply would not trust in the numerical results/phenomenology produced by the code. After all, I'm wasting 2 months of my PhD for the marriage of my own results with known results which -in principle- could have been done within one day if the code base would allow for it.

So yes, If it's a one-man-show I would not give too much on code quality (though unit tests and git can safe quite a lot of time during development) but if there is a chance that someone else is going to touch the code in near future it will save time to your colleagues and improve the overall (scientific) productivity.

PS: quite excited about my first post here

This makes me a little uneasy, as I'm not too worried about code quality can easily translate into Yes I know my code is full of undefined behaviour, and I don't care.

> PS: quite excited about my first post here

Welcome to HN! reddit has more cats, Slashdot has more jokes about sharks and laserbeams, but somehow we get by.

The latter isn't great practice, but if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

'Undefined behaviour' is a term-of-art in C/C++ programming, there's no ambiguity.

> if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

Code should be correct by construction, not correct by coincidence. Results from such code shouldn't be considered publishable. Mathematicians don't get credit for invalid proofs that happen to reach a conclusion which is correct.

Again, this isn't some theoretical quibble. There are plenty of sneaky ways undefined behaviour can manifest and cause trouble. [0][1][2]

In the domain of safety-critical software development in C, extreme measures are taken to ensure the absence of undefined behaviour. If scientists adopt a sloppier attitude toward code quality, they should expect to end up publishing invalid results. Frankly, this isn't news, and I'm surprised the standards seem to be so low.

Also, of all the languages out there, C and C++ are among the most unforgiving of minor bugs, and are a bad choice of language for writing poor-quality code. Ada and Java, for instance, won't give you undefined behaviour for writing int i; int j = i;.

[0] https://devblogs.microsoft.com/oldnewthing/20140627-00/?p=63...

[1] https://blog.regehr.org/archives/213

[2] https://cryptoservices.github.io/fde/2018/11/30/undefined-be...

See also my longer ramble on this topic at https://news.ycombinator.com/item?id=24264376

Let the scientists publish UB code, and even the artifacts produced, the executables. Then, if such problems are found in the code by professionals, they can investigate it fully and find if it leads to a tangible flaw that invalidates the research or not.

You would drive yourself mad pointing out places in math proofs where some steps, even seemingly important ones, were skipped. But the papers are not retracted unless such a gap actually holds a flaw that invalidates the rest of thr proof.

Let thdm publish their gross, awful, and even buggy code. Sometimes the bugs don't effect the outcomes.

Granted, it's not a guarantee that the results are wrong, but it's a serious issue with the experiment. I agree it wouldn't generally make sense to retract a publication unless it can be determined that the results are invalid. It should be possible to independently investigate this, if the source-code and input data are published, as they should be.

(It isn't universally true that reproduction of the experiment should be practical given that the source and data are published, as it may be difficult to reproduce supercomputer-powered experiments. iirc, training AlphaGo cost several million dollars of compute time, for instance.)

> this mindset is what keeps people from publishing the code in the first place

As I explained in [0], this attitude makes no sense at all. It has no place in modern science, and it's unfortunate the publication norms haven't caught up.

Scientific publication is meant to enable critical independent review of work, not to shield scientists from criticism from their peers, which is the exact opposite.

> Let the scientists publish UB code, and even the artifacts produced, the executables. Then, if such problems are found in the code by professionals, they can investigate it fully and find if it leads to a tangible flaw that invalidates the research or not.

I'm not sure what to make of 'professionals', but otherwise I agree, go ahead and publish the binaries too, as much as applicable. Could be a valuable addition. (In some cases it might not be possible/practical to publish machine-code binaries, such as when working with GPUs, or Java. These platforms tend to be JIT based, and hostile to dumping and restoring exact binaries.)

I agree with your final two paragraphs.

[0] https://news.ycombinator.com/item?id=24264376

Glad we agree, if you're aware of how your compiler handles these things, you can construct it to be correct in this way.

It won't be portable at all (even to the next patch version of the compiler), I would never let it pass a code review, but that doesn't sound like an issue that's relevant here.

I presume we agree but I'll do my usual rant against UB: Deliberately introducing undefined behaviour into your code is playing with fire, and trying to outsmart the compiler is generally a bad idea. Unless the compiler documentation officially commits to a certain behaviour (rollover arithmetic for signed types, say), then you should take steps to avoid undefined behaviour. Otherwise, you're just going with guesswork, and if the compiler generates insane code, the standards documents define it to be your fault.

It might be reasonable to make carefully disciplined and justified exceptions, but that should be done very cautiously. JIT relies on undefined behaviour, for instance, as ultimately you're treating an array as a function pointer.

> It won't be portable at all (even to the next patch version of the compiler)

Right, doing this kind of thing is extremely fragile. Does it ever crop up in real-life? I've never had cause to rely on this kind of thing.

It would be possible to use a static assertion to ensure my code only compiles on the desired compiler, preventing unpleasant surprises elsewhere, but I've never seen a situation where it's helpful.

This isn't the same thing as relying on 'ordinary' compiler-specific functionality, such as GCC's fixed-point functionality. Such code will simply refuse to compile on other compilers.

> I would never let it pass a code review, but that doesn't sound like an issue that's relevant here.

Disagree. It should be possible to independently reproduce the experiment. Robust code helps with this. Code shouldn't depend on an exact compiler version, there's no good reason code should.

Sounds like it is quite good science to do that, because it puts the computation on a pair of independent feet.

Otherwise, it could just be that the code you are using as a bug and nobody notes until it is too late.

The cross-checking is anyways good scientific practise, not only because of bugs in the code (that's actually a sub-leading problem imho), but because of the degree of difficulty of the problems and the complexity of their solutions (and their reproducibility). In that sense, cross-checking should discover both, scientific "bugs" and programming-bugs. The "debugging" is partly also done at the community level - at least in our field of research.

However, it is also a matter of efficiency. I -and many others too- need to re-implement not because of bug-hunting/cross-checking but simply because we do not understand the "ugly" code of our colleagues and instead of taking the risk to break existing code we simply write new one which is extremely inefficient (others may take the risk and then waste months on debugging and reverse-engineering which is also inefficient). So my point on writing "good code" is not so much about avoiding bugs but about being kind to you colleagues, saving them nerves and time (which they can then spend on actual science) and thus also saving taxpayers money...

Yes, that should be this way.

Also all cases where some company research team goes to a scientific conference and presents a nifty solution for problem X without telling how it was purportedly done, it should be absolutely required to publish code and data for this.

*And that's also something which is broken about software patents - patents are about open knowledge, software which uses such patents is not open - this combination should not be allowed at all).

I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

In-code selection statements like `if( True ) {...}`, where you have no idea what is being selected or why.

Code that only runs in the particular workspace image that contains some function that was hacked out to make things work during a debugging session 5 years ago.

Distributed projects where one person wrote the preprocessor, another wrote the simulation software, and a third wrote the analysis scripts, and they all share undocumented assumptions worked out between the three researchers over the course of two years.

Depending on implementation-defined behavior (like zeroing out of data structures).

Function and variable names, like `doit()` and `hold`, which make it hard to understand the intention.

Files that contain thousands of lines of imperative instructions with documentation like "Per researcher X" every 100 lines or so.

Code that runs fine for 6 hours, then stops because some command-line input had the wrong value.

I've seen all of these over the years. Even as a domain expert who has spoken directly with authors and project leads, this kind of stuff makes it very hard to tease out what the code actually does, and how the code corresponds to the papers written about the results.

As an admin it was quite frustrating, but I understand it sometimes when you know the person/project isn’t tested in a distributed environment. But when it’s the projects that do know how they’re used and still do those things...

Code exhibiting undefined behavior is a different kettle of fish...

How is "your code has error and I rebuke you" a more painful critique than "you are hiding your methodology and so I rebuke you"?

Here's a quick pass, I'm sure I'm missing stuff, but I've needed to code review a lot of science and production output and below is how I tend to think of it, especially taking efficiency of engineer/scientist time into account.

Production Ready?

* code well factored for extensibility, feature change, and multi-engineer contribution

* robust against hostile user input

* unit and integration tested

Science Ready?

* code well factored for readability and reproducibility (e.g. random numbers seeded, time calcs not set against 'now')

* robust against expected user input

* input data available? testing optional but desired, esp unit tests of algorithmic functions

* input data not available? a schema-correct facsimile of input data available in a unit test context to verify algorithms correct

Both?

* security needs assessed and met (science code might be dealing with highly secure data, as might production code)

* performance and stability needs met (production code more often requires long term stability, science sometimes needs performance within expected Big O to save compute time if it's a big calculation)

The key aspect of that code is that it's going to be run once or twice, ever, and it's only ever going to be run on a particular known set of input data. It's a tool (though complex) that we used (once) to get from A to B. It does not need to get refactored, because the expectation is that it's only ever going to be used as-is (as it was used once, and will be used only for reproducing results), it's not intended to be built upon or maintained. It's not the basis of the research, it's not the point of research, it's not a deliverable in that research, it's just a scaffold that was temporarily neccessary to do some task - one which might have been done manually earlier through great effort, but that's automated now. It's expected that the vast majority of the readers of that paper won't ever need to touch that code, they care only about the results and a few key aspects of the methodology, which are (or should be) all mentioned in the paper.

It should be reproducible to ensure that we (or someone else) can obtain the same B from A in future, but that's it, it does not need to be robust to input that's not in the input datafile - noone in the world has another set of real data that could/should be processed with that code. If after a few years we or someone else will obtain another dataset, then (after those few years, if that dataset happens) there would be a need to ensure that it works on that dataset before writing a paper about that dataset, but it's overwhelmingly likely that you'd want to modify that code anyway both because that new dataset would not be 'compatible' (because the code will be tightly coupled to all the assumptions in the methodology you used to get that data, and because it's likely to be richer in ways you can't predict right now) and you'd want to extend the analysis in some way.

It should have a 'toy example' - what you call 'a schema-correct facsimile of input data' that's used for testing and validation before you run it on the actual dataset, and it should have test scenarios and/or unit tests that are preferably manually verifiable for correctness.

But the key thing here is that no matter what you do, that's still in most cases going to be "write once, run once, read never" code, as long as we're talking about the auxiliary code that supports some experimental conclusions, not the "here's a slightly better method for doing the same thing" CS papers. We are striving for reproducible code, but actual reproductions are quite rare, the incentives are just not there. We publish the code as a matter of principle, knowing all well that most likely noone will download and read it. The community needs the possibility for reproduction for the cases where the results are suspect (which is the main scenario where someone is likely to attempt reproducing that code), it's there to ensure that if we later suspect that the code is flawed in a way where the flaws affect the conclusions then we can go back to the code and review it - which is plausible, but not that likely. Also, if someone does not trust our code, they can (and possibly should) simply ignore it and perform a 'from scratch' analysis of the data based what's said in the paper. With a reimplementation, some nuances in the results might be slightly different, but all the conclusions in the paper should still be valid, if the paper is actually meaningful - if a reimplementation breaks the conclusions, that would be a successful, valuable non-reproduction of the results.

This is a big change from industry practice where you have mantras like "a line of code is written once but read ten times", in a scientific environment that ratio is the other way around, so the tradeoffs are different - it's not worth investing refactoring time to improve readability, if it's expected that most likely noone will ever read that code; it makes sense to spend that effort only if and when you need it.

But I seem to get the vibe that this original Nature article is mostly about the auxiliary data analysis code for "non-simulated" experiments, while Hacker News seems biased towards fields like computer science, machine learning, etc.

In the first case, you must take into account all (un)imaginable corner cases and never allow the code to fail or hang up. In the second case it needs to produce a reproducible result at least for the published case. And do not expect it to be user-friendly at all.

Any effort to get code working for other people, or documented in any way would probably be seen as wasted effort that could be used to write more papers or create more results to create new papers.

This kind of reasoning was one of the many reasons I left academic research - I personally didn't value publications as deliverables.

Still, there's plenty of room to encourage good(/better) practices which cost essentially nothing, e.g. using $PWD rather than /home/bob/foo

Any effort to write a paper readable for other people, or document the experiment in any way would probably be seen as wasted effort that could be used to create more results.

The "don't show your work" argument only makes sense if you are doing PR, not science.

The kind of code I'll ship for production will include unit testing designed around edge or degenerate cases that arose from case analysis, usually some kind of end to end integration test, aggressive linting and crashing on warnings, and enforcing of style guidelines with auto formatting tools. The last one is more important than people give it credit for.

For research it would probably be sufficient to test that the code compiles and given a set of known valid input the program terminates successfully.

In general, scientists don't care how long it takes or how many resources the code uses. It is not a big deal to run a script for an extra hour, or use up a node of supercomputer. Extravagent solutions or added packages to make the code run smoother or faster is only wasting time. It speed/elegance only really matters when you know the code is going to be distributed to the community.

Basically scientists only care if the result, is true. If the result it outputs is sensible, defensible, reliable, reproducible. It would be considered a dick move to criticism someones code, if the code was proven to produce the correct result.

Formal proof is much much harder than making code understandable and reviewable. It can be done but it is not easy, and can yield surprising results:

https://en.wikipedia.org/wiki/CompCert

http://envisage-project.eu/proving-android-java-and-python-s...

If not by unit tests, code review or formal logic, then what?

You can then use unit testing for low-level infrastructure (e.g. checking that your ODE solver works as expected), but do the high-level testing via scientific validation. The first line of defense is to check that you don’t break any laws of physics, e.g. that energy and electric charge is conserved in your end results. Even small implementation mistakes can violate these.

Then you search for related existing publications of a theoretical or numerical nature, trying to reproduce their results; the more existing research your code can reproduce, the more certain you can be that it is at least consistent with known science. If this fails, you have something to guide your debugging; or if you’re very lucky, something interesting to write a paper about :).

The final validation step is of course to validate against experiments. This is not suited for debugging though, since you can’t easily say whether a mismatch is due to a software bug, experimental noise, neglected effects in the mathematical model, etc.

Cross referencing independent experiments and external datasets.

Science doesn't work like software. The code can be perfect and still not give results that reflect reality. The code can be logical and not reflect reality. Most scientists I know go in with the expectation that "the code is wrong" and its results must be validated by at least one other source.

Part of our process involves cleaning up code to a higher state of refinement as it gets closer to entering the production pipeline.

I've tested 30 year old code, and it still runs, though I had to dig up a copy of Turbo Pascal, and much of it no longer exists in computer readable form but would have to be re-entered by hand. Life was actually simpler back then -- with the exception of the built-ins of Turbo Pascal, it has no dependencies.

My code was in fact adopted by two other research groups with only minor changes needed to suit slightly different experimental conditions. It contained many cross-checks, though we were unaware of modern software testing concepts at the time.

For a result to have broader or lasting impact, replication is not enough. The result has to fit into a broader web of results that reinforce one another and are extended or turned into something useful. That's the point where precise replication of minor supporting results becomes less important. The quality of any specific experiment done in support of modern electromagnetic theory would probably give you the heebie jeebies, but the overall theory is profoundly robust.

The same thing has to happen when going from prototype to production. Also, production requires what I call push-button replication. It has to replicate itself at the click of a mouse, because the production team doesn't have domain experts who can even critique the entirety of their own code, and maintaining their code would be nearly impossible if it didn't adhere to standards that make it maintainable by multiple people at once.

My own code was quite clean in the 1980s, when the limitations on the machines themselves tended to keep things fairly compact with minimal dependencies. And I learned a decent "structured programming" discipline.

As I moved into more modern languages, my code kind of degenerated into a giant hairball of dependencies and abstractions. "Just because you can do that, doesn't mean you should." I've kind of learned that the commercial programmers limit themselves to a few familiar patterns, and if you try to create a new pattern for every problem, your code will be hard to hand off.

Scientists would benefit from receiving some training in good programming hygiene.

In the common case, the software behaves as expected (random output), but it is reproducible for tests. You can then publish your RNG seed with the commit hash when you release your code/paper, and others may see your results and investigate that particular code execution.

Code belongs with the paper. Otherwise we can just continue to make up numbers and pretend we found something significant.

I've spent the intervening years trying to find a way to implement this myself, going as far as to buy things like the ice40 fpga because the bitstreams are supposedly unlocked; this is a pre-req for modifying the actual gate/logic on the chip.

I've emailed the professor listed as the headliner in the articles published about it to no avail.

Nearly my entire adult life has been spent reading some interesting article, chasing down the paper, finding out if any code was published, and seeing if I could run the code myself.

It wasn't until machine learning with pytorch became mainstream that I started having luck replicating results. Just some more data points for this discussion.

In what way do idiots making idiotic comments about your correct code invalidate your scientific production? You can still turn out science and let people read and comment freely on it.

> As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result.

I guess you would not need to engage personally with the idiots at "acceleratorskeptics.com", but likely most of their critique would be easily shut off by a simple sentence such as this one. Since most of your readers would not be idiots, they could scrutinize your code and even provide that reply on your behalf. This is called the scientific method.

I agree that you produce science, not merely code. Yet, the code is part of the science and you are not really publishing anything if you hide that part. Criticizing scientific code because it is bad software engineering is like criticizing it because it uses bad typography. You should not feel attacked by that.

How would a layperson identify a faulty critique? It would be picked up by the media who would do their usual “both sides” thing.

An educated and motivated layperson at least would have the chance to learn whether the critique is faulty. Today, with secret code, it is impossible to verify for almost everybody.

However, a methods section is always under-specified. Code provides the unique opportunity to actually see the full methods on display and properly review their work. It should be mandated by all reputable journals and worked into the peer review process.

But that's exactly the problem.

Are you familiar with that bug in early Civ games where an overflow was making Ghandi nuke the crap out of everyone? What if your code has a similar issue?

What if you have a random value right smack in the middle of your calculations and you just happened to be lucky when you run your code?

I'm not that familiar with Monte Carlo, my understanding is that this is just a way to sample the data. And I won't be testing your data sampling, but I will expect that given the same data to your calculations part (eg, after the sampling happens), I get exactly the same results every time I run the code and on any computer. And if there are differences I expect you to be able to explain why they don't matter, which will show you were aware of the differences in the first place and you were not just lucky.

And then there is the matter of magic values that plaster research code.

Researchers should understand that the rules for "software engineering grade code" are not there just because we want to complicate things, but because we want to make sure the code is correct and does what we expect it to do.

/edit: The real problem is not getting good results with faulty code, is ignoring good solutions because faulty code.

If the proof on which the paper is based is in the code that produced the evidence, you absolutely need to be able to let a lambda user run it without specific knowledge to abide to the reproducible principle. Asking a reviewer to fiddle about like a IT professional to get something working is bound to promote lazy reviewing and either will result into dismissing the result or approval without real review.

And by the way producing a paper could be argued it isn't really science either, but if you are working with MSFT Office, you know there is a fair amount of non science work hours that has been put into that as well.

Not so fast. Monte Carlo code turns arbitrary RNG seeds into outputs. That process can, and arguably should be, deterministic.

To do your study, you feed your Monte Carlo code 'random enough' seeds. Coming up with the seeds does not need to be deterministic. But once the seeds are fixed, the rest can be deterministic. Your paper should probably also publish the seeds used, so that people can reproduce everything. (And so they can check whether your seeds are carefully chosen, or really produce typical outcomes.)

Sure, and that rationale works OK when your code operates in a limited, specialized domain.

But if you're modeling climate change or infectious diseases, and you expect your work to affect millions of human lives and trillions of dollars in spending, then you owe us a full accounting of it.

There are a lot of domains which do have a deep impact on society but are completely underfunded. For example research on ecology and declining insect populations. Or research on education. And domains like epidemiology, climate change research, cancer research or such are not known to earn better salaries to scientists as on average. Most scientists earn a pity.

> But if you're modeling climate change or infectious diseases, and you expect your work to affect millions of human lives and trillions of dollars in spending, then you owe us a full accounting of it.

What one can expect from scientists, in whatever subject they work, is honesty, integrity, a full account of their findings. What you can't expect is that they just turn into expert software engineers or make their working code beautiful. You can't expect them to work for free. What the academic system demands from them is that they work on their next paper instead, so if you want pretty code, you need at least in part to change the system.

Two nitpicks: a) it shouldn't change the conclusions, but MC calculations will get different results depending on the seed. and b) it is considered good practice in reproducible science to fix the seed so that the results of subsequent runs give exactly the same results.

Ultimately, I think there is a balance: really poor code can lead to incorrect conclusions... but you don't need production ready code for scientific exploration.

Anybody who has conducted experimental research will say they spent 80% of the time using a hammer or a spanner. Repairing faulty lasers or power supplies. This process of reliable and repeatable experimentation is the basis of science itself.

Computational experiments must be held to the same standards as physical experiments. They must be reproducible and they should be publicly available (if publicly funded).

Sounds like I should just become a scientist then.

Do you guys write unit tests or is that beneath you too?

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points.

The person which wrote the linked blog post writes that it was a software engineer at google. Unfortunately, that claim is not falsifiable as the person decided to remain anonymous.

> As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result.

The claim is that even with the same random seed for the random generator, the program produces different results, and this is explained by the allegation that it runs non-deterministic (in the sense of undefined behavior) in multiple threads. It claims also that it produces significantly different results depending on which output file format is chosen.

If this is true, the code would have race conditions, and as being impacted by race conditions is a form of undefined behavior, this would make any result of the program questionable, as the program would not be well-defined.

Personally, I am very doubtful whether this is true, this would be incredibly sloppy by the imperial college scientists. Some more careful analysis by a recognized programmer might be warranted.

However it underlines well the importance of the main topic that scientific code should be open to analysis.

> What I'm saying is that scientific code doesn't need to handle every special case or be easily usable by non-experts.

Fully agree with this. But it should try to document its limitations.

That’s not at all what that means. What are you talking about? As long as a Monte Carlo process works towards the same result it’s equivalent.

You’re speaking genuine nonsense as far as I’m concerned. Randomness doesn’t imply non deterministic. Non-determinitism in no way implies race conditions or undefined behavior. We care that the random process reaches the same result, not that the exact sequence of steps is the same.

This is what scientists are talking about. A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet.

https://lockdownsceptics.org/code-review-of-fergusons-model/

It says, word-by-word:

> Clearly, the documentation wants us to think that, given a starting seed, the model will always produce the same results.

>

>Investigation reveals the truth: the code produces critically different results, even for identical starting seeds and parameters.

> I’ll illustrate with a few bugs. In issue 116 a UK “red team” at Edinburgh University reports that they tried to use a mode that stores data tables in a more efficient format for faster loading, and discovered – to their surprise – that the resulting predictions varied by around 80,000 deaths after 80 days: ...

The bugs which the blog post implies here are such ones as described by Jens Regehr: https://blog.regehr.org/archives/213

Not that I do not endorse these statements in the blog - I am rather skeptical whether they are true at all.

What the authors of the blob post mean is clearly "undefined behaviour" in the sense of non-deterministic program execution of a program that is not well-formed. It is clear that many non-experts could confuse that with the pseudo-randomness implicit in Monte-Carlo simulations, but this is a very different thing. The first is basically a broken, invalid, and untrustworthy program. The second is the established method to produce a computational result by introducing stochastic behavior, which is for example how modern weather models work.

These are wildly different things. I do not understand why your comment just adds to the confusion between these two things??

> A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet.

As said, I don't endorse the critique in the blog. However, critique in a software implementation, as well as in scientific matters, should never carry a call on authority - it should logically explain what is the problem, with concrete points. Unfortunately, the cited blog post remains very vague about this, while claiming:

> My background. I have been writing software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company’s database product, amongst other jobs and projects. I was also an independent consultant for a couple of years.

It would be much better if, instead claiming that there could be race conditions, it could point to lines in the code with actual race conditions, and show how the results of the simulation are different when the race conditions are fixed. Otherwise, it just looks like he claims that the program is buggy, because he is in no position to question the science, and does not like the result.

Non-determinism can be caused by

a) random seeds derived from hardware, such as seek times in a HDD controller, which is fed into pseudo random number (PRNG) generation. This is not a problem. For debugging, or comparison, it can make sense to switch it off, though.

b) data race conditions, which is a form of undefined behavior. This not only can dramatically change results of a program run, but also invalidates the program logic, in languages such as C and C++. This is what he blog post in "lockdownskeptics.org" suggests. For the application area and its consequences, this would be a major nightmare.

c) What I had forgotten is that parallel execution (for example in LAM/MPI, map/reduce or similar frameworks) is inherently non-deterministic and, in combination with properties of floating-point computation, can yield different but valid results.

Here an example:

A computation is carried out on five nodes and they return the values 1e10, 1e10, 1e-20, -1e10, -1e10, in random order. The final result is computed by summing these up.

Now, the order of computation could be:

((((1e10 + 1e10) + 1e-20) + -1e10) + -1e10)

or it could be:

(((1e10 + -1e10) + 1e-20) + (+1e10 + -1e10))

In the first case, the result would be zero, in the second case, 1e-20, because of the finite length of floating point representation.

_However_... if the numerical model or simulation or whatever is stable, this should not lead to a dramatic qualitative difference in the result (otherwise, we have a stability problem with the model).

Finally, I want to cite one last paragraph from the post on lockdownskeptics.org:

> Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.

> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.

You are right with the distinction, I had data race conditions in mind.

Race conditions can well happen in a correct C/C++ multi-threaded program in the sense that the order of specific computation steps is sometimes random. And for operations such as floating-point addition, where order of operations does matter, the exact result can be random as a consequence. But the end result should not depend dramatically on it (which is what the poster at lockdownskeptics.org claims).

GPT-3 FTW!

> you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study.

It's profoundly unscientific to suggest that researchers should be given the choice to withhold details of their experiments that they fear will not withstand peer review. That's much of the point of scientific publication.

Researchers who are too ashamed of their code to submit it for publication, should be denied the opportunity to publish. If that's the state of their code, their results aren't publishable. Unpublishable garbage in, unpublishable garbage out. Simple enough. Journals just shouldn't permit that kind of sloppiness. Neither should scientists be permitted to take steps to artificially make it difficult to reproduce (in some weak sense) an experiment. (Independently re-running code whose correctness is suspect, obviously isn't as good as comparing against a fully independent reimplementation, but it still counts for something.)

If a mathematician tried to publish the conclusion of a proof but refused to show the derivation, they'd be laughed out of the room. Why should we hold software-based experiments to such a pitifully low standard by comparison?

It's not as if this is a minor problem. Software bugs really can result in incorrect figures being published. In the case of C and C++ code in particular, a seemingly minor issue can result in undefined behaviour, meaning the output of the program is entirely unconstrained, with no assurance that the output will resemble what the programmer expects. This isn't just theoretical. Bizarre behaviour really can happen on modern systems, when undefined behaviour is present.

A computer scientist once told me a story of some students he was supervising. The students had built some kind of physics simulation engine. They seemed pretty confident in its correctness, but in truth it hadn't been given any kind of proper testing, it merely looked about right to them. The supervisor had a suggestion: Rotate the simulated world by 19 degrees about the Y axis, run the simulation again, and compare the results. They did so. Their program showed totally different results. Oh dear.

Needless to say, not all scientific code can so easily be shown to be incorrect. All the more reason to subject it to peer review.

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points.

Why would you care? Science is about advancing the frontier of knowledge, not about avoiding invalid criticism from online communities of unqualified fools.

I sincerely hope vaccine researchers don't make publication decisions based on this sort of fear.

How exactly is this a bad thing?

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code.

But it should be noted that what you didn't say is that you're here to turn out accurate science.

This is the software version of statistics. Imagine if someone took a random sampling of people at a Trump rally and then claimed that "98% of Americans are voting for Trump". And now imagine someone else points out that the sample is biased and therefore the conclusion is flawed, and the response was "Hey, I'm just here to do statistics".

---

Do you see the problem now? The poster above you pointed out that the conclusions of the software can't be trusted, not that the coding style was ugly. Most developers would be more than willing to say "the code is ugly, but it's accurate". What we don't want is to hear "the conclusions can't be trusted and 100 people have spent 10+ years working from those unreliable conclusions".

This is exactly the sort of pedantic cluelessness that scientists are seeking to avoid by not publishing their code.

To paraphrase what the other developer said: "I don't want my work to be checked, I'm not here for accuracy, just the act of doing science".

When I was young, the ability to invalidate was the core aspect of science, but apparently that's changed over the years.

If you want to help me (and others who agree with me), please sign this petition: https://publiccode.eu. It demands that all publicly funded code must be public.

P.S. Yes, my 10-year-old code is working.

Well ... that part isn't nonsense, though I agree it shouldn't be a dealbreaker. And it means we should work towards making such support demands minimal or non-existent via easy containerization.

I note with frustration that even the Docker people, whose entire job is containerization, can get this part wrong. I remember when we containerized our startup's app c. 2015, to the point that you should be able to run it locally just by installing docker and running `docker-compose up`, and it still stopped working within a few weeks (which we found when onboarding new employees), which required a knowledgeable person to debug and re-write.

(They changed the spec for docker-compose so that the new version you'd get when downloading Docker would interpret the yaml to mean something else.)

This hits close to home. Back in college, I developed software, for a lab, for a project-based class. I put the code up on GitHub under the GPL license (some code I used was licensed under GPL as well), and when the people from the lab found out, they lost their minds. A while later, they submitted a paper and the journal ended up demanding the code they used for analysis. Their solution? They copied and pasted pieces of my project they used for that paper and submitted it as their own work. Of course, they also completely ignored the license.

Individual researchers, and in many disciplines (like physics), there is almost no emphasis on quality.

I left academia a decade ago, but at the time all except one of my colleagues protested when version control was suggested to them. Some of these have code in the 30-40K lines.

It’s bit disconcerting seeing how much quality is brushed aside particularly in software. Researchers seem to intuitively grasp how they need quality hardware to do their job, yet software rarely gets the same consideration. I’ve never been able to get many to come around to the idea that software should be treated the same as any other engineered product that enables their research

Academics are strange like this. The root reason is fear: fear that you're complicating their process, that you're going to interrupt their productivity or flow state, that you're introducing complication that has no benefit. They then build up a massive case in their minds for why they shouldn't do this; good luck fighting it.

Doubly so if you're IT staff and don't have a PhD. There's a fundamental lack of respect on behalf of (a vocal minority) of academics about bit plumbers, until of course when they need us to do something laughably basic. It's the seeds of elitism; in reality we should be able to work together, each of us understanding our particular domain and working to help the other.

Yes, but how does it compare to all the complicated processes that exist in academic institutions currently? Almost all of which originated from academics themselves, mind you.

This means it makes most sense to pick up processes that are portable and have longevity. Learning Git is a pretty solid example.

I think software development as a field is incredibly vast and diverse. Programming is an amazing tool, but it's a tool that requires a lot of knowledge in a lot of different areas.

^[1] This isn't universally true of course, I'm not trying to be insulting here.

Individual researchers, and in many disciplines (like physics), there is almost no emphasis on correct results, merely on believable results.

As an example, if your science has the requirement of being replicable (as it should) there are a host of best practices that should flow down to the software development requirements. Not implementing those best practices would be indicative of lower quality

In other cases, a couple of other researchers may have a look at my code or continue its development. I worked with 4+ research teams and only saw one professional programmer in one of them helping the development. Never heard about a "dedicated software assurance team".

One of my favorite points from the book Clean Code was that professional developers aren’t satisfied with “working code”, they aim to make it maintainable. Which may mean writing it in a way that is more clear and concise than we are used to

Are these a thing in any research area? I've heard of exactly one case of an academic lab (one that was easily 99th+ percentile in terms of funding) hiring one software engineer not directly involved in leading a research effort, and when I tell other academics about this they're somewhat incredulous. (I admittedly have a bit of trouble believing it myself -- I can't imagine the incentive to work for low academic pay in an environment where you're inevitably going to feel a sense of inferiority to first year PhD students who think they're hot shit because they're doing "research".)

I can say there are some that have the explicit intent but it can often fall to the wayside due to cost pressure. For example, government funded research from large organizations (think DoD or NASA) have these quality requirements but they can often be hand-waved away or just plain ignored due to cost concerns

You are blaming scientists but speaking from my personal experience as a computational scientist, this exists because there are few structures in place that incentivize strong programming practices.

* Funding agencies do not provide support for verification and validation of scientific software (typically)

* Few journals require assess code reproducibility and few require public code (few require even public data)

* There are few funded studies to reproduce major existing studies

Until these structural challenges are addressed, scientists will not have sufficient incentive to change their behavior.

> Scientific code needs to have tests, a minimal amount of test coverage, and code/data used really need to be published and run by volunteers/editors in the same way papers are reviewed, even for non-computer science journals.

I completely agree.

In my opinion, enforcing standards without addressing this root cause is not gonna fix the problem. Worse, students and early career researchers will bear the brunt of increased workload and code compliance requirements from journals. Big, well-funded labs that can afford a research engineer position is gonna have an edge over small labs that cannot do so.

After a paper has been accepted, authors can submit a repository containing a script which automatically replicates results shown in the paper. After a reviewer confirms that the results were indeed replicable, the paper gets a small badge next to its title.

While there could certainly be improvements, I think it's a step in the right direction.

Informal reputation does matter though. If you want to get things done and not just get promoted, you need the cooperation of people with a similar mindset, and collaboration is entirely voluntary.

All is well and good then, because journals absolutely don't care about science. They care about money and prestige. From personal experience, I'd say this intersects with the interests of most high-ranking academics. So the only unhappy people are idealistic youngsters and science "users".

Let's get back to non-profit journals.

In the research germane to this conversation, software is the means by which the scientific data is generated. If the software is flawed, it undermines the confidence in the data and thus the conclusions.

I think it’s too easy to game the data (whether knowingly or not) with poor methodology. I advocate process before product, in other words.

In terms of tangible results, Princeton at least has created a dedicated team of software engineers as part of their research computing unit (https://researchcomputing.princeton.edu/software-engineering).

Realistically though even if the necessity of research software engineering were acknowledged at the institutional level at the bulk of universities, there would still be the problem of universities paying way below market rate for software engineering talent...

To some degree, universities alone cannot effect the change needed to establish a professional class of software engineers that collaborate with researchers. Funding agencies such as the NIH and NSF are also responsible, and need to lead in this regard.

Most research programmers, in my experience, work in a lab for a PI. Over time, these programmers have become more valued by their team. However, they often still face a hard cap on career advancement. They generally are paid considerably less than they'd earn in the private sector, with far less opportunity for career growth. I think they often make creative contributions to research that would be "co-author" level worthy if they came from someone in an academic track, but they are frequently left off publications. They don't get the benefits that come with academic careers, such as sabbaticals, and they often work to assignment, with relatively little autonomy. The right career path and degree to build the skills required for this kind of programming is often a mismatch for the research-oriented degrees that are essential to advancement in an academic environment (including leadership roles that aren't research roles).

In short, I think there is a deep need for the emerging "research software engineer" you mention, but at this point, I can't recommend these jobs to someone with the talent to do them. There are a few edge cases (lifestyle, trailing spouse in academic, visa restrictions), but overall, these jobs are not competitive with the pay, career growth, autonomy, and even job security elsewhere (university jobs have a reputation for job security, but many research programmers are paid purely through a grant, so often these are 1-2 year appointments that can be extended only if the grant is renewed).

The Princeton group you linked to is encouraging - working for a unit of software developers who engage with researchers could be an improvement. Academia is still a long, long way away from building the career path that would be necessary to attract and keep talent in this field, though.

https://mobile.twitter.com/id_aa_carmack/status/125819213475...

Why is there so much c++ code?

Skeptics could have a field day tearing apart the estimates for the large number of input parameters to models like that, but they choose not to? I don't get it.

Many years ago, a paper on the PageRank algorithm was written, and the code behind that paper was monetized to unprecedented levels. Should computer science journals also require working proof of concept code, even if that discourages companies from sharing their results; even if it prevents students from monetizing the fruits of their research?

Scientists, not programmers, should be the ones spear-heading the development of standards and rules of thumb.

Still, there are real problematic practices that an emphasis on sharing scientific code would discourage. One classic one is the use of a single script that you edit each time you want to re-parameterize a model. Unless you copy the script into the output, you lose the informational channel between your code and its output. This can have real consequences. Several years ago I started up a project with a collaborator to follow up on their unpublished results from a year prior. Our first task was to take that data and reproduce the results they obtained before, because the person no longer had access to the exact copy of the script that they ran. We eventually determined that the original result was due to a software error (which we eventually identified). My colleague took it well, but the motivation to continue the project was much diminished.

Why isn't there a common language that all other languages compile to, and that will be supported on all possible platforms, for the rest of time?

(Perhaps WASM could be such a language, but the point is that this would be just coincidental and not a planned effort to conservate software)

And why aren't package managers structured such that packages will live forever (e.g. in IPFS) regardless of whether the package management system is online? Why is Github still a single point of failure in many cases?

After they embark on solving problems, it does become an eyeopening experience, and one that becomes now about keeping things running.

For those who have a STEM discipline in addition to a software development background >5Y, would you agree with seeing the above?

I would have thought the scientists among us would approach someone with familiarity with software development expertise. (something abstract and requiring a different set of muscles)

One positive emerging is the variety of low/no-code tooling that can replace a lot of this hornets nest coding.

In any case you'd need to teach them the problem domain, and it's considered cheaper (and simpler from organizational perspective) to get some phd students or postdocs from your domain to spend half a year getting up to speed on coding (and they likely had a few courses in programming and statistics anyway) than to hire an experienced software developer and have them learn the basics of your domain (which may well take a third or half of the appropriate undergraduate bachelor's program).

This is a really key point that is lost on devs outside of science looking in. In our case, good devs are out of budget by a factor of 2x at least (at an EU public university in a lab doing lots of computational work).

The best we get are engineers which are expected to keep the cluster running, order computers, organize seminars.. and eventually resolve any software or dev problems. This doesn’t leave much time for caring about reproducibility outside the very core algorithms. The overall workflow can fade away since the next post doc is going to redo it anyway.

The big issue with hiring software developers is that the 'payscale' is set according to the academic criteria, and an external developer coming from the industry - no matter how experienced or skilled - can usually be offered only a junior position with pay appropriate for that because they do not meet the criteria required for non-junior positions (no PhD, often not even a masters', no relevant publications, etc). From that perspective the only difference between a grad student who just started and a seasoned software developer is that the grad student can be employed as a part-time research assistant while a 'pure' developer could be full-time; but the hourly rate and conditions would be pretty much the same, targeted at less experienced employees. We can hire skilled mid-level individual contributors with reasonable pay for post-doc positions, but post-doc positions are limited to candidates who have a PhD. And it's not a that big limitation, since it's expected that everybody who's working "in the field" will get a PhD during their first few years of practical work experience as a grad student, the concept of "experienced/skilled but no degree" is not considered by the system as such people are rare in academia, and they stay rare due to the existing system.

So the disparity in evaluation criteria means that it's tricky to transfer between the different "career paths" - if you come from an environment where degrees mostly don't matter to an environment where a PhD is almost table stakes (to be a "hiring scientist", PhD is mandatory but nowhere near sufficient), then "getting your worth" is possible only if you are willing to put in quite some time and effort to fit the criteria used to evaluate scientists, even if you're there just to do software development.

My point is that it places outsiders (no matter if they're going to do software development or something else) on a lower payscale until they catch up on all the academia-specific factors of evaluation.

It's not a caste system between different types of activities, but rather a barrier of entry - in some sense, you have to start from 'level 1' no matter how much experience you have in other fields, so inexperienced people can join easily, but for senior/experienced people doing it is possible but costly.

There are many problems with using a dedicated programmer, or any other technical specialist in a small R&D team. The first is keeping them occupied. There was programming to be done, but not full time. And it had to be done in an extremely agile fashion, with requirements changing constantly, often at the location where the problem is occurring, not where their workstation happens to be set up. Many developers hate this kind of work.

Second is just managing software development. Entire books have been written about the topic, and it's not a solved problem how to keep software development from eating you alive and taking ownership of your organization. Nobody knows how to estimate the time and effort. You never know if you're going to be able to recover your source code and make sense of it, if your programmer up and quits.

With apologies to Clemenceau, programming is too important to be left to the programmers. ;-)

This is a general problem we all have, whenever we should employ a professional to do necessary work. The right professional will take a tenth of the time and the job will be done some multiple better. But how do you pick the right person?

I have two experiences with post-grad work that I think are relevant:

1. A friend needed some work done in a statistics package that used a language that felt like it was from the 80's. I was able to complete the work in a few hours, but I don't think a student could have done it (complicated need combined with a crappy language and IDE).

2. Another postdoc engineering friend needed to do some heavy duty data analysis, and she was recommended to learn C++. I suspect she wasted years learning C++, time which should have been spent on investigating different forms of analysis. She wanted to listen to her engineering fellows, not some practicing software engineer, so wasted her life not achieving much...

Is there a pool of skilled software architects willing to provide consultations at well-below market wages? Or a Q&A forum full of people interested in giving this kind of advice? (StackOverflow isn't useful for this; the allowed question scope is too narrow.) I guess one incentive to publish one's code is to get it criticized on places like Hacker News. The best way to get the right answer on the internet is to post the wrong answer, after all.

The idiosyncrasies and tastes of many 'senior' software engineers would likely make the code unreadable and unmaintainable for the average scientist and possibly discourage them from programming altogether.

Software architecture is an unsolved problem as evident in the frequent fundamental discussions about even trivial things, highlighted by a cambrian explosion of frameworks who try to help herding cats, and made obvious in senior programmers struggling to get a handle on moderately complex code.

I propose scientists keep their code base as simple as possible, review the code along with the ideas with their peers, maybe use Jupyter notebooks to show the iterations and keep intermediate steps, and, as others state, show the code as appropriate and try to keep it running. There is no silver bullet and very few programmers could walk into your lab or office and really clean things up the way you'd hope.

Life is different when you might have the a relationship with a single code base for 2-5 years, or even more. Complexities will happen on their own, no need to add them in.

Yes. Well, in engineering anyway. That's why most engineers use academia as a stepping stone to something else. Working in science is, I think, sort of like working at a startup that's perpetually short on cash with no possibility of an exit.

With a little digging you can find published tables of job codes and salaries for many universities, e.g., https://www.udel.edu/content/dam/udelImages/human-resources/...

Faculty are listed separately for some reason: https://www.udel.edu/faculty-staff/human-resources/compensat...

Positions requiring a PhD start being listed at 29E (midrange of $55k) or 30E (midrange of $63,800). You could easily get that with a bachelor's degree in engineering 10 years ago. I suspect you will find the "Information Technology" Job Family salaries particularly amusing.

However, when working as equals scientists and engineers can create truly transformative projects. Algorithms accounts for 10% of the solution. The code, infrastructure and system design accounts for 20% of the final result. The remaining 70% of the value, is directly coming from its impact. A projects that nobody uses is a failure. Something that perfectly solves a problem that nobody cares about is useless.

>

> [0] https://lockdownsceptics.org/code-review-of-fergusons-model/

This does not looks like a good example at all, as it appears that the blog author there just tries to discredit the program because he does not like the results. He also writes that all epidemiological research should be defunded.

If someone is trying to reproduce someone else's results, the data and methods are the only ingredients they need. If you add code into this mix, all you do is introduce new sources of bias.

(Ideally the results would be blinded too.)

Show me the grant announcements that identify reproducible long term code as a key deliverable, and I’ll show you 19 out of 20 scientists who start worrying about it.