Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same instance of the program running on a central machine.

Does scientific-grade code need to publish telemetry? Eh, usually no. Set up alerting so that on-call engineers can be paged when (not if) it falls over? Nope.

Does scientific-grade code need to handle the authorization and authentication of users? Nope.

Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. The reproducibility of results is core to the scientific method. Yes, that includes Monte Carlo code, when there is no such thing as truly random number generation on contemporary computers, only pseudorandom number generation, and what matters for cryptographic purposes is that the seed numbers for the pseudorandom generation are sufficiently hidden / unknown. For scientific purposes, the seed numbers should be published on purpose, so that a) the exact results you found, sufficiently random as they are for the purpose of your experiment, can still be independently verified by a peer reviewer, b) a peer reviewer can intentionally decide to pick a different seed value, which will lead to different results but should still lead to the same conclusion if your decision to reject / refuse to reject the null hypothesis was correct.

I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication.

Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction between experiment and human brain meats that's missing when code is simply reused (whether we consider it apparatus or procedure).

Once we have multiple implementations, if there is a meaningful difference between them, at that point replayability is of tremendous value in identifying why they differ.

But it is not reproducibility, as we want that term to be used in science.

The software was "reproducible" in that the same starting conditions produced the same output, but that didn't mean the _science_ was reproducible, as every study used the same software.

I repeatedly brought it up, but I wasn't advanced enough in my studies to be able to do anything about it. By the time I felt comfortable with that, I was on my way out of the field and into an non-academic career.

I have kept up with the field to a certain extent, and there is now a project in progress to create a fully independent replacement for that original code that should help shed some light (in progress for a few years now, and still going strong).

This is the difference between reproducibility and replicability [1]. Reproducibility is the ability to run the same software on the same input data to get the same output; replication would be analyzing the same input data (or new, replicated data following the original collection protocol) with new software and getting the same result.

I've experienced the same lack of interest with established researchers in my field, but I can at least ensure that all my studies are both reproducible and replicable by sharing my code and data.

[1] Plesser HE. Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Front Neuroinform. 2018;11:76.

Something like this is the norm in some more mathematical fields, where only the polished final version is published, as if done by pure thought. To build that, first you have to reproduce it, invariably by building your own code -- perhaps equally awful, but independent.

The fact that we can have that option is an amazing opportunity that a confluence of attributes of software (specificity, replayability, easy of copying) afford us. Where we are not exploiting this like we could be, it is a failure of our institutions! But it is different-in-kind from traditional reproducibility.

More concretely, it is 100% fair (and I might argue necessary) to demand more of our institutions and work to improve their failures. I'm sure many researchers have encountered publications of the form "we applied <proprietary model (TM)> (not explained) to <proprietary data> (partially explained) after <two sentence description of preprocessing> and obtained SOTA results!" in a reputable venue. Sure, this might be even less reproducible 200 years ago than now, but the authors would also be less likely to be competing with you for limited funding! Debating about the traditional definition of reproducibility has its place, but we should also be doing as much as possible to give reviewers and replicators a leg up. This is often flies in the face of many incentives the research community faces, but shifting blame to institutions by default (not saying you're doing this, but I've seen many who do) is taking the easy road out and does little to help the imbalanced ratio of discussion:progress.

But repeating the simulation a large number of times, with different random seeds, should produce statistically similar output if the code is rigorous. So even if each simulation is not reproducible, as long as the statistical distribution of outputs is reproducible, that should be sufficient.

There are some kinds of code (a script that controls a sensor or an actuator) where I think that doesn't match up well at all. There are plenty of kinds of code where they are, in fact, simply crunching numbers produced earlier. For the latter, I'm honestly not sure the best way to treat it, except to say that we should be sure that enough information is included in some form that replication should be possible, and that we keep in mind the idea that replication should involve human interaction.

Edit: more careful phrasing.

On the contrary; in that case there are four possibilities:

a: your algorithm doesn't work at all, and your observations are a artifact of convenient inputs or inept measurements.

b: your algorithm works, but the description in the paper is wrong or incomplete

c: your algorithm works as described, but the replicater implemented it incorrectly

d: other

Having the original implementation code is necessary to distinguish between cases a and b versus case c, and if the former, the code for the test harness is likely to help distinguish a versus b. (Case d is of course still a problem, but that doesn't mean it's reasonable to just give up.)

I'm all for more transparency, and this includes making codes and data public as much as is reasonable. But the real test is if someone can independently replicate the result, and how to incentivize replication studies (in both computational and experimental science) is also important, and in my view should not be divorced from discussions of reproducibility.

Edit: rewrote to clarify my position.

We've certainly done a lot, scientifically speaking (in terms of post-validated studies), without that level of reproducibility.

well that's just not true. there's no shortage of noise we can sample to get true random numbers. we just often stretch the random numbers for performance purposes.

This reminds me of being in gradschool and the comp-sci people complaining that we don't get bit-wise equal floats when we solve DEs.

Having to re-implement a library from scratch for a project is much more valuable than running the same code in two places. The same way that getting the same results from two different machines is a lot more significant than getting the same result from two cloned machines.

In short: code does not need to be reproducible because scientists know how to average.

This is definitely not correct. The experiment as a whole needs to be reproducible independently. This is very different, and more robust, from requiring that a particular portion of a previous version of the experiment to be reproducible in isolation.

Reproducibility in the scientific sense is different from running the same program with the same input, and getting exactly the same result. Repreducibility means that if you repeat the measurements in another environment, getting somewhat different data, and apply the same theory and methods, you get to the same conclusion.

The property of a computer program that when you run it again with the same input, you get the same output, is nice and very helpful for debugging. But the fact that you can run the same program does not mean that it is bug-free, as much as the fact that you can copy a paper with a mathematical proof does not mean that the proof is correct.

Also, multi-threaded and parallel code is inherently non-deterministic.

> when there is no such thing as truly random number generation on contemporary computers, only pseudorandom number generation,

That is wrong. Linux for example uses latency measurements from drivers such as HDD drive seek latencies or keyboards to generate entropy. While it might not the best thing to rely on for purposes of cryptography, it is surely not deterministic. If it would matter, you could download real-time astronomical noise measurements and use them to seed your Mersenne Twister generator.

Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome.

That an identical experiment leads to an identical outcome fails to validate the hypothesis, because the casual factors may have been misidentified.

Precise reproducibility still matters if generalized reproduction fails, however, because the differences in the experimental implementation may lead to new and more accurate hypothesis about causality.

The problem here is a step before this: The results of these "identical" experiments are so wildly different there is nothing valid to propose, let alone for a recreation to compare against.

So what happens if/when an identical experiment fails to lead to an identical outcome?

Scientific reproducibility does not mean, and has never meant, you rerun the code and the output perfectly matches bit-for-bit every time. If you can achieve that, great -- it's certainly a useful property to have for debugging. But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold.

To paraphrase a comment I saw from another thread on HN: "Plenty of good science got done before modern devops came to be."

[0] https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/18096

[1] https://pytorch.org/docs/stable/notes/randomness.html

==========

EDIT to reply to solatic's replies below (I'm being rate-limited):

The social science arguments are probably fair (or at least I'll leave it to someone more knowledgeable to defend them if they want) -- perhaps I shouldn't have led with the example of "different groups of participants".

> If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice.

This might be a reasonable thing to enforce if everyone in the field were using the same computing platform. Given that they're not (and that telling everyone that all published results have to be done using AWS with this particular machine configuration is not a tenable solution) I don't see how this could ever be a realistic requirement. Or if you don't want to enforce that the results remain identical across different platforms, what's the point of the requirement in the first place? How would it be enforced if nobody else has the exact combination of hardware/software to do so? And then even if someone does, almost inevitably there'll be some detail of the setup that the researcher didn't think to report and results will differ slightly anyway.

Besides, if you're allowing for exemptions, just about every paper in machine learning studying datasets larger than MNIST (where asynchronous prefetching of data is pretty much required to achieve decent speeds) would have a good reason to be exempt. It's possible that there are other fields where this sort of requirement would be both useful and feasible for a large amount of the research in that field, but I don't know what they are.

> Also, reading through the issues you linked points to: https://github.com/NVIDIA/framework-determinism which is a relatively recent attempt by nVidia to support deterministic computation for TensorFlow. Not perfect yet, but the effort is going there.

(From your other comment.) Yes, there exists a $300B company with an ongoing-but-incomplete funded effort of so far >6 months' work (and that's just the part they've done in public) to make one of its own APIs optionally deterministic when it's being used through a single downstream client framework. If this isn't a perfect illustration that it's not realistic to expect exact determinism from software written by individual grad students studying chemistry, I'm not sure what to say.

==========

EDIT to reply to BadInformatics' reply below (I'm being rate-limited): I fully agree that a lot of ML code releases could be better about this, and it's even reasonable to expect them to do some of these more basic things like you mention. I don't agree that bit-for-bit reproducibility is a realistic standard that will get us there.

Because I have and I pity anyone who tries and builds a kernel that can run it.

A lot of open sourced CS research is not reproducible.

"the code still runs and gives the same output" is not the same as reproducibility.

I'm not sure if this was meant to be a counter-argument to me, but I completely agree!

> "the code still runs and gives the same output" is not the same as reproducibility.

Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.

It wasn't :)

> Plenty of good science got done before modern devops came to be

This isn't as strong of an argument as you think. This is more-or-less the underlying foundation behind the social sciences, which argues that no social sampling can ever be entirely reproduced since no two people are alike, and even the same person cannot be reliably sampled twice as people change with time.

Has there been "good science" done in the social sciences? Sure. I don't think that you're going to find anybody arguing that the state of the social sciences today is about the same as it was in the Dark Ages.

With that said, one of the reasons why so many laypeople look at the social sciences as a kind of joke is because so many contradictory studies come out of these peer-reviewed journals that their trustworthiness is quite low. One of the reasons why there's so much confusion surrounding what constitutes a healthy diet and how people should best attempt to lose weight is precisely because diet-and-exercise studies are more-or-less impossible to reproduce.

> If you can achieve that, great -- it's certainly a useful property to have for debugging

If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice.

I don't understand why it is so hard to see that you need either a pretty big collaboration where somebody else has isolated the specifications so you don't need to know anything about the problem your code solves really, or becoming a physics graduate student yourself for this line of work.

0: specifically, for results claiming a 95% confidence level (p<0.05), if nineteen replications are attempted, you should encounter a replication failure exactly once. I would accept perhaps four or five out of nineteen (or one out of two or three) under the reasoning that the law of large numbers hasn't kicked in yet, but anything with zero successful replications is not science, it's evidence (in this, against the entire field of study).

Contra your assertion, many people do some sort of regression testing like this but it's isn't terribly useful for verification or validation - but it is good at catching bad patches.

I used to be a scientist. I get it, getting scientists to do this is like pulling teeth, but it's the least you could do to give other people confidence in your results.

Ok, that's a reasonable ask :) But yeah as you implied, good luck getting the average scientist, who in the best case begrudgingly uses version control, to care enough to do this.

Scientists need to get over this fear about their code. They need to produce better code and need to actually start educating their students on how to write and produce code. For too long many in the physics community have trivialized programming and seen it as assumed knowledge.

Having open code will allow you to become better and you’ll produce better results.

Side note: 25 years ago I worked in accelerator science too.

Yes I understand how seeding PRNGs work and I personally do that for my own code for debugging purposes. My point was that not using a fixed seed doesn't invalidate their result. It's just a cheap shot and, to me, demonstrates that the lockdownskeptics author doesn't have a real understanding of the methods being used.

Also, to be clear, I support open science and have some of my own open-source projects out in the wild (which is not the norm in my own field yet). I'm not arguing against releasing code, I'm arguing against OP arguing against this particular piece of code.

The main issue is if it used sensible inputs, but that's entirely different from code quality and requires subject matter expertise, so programmers don't bother with such details -_-

I guess it can be made so, but not necessarily easy / fast (if it's parallel, and sensitive to floating point rounding). And sounds like the kind of engineering effort GP is saying isn't worth it. Re-running exactly the same monte-carlo chain does tell you something, but is perhaps the wrong level to be checking. Re-running from a different seed, and getting results that are within error, might be much more useful.

But still, for code running on different machines, especially for numeric-heavy code that might be running on a particular GPU setup, distributed big data source (where you pull the first available data rather than read in a fixed order), or even on some special supercomputer, it's hard to ask that it be totally reproducible down to the smallest rounding error.

That's a nitpick, but if the computation is executed in parallel threads (e.g. on multicore, or on a multicomputer), and individual terms are, for example, summed in a random order, caused by the non-determinism introduced by the parallel computation, then the result is not strictly deterministic. This is a property of floating-point computation, more specifically, the finite accuracy of real floating-point implementations.

So, it is not deterministic, but that should not cause large qualitative differences.

Perhaps if you use only single-threaded computation, you are interested in averages, and the processes you are interested in behave well and mostly linear.

But

- running code in parallel easily introduces non-determinism, even if your result computation is as simple as summing up results from different threads

- the processes one is examining might be highly non-linear - like lightning, weather forecasts, simulation of wildfires, and also epidemic simulations

- especially for all kind of safety research, you might actually be interested not only in averages, but in freak events, like "what is the likelihood that you have two or three hurricanes at the same time in the Gulf of Mexico", or "what happens if your nuclear plant gets struck by freak lightning in the first second of a power failure".

What should be reproducible are the conclusions you come to, not the hashed bits of program output.

> If you aren’t doing that, you aren’t running your Monte- Carlo correctly. That’s a huge red flag.

No, it does not follow from that.

Given that such software forms the very foundation of the results of such papers, why shouldn't it fall under scrutiny, even for "minor" points? If you are unable to produce good technical content, why are you qualified to declare what is or isn't minor? Isn't the whole point that scrutiny is best left to technical experts (and not subject experts)?

A non-native English speaker may make grammatical mistakes when communicating their research in English—it does not in any way invalidate their results or hint that there is anything amiss. It is simply what happens when you are a non-native speaker.

Some (many?) code critiques by people unfamiliar with the field of study the research will be about superficial mistakes that do not invalidate the results. They are the code equivalents of grammatical mistakes. That's what the OP is talking about.

First, it's not no different--it's completely different. Third parties have always constructed their own apparatus to reproduce an experiment. They don't go to the original author's lab to perform the experiment!

Second, a lot of scientific code won't run at all outside the environment it was developed in.

If it's HPC code, it's very likely that the code makes assumptions about the HPC cluster that will cause it to break on a different cluster. If it's experiment control / data-acquisition code, you'll almost certainly need the exact same peripherals for the program to do anything at all sensible.

I see a lot of people here on HN vastly over-estimating the value of bit-for-bit reproducibility of one implementation, and vastly underestimating the value of having a diversity of implementations to test an idea.

Same with grammatical or spelling errors. I don't review research but I do review resumes, and I've seen atrocious spelling on resumes. Here's the candidate's first chance to make an impression. They have all the time in the world to proofread, hone, and have other eyes edit it. Yet, they still miss obvious mistakes. If hired, will their work product also be sloppy?

If we're supposed to accept scientific results on faith, why bother with science at all?

If code is what is substantiating a scientific claim, then code needs to stand up to scientific scrutiny. This is how science is done.

I came from physics, but systems and computer engineering was always an interest of mine, even before physics, I thought it was kooky-dooks that CS people can release papers w/o code, fine if the paper contains all the proofs but otherwise it shouldn't even be looked at. PoS (proof-of-science) or GTFO.

We are the point in human and scientific civilization that knowledge needs to prove itself correct. Papers should be self contained execution environments that generate PDFs and resulting datasets. The code doesn't need to be pretty, or robust, but it needs to be sealed inside of a container so that it can be re-run, re-validated and someone else can confirm the result X years from now. And it isn't about trusting or not trusting the researcher, we need to fundamentally trust the results.

I doubt any physicist believes we need to get the Tevatron running again just to check decade-old measurements of the top quark. I don't understand why decade-old scientific software code must meet that bar.

Anyways, in physics they always require several experimental proves for our theory. They also have several "software experiments" for e.g. predicting the same observables. Therefore, researchers need to be able to compile and run the code of their competitors in order to compare and verify the results in detail. In this place, bug-hunting/fixing is sometimes also taking place - of course. So applying the articles suggestions would have the potential to accelerate scientific collaboration.

btw; I know some people who do still work with the data taken at the LEP experiment which was shut down almost 20 (!) years ago and they have a hard time in combining old detector-simulations, monte-carlos etc. with new data-analysis techniques for the exact same reasons mentioned in the article. For large-scale experiments it is a serious problem which nowadays has much more attention than at LEP ages, since LHC has anyways obvious big-data problems to solve before their next upgrade, including also software-solutions.

I suspect that software really is different in this way, and treating it like it's complex, one off hardware is cultural inertia that's going to fade away.

A container can be a .tar.gz, a zip or a disk image of artifacts, code, data and downstream deps. The generic word has been co-opted to mean a specific thing which is very unfortunate.

But the "implementation detail" of which container format you use, and whether it will still be supported in 10 years, is not an implementation detail at all -- since this will determine whether containerization actually solves the problem of helping your code run a decade later. This gets worse as the number, complexity and of container formats expands.

Of course if what you mean is that researchers should provide perpetual maintenance for their older code packages, moving them from one obsolete platform to a more recent one, then you're making a totally different and very expensive suggestion.

The first step is just having your code, data and deps in an archive. Depending on the project and the age, more stuff makes it into the archive. I have been on projects where the source to the compiler toolchain was checked into the source repo and the first step was to boostrap the tooling (from a compiler binary checked into the repo).

We aren't even to the .tar.gz stage yet.

Specifically, to that point, I want to cite the saying:

"The dogs bark, but the caravan passes."

(There is a more colorful German variant which is, translated: "What does it bother the mighty old oak tree if a dog takes a piss...").

Of course, if you publish your code, you expose it to critics. Some of this will be unqualified. And as we have seen in the case e.g. of climate scientists, some might be even nasty. But who cares? What matters is open discussion which is a core value of science.

It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

In science, code is not an end in-and-of-itself. It is a tool for simulation, data reduction, calculation, etc. It is a way to test scientific ideas.

> how do you expect anyone with the right expertise to assess your findings

I would expect other experts in the field to write their own implementation of the scientific ideas expressed in a paper. If the idea has any merit, their implementations should produce similar results. Which is exactly what they would do if it were a physical experiment.

If I'm given bad information and I act on that information, then problems can occur.

Similarly, if the software is giving the scientist bad information, problems can occur.

How many more stories do we have to read about some research getting published in a journal only to have to retract it down the road because they had a bug in the software before we start asking if maybe there needs to be more rigor in the software portion of the research as well?

There was a story on HN a while back about a professor who had written software, had come to some conclusions, and even had a Ph.D. student working on research based on that work. Only to find out that a software flaw meant the conclusions weren't useful to anyone and that student ended up wasting years of their life.

---

This stuff matters. This isn't a model of reality, it's an exploration of reality. It would be like telling a hiker that terrain doesn't matter. They would, rightfully, disagree with you.

We will always hear stories like that, as we will always hear stories about major bugs in stable software releases. Asking a scientist to do better than whole teams of software engineers makes little sense to me.

Of course, a bug that was introduced or kept with the counscious intention of fooling the reviewers and the readers is another story.

This is not what is being asked, shame on you for the strawman.

Your entire post can be summed up with the following sentence: "if we can't be perfect then we may as well not try to be better".

The thing is that it has little to do about rigor -- or if I may sin again, it is equivalent to say that software developers lack rigor: sure, some of them do (as some scientists do), but even among the most significant and severe bugs of the history of software, it is seldom the case that we can tell "right, definitely the guy who wrote that lacked rigor and seriousness".

Of course this is not a blank forgiveness for every bad scientist out there. Of course we should aim at getting better. But we should make the difference between the ideal science process and the science as performed by a human, prone to errors, misunderstandings and mistakes, and realize that these things will always happen, however many times we call for "more rigor because bugs have consequences".

And stop comparing scientists to software developers, it's a hidden argument by authority, and it isn't needed.

How this "rigor" you are calling for should manifest, then? Put bluntly, my point was that every software has bug, so how "more rigor" would help? What should we do, what should we ask for in _practical_ terms?

Also, please do not rephrase this last sentence as "oh so since every software has bugs, then you obviously say that we shouldn't fix bugs, anyway other bugs will remain!".

That's exactly what I'm going to do. Point out that we can demand better even in the face of a lack of perfection.

There are two problems here with your stance.

1. The assumption that all bugs are created equal, and 2. The assumption that the truth isn't the overriding concern of science.

It's real easy to define the set of bugs that are unacceptable in science. Any bug that would render the results inaccurate is unacceptable.

The fact that some jackass web developer wrote a bug that deleted an entire database in no way obviates that responsibility of the scientists.

I think that, while we could use a bit more training in software engineering best-practices in the science, the thesis is still that science is hard and we need real replication of everything before reaching important conclusions, and over-focusing on one specific type of errors isn't all that helpful.

It's not clear to me why you think I would argue that inaccuracies should be avoided in software but accept that they're ok for electrical systems.

And if you're a map maker, it's a bit rich to start claiming that the accuracy of your maps is unimportant. If code is "a way to test scientific ideas", then it kinda needs to work if you want meaningful results. Would you run an experiment with thermometers that were accurate to +-30° and reactants from a source known for contamination?

Of course, it is a difference whether you make a clinical study on drugs, and use a pocket calculator to compute a mean, or whether you research in numerical analysis, or are presenting a paper in how to use Coq to more efficiently prove the four-color theorem or Fermat's last theorem.

In short, much of science is not computer science, and for it, computation is just a tool.

Sure, some people might point out spurious bugs and "design issues" or whatever, boo hoo. But others might actually find flaws in the code that meaningfully affect science itself: true bugs.

Sure, they could do this by doing a full replication in a lab and then custom coding everything from scratch. But even then, all you have is two conflicting results, with no good way yet to determine which one is more right or why they disagree. Technically, you can use the scientific progress to eventually find bugs in the scientific process, but why waste so much time when publishing the code will allow for reviews to find bugs so much faster. Its pure benefit to science to not obscure its proofs and rigor.

How many actually try to reproduce the results by writing corresponding code themselves? Apparently lot of papers with slightly wrong findings because code errors have passed the peer review (all of us in the SWE bubble know how often bugs occur), at least in less prestigious journals.

There is nothing wrong with mandating the code to be supplied with the paper. Because, many time code is somewhere between the experimental setup and proof / result.

I myself had very bad experience with extending the undocumented Fortran 77 code (lots of gotos and common blocks) of my supervisor. Finally, I decided to rewrite the whole thing including my new results instead of just somehow embedding my results into the old code for two reasons: (1) I'm presumably faster in rewriting the whole thing including my new research rather than struggling with the old code and (2) I simply would not trust in the numerical results/phenomenology produced by the code. After all, I'm wasting 2 months of my PhD for the marriage of my own results with known results which -in principle- could have been done within one day if the code base would allow for it.

So yes, If it's a one-man-show I would not give too much on code quality (though unit tests and git can safe quite a lot of time during development) but if there is a chance that someone else is going to touch the code in near future it will save time to your colleagues and improve the overall (scientific) productivity.

PS: quite excited about my first post here

This makes me a little uneasy, as I'm not too worried about code quality can easily translate into Yes I know my code is full of undefined behaviour, and I don't care.

> PS: quite excited about my first post here

Welcome to HN! reddit has more cats, Slashdot has more jokes about sharks and laserbeams, but somehow we get by.

The latter isn't great practice, but if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

'Undefined behaviour' is a term-of-art in C/C++ programming, there's no ambiguity.

> if your environment handles behavior deterministically, and you publish the version of the compiler you're using, it doesn't seem to be a problem for this type of code.

Code should be correct by construction, not correct by coincidence. Results from such code shouldn't be considered publishable. Mathematicians don't get credit for invalid proofs that happen to reach a conclusion which is correct.

Again, this isn't some theoretical quibble. There are plenty of sneaky ways undefined behaviour can manifest and cause trouble. [0][1][2]

In the domain of safety-critical software development in C, extreme measures are taken to ensure the absence of undefined behaviour. If scientists adopt a sloppier attitude toward code quality, they should expect to end up publishing invalid results. Frankly, this isn't news, and I'm surprised the standards seem to be so low.

Also, of all the languages out there, C and C++ are among the most unforgiving of minor bugs, and are a bad choice of language for writing poor-quality code. Ada and Java, for instance, won't give you undefined behaviour for writing int i; int j = i;.

[0] https://devblogs.microsoft.com/oldnewthing/20140627-00/?p=63...

[1] https://blog.regehr.org/archives/213

[2] https://cryptoservices.github.io/fde/2018/11/30/undefined-be...

See also my longer ramble on this topic at https://news.ycombinator.com/item?id=24264376

Let the scientists publish UB code, and even the artifacts produced, the executables. Then, if such problems are found in the code by professionals, they can investigate it fully and find if it leads to a tangible flaw that invalidates the research or not.

You would drive yourself mad pointing out places in math proofs where some steps, even seemingly important ones, were skipped. But the papers are not retracted unless such a gap actually holds a flaw that invalidates the rest of thr proof.

Let thdm publish their gross, awful, and even buggy code. Sometimes the bugs don't effect the outcomes.

Granted, it's not a guarantee that the results are wrong, but it's a serious issue with the experiment. I agree it wouldn't generally make sense to retract a publication unless it can be determined that the results are invalid. It should be possible to independently investigate this, if the source-code and input data are published, as they should be.

(It isn't universally true that reproduction of the experiment should be practical given that the source and data are published, as it may be difficult to reproduce supercomputer-powered experiments. iirc, training AlphaGo cost several million dollars of compute time, for instance.)

> this mindset is what keeps people from publishing the code in the first place

As I explained in [0], this attitude makes no sense at all. It has no place in modern science, and it's unfortunate the publication norms haven't caught up.

Scientific publication is meant to enable critical independent review of work, not to shield scientists from criticism from their peers, which is the exact opposite.

> Let the scientists publish UB code, and even the artifacts produced, the executables. Then, if such problems are found in the code by professionals, they can investigate it fully and find if it leads to a tangible flaw that invalidates the research or not.

I'm not sure what to make of 'professionals', but otherwise I agree, go ahead and publish the binaries too, as much as applicable. Could be a valuable addition. (In some cases it might not be possible/practical to publish machine-code binaries, such as when working with GPUs, or Java. These platforms tend to be JIT based, and hostile to dumping and restoring exact binaries.)

I agree with your final two paragraphs.

[0] https://news.ycombinator.com/item?id=24264376

Glad we agree, if you're aware of how your compiler handles these things, you can construct it to be correct in this way.

It won't be portable at all (even to the next patch version of the compiler), I would never let it pass a code review, but that doesn't sound like an issue that's relevant here.

I presume we agree but I'll do my usual rant against UB: Deliberately introducing undefined behaviour into your code is playing with fire, and trying to outsmart the compiler is generally a bad idea. Unless the compiler documentation officially commits to a certain behaviour (rollover arithmetic for signed types, say), then you should take steps to avoid undefined behaviour. Otherwise, you're just going with guesswork, and if the compiler generates insane code, the standards documents define it to be your fault.

It might be reasonable to make carefully disciplined and justified exceptions, but that should be done very cautiously. JIT relies on undefined behaviour, for instance, as ultimately you're treating an array as a function pointer.

> It won't be portable at all (even to the next patch version of the compiler)

Right, doing this kind of thing is extremely fragile. Does it ever crop up in real-life? I've never had cause to rely on this kind of thing.

It would be possible to use a static assertion to ensure my code only compiles on the desired compiler, preventing unpleasant surprises elsewhere, but I've never seen a situation where it's helpful.

This isn't the same thing as relying on 'ordinary' compiler-specific functionality, such as GCC's fixed-point functionality. Such code will simply refuse to compile on other compilers.

> I would never let it pass a code review, but that doesn't sound like an issue that's relevant here.

Disagree. It should be possible to independently reproduce the experiment. Robust code helps with this. Code shouldn't depend on an exact compiler version, there's no good reason code should.

Sounds like it is quite good science to do that, because it puts the computation on a pair of independent feet.

Otherwise, it could just be that the code you are using as a bug and nobody notes until it is too late.

The cross-checking is anyways good scientific practise, not only because of bugs in the code (that's actually a sub-leading problem imho), but because of the degree of difficulty of the problems and the complexity of their solutions (and their reproducibility). In that sense, cross-checking should discover both, scientific "bugs" and programming-bugs. The "debugging" is partly also done at the community level - at least in our field of research.

However, it is also a matter of efficiency. I -and many others too- need to re-implement not because of bug-hunting/cross-checking but simply because we do not understand the "ugly" code of our colleagues and instead of taking the risk to break existing code we simply write new one which is extremely inefficient (others may take the risk and then waste months on debugging and reverse-engineering which is also inefficient). So my point on writing "good code" is not so much about avoiding bugs but about being kind to you colleagues, saving them nerves and time (which they can then spend on actual science) and thus also saving taxpayers money...

Yes, that should be this way.

Also all cases where some company research team goes to a scientific conference and presents a nifty solution for problem X without telling how it was purportedly done, it should be absolutely required to publish code and data for this.

*And that's also something which is broken about software patents - patents are about open knowledge, software which uses such patents is not open - this combination should not be allowed at all).

I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.

In-code selection statements like `if( True ) {...}`, where you have no idea what is being selected or why.

Code that only runs in the particular workspace image that contains some function that was hacked out to make things work during a debugging session 5 years ago.

Distributed projects where one person wrote the preprocessor, another wrote the simulation software, and a third wrote the analysis scripts, and they all share undocumented assumptions worked out between the three researchers over the course of two years.

Depending on implementation-defined behavior (like zeroing out of data structures).

Function and variable names, like `doit()` and `hold`, which make it hard to understand the intention.

Files that contain thousands of lines of imperative instructions with documentation like "Per researcher X" every 100 lines or so.

Code that runs fine for 6 hours, then stops because some command-line input had the wrong value.

I've seen all of these over the years. Even as a domain expert who has spoken directly with authors and project leads, this kind of stuff makes it very hard to tease out what the code actually does, and how the code corresponds to the papers written about the results.

As an admin it was quite frustrating, but I understand it sometimes when you know the person/project isn’t tested in a distributed environment. But when it’s the projects that do know how they’re used and still do those things...

Code exhibiting undefined behavior is a different kettle of fish...

How is "your code has error and I rebuke you" a more painful critique than "you are hiding your methodology and so I rebuke you"?

Here's a quick pass, I'm sure I'm missing stuff, but I've needed to code review a lot of science and production output and below is how I tend to think of it, especially taking efficiency of engineer/scientist time into account.

Production Ready?

* code well factored for extensibility, feature change, and multi-engineer contribution

* robust against hostile user input

* unit and integration tested

Science Ready?

* code well factored for readability and reproducibility (e.g. random numbers seeded, time calcs not set against 'now')

* robust against expected user input

* input data available? testing optional but desired, esp unit tests of algorithmic functions

* input data not available? a schema-correct facsimile of input data available in a unit test context to verify algorithms correct

Both?

* security needs assessed and met (science code might be dealing with highly secure data, as might production code)

* performance and stability needs met (production code more often requires long term stability, science sometimes needs performance within expected Big O to save compute time if it's a big calculation)

The key aspect of that code is that it's going to be run once or twice, ever, and it's only ever going to be run on a particular known set of input data. It's a tool (though complex) that we used (once) to get from A to B. It does not need to get refactored, because the expectation is that it's only ever going to be used as-is (as it was used once, and will be used only for reproducing results), it's not intended to be built upon or maintained. It's not the basis of the research, it's not the point of research, it's not a deliverable in that research, it's just a scaffold that was temporarily neccessary to do some task - one which might have been done manually earlier through great effort, but that's automated now. It's expected that the vast majority of the readers of that paper won't ever need to touch that code, they care only about the results and a few key aspects of the methodology, which are (or should be) all mentioned in the paper.

It should be reproducible to ensure that we (or someone else) can obtain the same B from A in future, but that's it, it does not need to be robust to input that's not in the input datafile - noone in the world has another set of real data that could/should be processed with that code. If after a few years we or someone else will obtain another dataset, then (after those few years, if that dataset happens) there would be a need to ensure that it works on that dataset before writing a paper about that dataset, but it's overwhelmingly likely that you'd want to modify that code anyway both because that new dataset would not be 'compatible' (because the code will be tightly coupled to all the assumptions in the methodology you used to get that data, and because it's likely to be richer in ways you can't predict right now) and you'd want to extend the analysis in some way.

It should have a 'toy example' - what you call 'a schema-correct facsimile of input data' that's used for testing and validation before you run it on the actual dataset, and it should have test scenarios and/or unit tests that are preferably manually verifiable for correctness.

But the key thing here is that no matter what you do, that's still in most cases going to be "write once, run once, read never" code, as long as we're talking about the auxiliary code that supports some experimental conclusions, not the "here's a slightly better method for doing the same thing" CS papers. We are striving for reproducible code, but actual reproductions are quite rare, the incentives are just not there. We publish the code as a matter of principle, knowing all well that most likely noone will download and read it. The community needs the possibility for reproduction for the cases where the results are suspect (which is the main scenario where someone is likely to attempt reproducing that code), it's there to ensure that if we later suspect that the code is flawed in a way where the flaws affect the conclusions then we can go back to the code and review it - which is plausible, but not that likely. Also, if someone does not trust our code, they can (and possibly should) simply ignore it and perform a 'from scratch' analysis of the data based what's said in the paper. With a reimplementation, some nuances in the results might be slightly different, but all the conclusions in the paper should still be valid, if the paper is actually meaningful - if a reimplementation breaks the conclusions, that would be a successful, valuable non-reproduction of the results.

This is a big change from industry practice where you have mantras like "a line of code is written once but read ten times", in a scientific environment that ratio is the other way around, so the tradeoffs are different - it's not worth investing refactoring time to improve readability, if it's expected that most likely noone will ever read that code; it makes sense to spend that effort only if and when you need it.

But I seem to get the vibe that this original Nature article is mostly about the auxiliary data analysis code for "non-simulated" experiments, while Hacker News seems biased towards fields like computer science, machine learning, etc.

In the first case, you must take into account all (un)imaginable corner cases and never allow the code to fail or hang up. In the second case it needs to produce a reproducible result at least for the published case. And do not expect it to be user-friendly at all.

Any effort to get code working for other people, or documented in any way would probably be seen as wasted effort that could be used to write more papers or create more results to create new papers.

This kind of reasoning was one of the many reasons I left academic research - I personally didn't value publications as deliverables.

Still, there's plenty of room to encourage good(/better) practices which cost essentially nothing, e.g. using $PWD rather than /home/bob/foo

Any effort to write a paper readable for other people, or document the experiment in any way would probably be seen as wasted effort that could be used to create more results.

The "don't show your work" argument only makes sense if you are doing PR, not science.

The kind of code I'll ship for production will include unit testing designed around edge or degenerate cases that arose from case analysis, usually some kind of end to end integration test, aggressive linting and crashing on warnings, and enforcing of style guidelines with auto formatting tools. The last one is more important than people give it credit for.

For research it would probably be sufficient to test that the code compiles and given a set of known valid input the program terminates successfully.

In general, scientists don't care how long it takes or how many resources the code uses. It is not a big deal to run a script for an extra hour, or use up a node of supercomputer. Extravagent solutions or added packages to make the code run smoother or faster is only wasting time. It speed/elegance only really matters when you know the code is going to be distributed to the community.

Basically scientists only care if the result, is true. If the result it outputs is sensible, defensible, reliable, reproducible. It would be considered a dick move to criticism someones code, if the code was proven to produce the correct result.

Formal proof is much much harder than making code understandable and reviewable. It can be done but it is not easy, and can yield surprising results:

https://en.wikipedia.org/wiki/CompCert

http://envisage-project.eu/proving-android-java-and-python-s...

If not by unit tests, code review or formal logic, then what?

You can then use unit testing for low-level infrastructure (e.g. checking that your ODE solver works as expected), but do the high-level testing via scientific validation. The first line of defense is to check that you don’t break any laws of physics, e.g. that energy and electric charge is conserved in your end results. Even small implementation mistakes can violate these.

Then you search for related existing publications of a theoretical or numerical nature, trying to reproduce their results; the more existing research your code can reproduce, the more certain you can be that it is at least consistent with known science. If this fails, you have something to guide your debugging; or if you’re very lucky, something interesting to write a paper about :).

The final validation step is of course to validate against experiments. This is not suited for debugging though, since you can’t easily say whether a mismatch is due to a software bug, experimental noise, neglected effects in the mathematical model, etc.

Cross referencing independent experiments and external datasets.

Science doesn't work like software. The code can be perfect and still not give results that reflect reality. The code can be logical and not reflect reality. Most scientists I know go in with the expectation that "the code is wrong" and its results must be validated by at least one other source.

Part of our process involves cleaning up code to a higher state of refinement as it gets closer to entering the production pipeline.

I've tested 30 year old code, and it still runs, though I had to dig up a copy of Turbo Pascal, and much of it no longer exists in computer readable form but would have to be re-entered by hand. Life was actually simpler back then -- with the exception of the built-ins of Turbo Pascal, it has no dependencies.

My code was in fact adopted by two other research groups with only minor changes needed to suit slightly different experimental conditions. It contained many cross-checks, though we were unaware of modern software testing concepts at the time.

For a result to have broader or lasting impact, replication is not enough. The result has to fit into a broader web of results that reinforce one another and are extended or turned into something useful. That's the point where precise replication of minor supporting results becomes less important. The quality of any specific experiment done in support of modern electromagnetic theory would probably give you the heebie jeebies, but the overall theory is profoundly robust.

The same thing has to happen when going from prototype to production. Also, production requires what I call push-button replication. It has to replicate itself at the click of a mouse, because the production team doesn't have domain experts who can even critique the entirety of their own code, and maintaining their code would be nearly impossible if it didn't adhere to standards that make it maintainable by multiple people at once.

My own code was quite clean in the 1980s, when the limitations on the machines themselves tended to keep things fairly compact with minimal dependencies. And I learned a decent "structured programming" discipline.

As I moved into more modern languages, my code kind of degenerated into a giant hairball of dependencies and abstractions. "Just because you can do that, doesn't mean you should." I've kind of learned that the commercial programmers limit themselves to a few familiar patterns, and if you try to create a new pattern for every problem, your code will be hard to hand off.

Scientists would benefit from receiving some training in good programming hygiene.

In the common case, the software behaves as expected (random output), but it is reproducible for tests. You can then publish your RNG seed with the commit hash when you release your code/paper, and others may see your results and investigate that particular code execution.

Code belongs with the paper. Otherwise we can just continue to make up numbers and pretend we found something significant.

I've spent the intervening years trying to find a way to implement this myself, going as far as to buy things like the ice40 fpga because the bitstreams are supposedly unlocked; this is a pre-req for modifying the actual gate/logic on the chip.

I've emailed the professor listed as the headliner in the articles published about it to no avail.

Nearly my entire adult life has been spent reading some interesting article, chasing down the paper, finding out if any code was published, and seeing if I could run the code myself.

It wasn't until machine learning with pytorch became mainstream that I started having luck replicating results. Just some more data points for this discussion.

In what way do idiots making idiotic comments about your correct code invalidate your scientific production? You can still turn out science and let people read and comment freely on it.

> As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result.

I guess you would not need to engage personally with the idiots at "acceleratorskeptics.com", but likely most of their critique would be easily shut off by a simple sentence such as this one. Since most of your readers would not be idiots, they could scrutinize your code and even provide that reply on your behalf. This is called the scientific method.

I agree that you produce science, not merely code. Yet, the code is part of the science and you are not really publishing anything if you hide that part. Criticizing scientific code because it is bad software engineering is like criticizing it because it uses bad typography. You should not feel attacked by that.

How would a layperson identify a faulty critique? It would be picked up by the media who would do their usual “both sides” thing.

An educated and motivated layperson at least would have the chance to learn whether the critique is faulty. Today, with secret code, it is impossible to verify for almost everybody.

However, a methods section is always under-specified. Code provides the unique opportunity to actually see the full methods on display and properly review their work. It should be mandated by all reputable journals and worked into the peer review process.

But that's exactly the problem.

Are you familiar with that bug in early Civ games where an overflow was making Ghandi nuke the crap out of everyone? What if your code has a similar issue?

What if you have a random value right smack in the middle of your calculations and you just happened to be lucky when you run your code?

I'm not that familiar with Monte Carlo, my understanding is that this is just a way to sample the data. And I won't be testing your data sampling, but I will expect that given the same data to your calculations part (eg, after the sampling happens), I get exactly the same results every time I run the code and on any computer. And if there are differences I expect you to be able to explain why they don't matter, which will show you were aware of the differences in the first place and you were not just lucky.

And then there is the matter of magic values that plaster research code.

Researchers should understand that the rules for "software engineering grade code" are not there just because we want to complicate things, but because we want to make sure the code is correct and does what we expect it to do.

/edit: The real problem is not getting good results with faulty code, is ignoring good solutions because faulty code.

If the proof on which the paper is based is in the code that produced the evidence, you absolutely need to be able to let a lambda user run it without specific knowledge to abide to the reproducible principle. Asking a reviewer to fiddle about like a IT professional to get something working is bound to promote lazy reviewing and either will result into dismissing the result or approval without real review.

And by the way producing a paper could be argued it isn't really science either, but if you are working with MSFT Office, you know there is a fair amount of non science work hours that has been put into that as well.

Not so fast. Monte Carlo code turns arbitrary RNG seeds into outputs. That process can, and arguably should be, deterministic.

To do your study, you feed your Monte Carlo code 'random enough' seeds. Coming up with the seeds does not need to be deterministic. But once the seeds are fixed, the rest can be deterministic. Your paper should probably also publish the seeds used, so that people can reproduce everything. (And so they can check whether your seeds are carefully chosen, or really produce typical outcomes.)

Sure, and that rationale works OK when your code operates in a limited, specialized domain.

But if you're modeling climate change or infectious diseases, and you expect your work to affect millions of human lives and trillions of dollars in spending, then you owe us a full accounting of it.

There are a lot of domains which do have a deep impact on society but are completely underfunded. For example research on ecology and declining insect populations. Or research on education. And domains like epidemiology, climate change research, cancer research or such are not known to earn better salaries to scientists as on average. Most scientists earn a pity.

> But if you're modeling climate change or infectious diseases, and you expect your work to affect millions of human lives and trillions of dollars in spending, then you owe us a full accounting of it.

What one can expect from scientists, in whatever subject they work, is honesty, integrity, a full account of their findings. What you can't expect is that they just turn into expert software engineers or make their working code beautiful. You can't expect them to work for free. What the academic system demands from them is that they work on their next paper instead, so if you want pretty code, you need at least in part to change the system.

Two nitpicks: a) it shouldn't change the conclusions, but MC calculations will get different results depending on the seed. and b) it is considered good practice in reproducible science to fix the seed so that the results of subsequent runs give exactly the same results.

Ultimately, I think there is a balance: really poor code can lead to incorrect conclusions... but you don't need production ready code for scientific exploration.

Anybody who has conducted experimental research will say they spent 80% of the time using a hammer or a spanner. Repairing faulty lasers or power supplies. This process of reliable and repeatable experimentation is the basis of science itself.

Computational experiments must be held to the same standards as physical experiments. They must be reproducible and they should be publicly available (if publicly funded).

Sounds like I should just become a scientist then.

Do you guys write unit tests or is that beneath you too?

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points.

The person which wrote the linked blog post writes that it was a software engineer at google. Unfortunately, that claim is not falsifiable as the person decided to remain anonymous.

> As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result.

The claim is that even with the same random seed for the random generator, the program produces different results, and this is explained by the allegation that it runs non-deterministic (in the sense of undefined behavior) in multiple threads. It claims also that it produces significantly different results depending on which output file format is chosen.

If this is true, the code would have race conditions, and as being impacted by race conditions is a form of undefined behavior, this would make any result of the program questionable, as the program would not be well-defined.

Personally, I am very doubtful whether this is true, this would be incredibly sloppy by the imperial college scientists. Some more careful analysis by a recognized programmer might be warranted.

However it underlines well the importance of the main topic that scientific code should be open to analysis.

> What I'm saying is that scientific code doesn't need to handle every special case or be easily usable by non-experts.

Fully agree with this. But it should try to document its limitations.

That’s not at all what that means. What are you talking about? As long as a Monte Carlo process works towards the same result it’s equivalent.

You’re speaking genuine nonsense as far as I’m concerned. Randomness doesn’t imply non deterministic. Non-determinitism in no way implies race conditions or undefined behavior. We care that the random process reaches the same result, not that the exact sequence of steps is the same.

This is what scientists are talking about. A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet.

https://lockdownsceptics.org/code-review-of-fergusons-model/

It says, word-by-word:

> Clearly, the documentation wants us to think that, given a starting seed, the model will always produce the same results.

>

>Investigation reveals the truth: the code produces critically different results, even for identical starting seeds and parameters.

> I’ll illustrate with a few bugs. In issue 116 a UK “red team” at Edinburgh University reports that they tried to use a mode that stores data tables in a more efficient format for faster loading, and discovered – to their surprise – that the resulting predictions varied by around 80,000 deaths after 80 days: ...

The bugs which the blog post implies here are such ones as described by Jens Regehr: https://blog.regehr.org/archives/213

Not that I do not endorse these statements in the blog - I am rather skeptical whether they are true at all.

What the authors of the blob post mean is clearly "undefined behaviour" in the sense of non-deterministic program execution of a program that is not well-formed. It is clear that many non-experts could confuse that with the pseudo-randomness implicit in Monte-Carlo simulations, but this is a very different thing. The first is basically a broken, invalid, and untrustworthy program. The second is the established method to produce a computational result by introducing stochastic behavior, which is for example how modern weather models work.

These are wildly different things. I do not understand why your comment just adds to the confusion between these two things??

> A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet.

As said, I don't endorse the critique in the blog. However, critique in a software implementation, as well as in scientific matters, should never carry a call on authority - it should logically explain what is the problem, with concrete points. Unfortunately, the cited blog post remains very vague about this, while claiming:

> My background. I have been writing software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company’s database product, amongst other jobs and projects. I was also an independent consultant for a couple of years.

It would be much better if, instead claiming that there could be race conditions, it could point to lines in the code with actual race conditions, and show how the results of the simulation are different when the race conditions are fixed. Otherwise, it just looks like he claims that the program is buggy, because he is in no position to question the science, and does not like the result.

Non-determinism can be caused by

a) random seeds derived from hardware, such as seek times in a HDD controller, which is fed into pseudo random number (PRNG) generation. This is not a problem. For debugging, or comparison, it can make sense to switch it off, though.

b) data race conditions, which is a form of undefined behavior. This not only can dramatically change results of a program run, but also invalidates the program logic, in languages such as C and C++. This is what he blog post in "lockdownskeptics.org" suggests. For the application area and its consequences, this would be a major nightmare.

c) What I had forgotten is that parallel execution (for example in LAM/MPI, map/reduce or similar frameworks) is inherently non-deterministic and, in combination with properties of floating-point computation, can yield different but valid results.

Here an example:

A computation is carried out on five nodes and they return the values 1e10, 1e10, 1e-20, -1e10, -1e10, in random order. The final result is computed by summing these up.

Now, the order of computation could be:

((((1e10 + 1e10) + 1e-20) + -1e10) + -1e10)

or it could be:

(((1e10 + -1e10) + 1e-20) + (+1e10 + -1e10))

In the first case, the result would be zero, in the second case, 1e-20, because of the finite length of floating point representation.

_However_... if the numerical model or simulation or whatever is stable, this should not lead to a dramatic qualitative difference in the result (otherwise, we have a stability problem with the model).

Finally, I want to cite one last paragraph from the post on lockdownskeptics.org:

> Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.

> On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.

You are right with the distinction, I had data race conditions in mind.

Race conditions can well happen in a correct C/C++ multi-threaded program in the sense that the order of specific computation steps is sometimes random. And for operations such as floating-point addition, where order of operations does matter, the exact result can be random as a consequence. But the end result should not depend dramatically on it (which is what the poster at lockdownskeptics.org claims).

GPT-3 FTW!

> you understand that the links in your post are the exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study.

It's profoundly unscientific to suggest that researchers should be given the choice to withhold details of their experiments that they fear will not withstand peer review. That's much of the point of scientific publication.

Researchers who are too ashamed of their code to submit it for publication, should be denied the opportunity to publish. If that's the state of their code, their results aren't publishable. Unpublishable garbage in, unpublishable garbage out. Simple enough. Journals just shouldn't permit that kind of sloppiness. Neither should scientists be permitted to take steps to artificially make it difficult to reproduce (in some weak sense) an experiment. (Independently re-running code whose correctness is suspect, obviously isn't as good as comparing against a fully independent reimplementation, but it still counts for something.)

If a mathematician tried to publish the conclusion of a proof but refused to show the derivation, they'd be laughed out of the room. Why should we hold software-based experiments to such a pitifully low standard by comparison?

It's not as if this is a minor problem. Software bugs really can result in incorrect figures being published. In the case of C and C++ code in particular, a seemingly minor issue can result in undefined behaviour, meaning the output of the program is entirely unconstrained, with no assurance that the output will resemble what the programmer expects. This isn't just theoretical. Bizarre behaviour really can happen on modern systems, when undefined behaviour is present.

A computer scientist once told me a story of some students he was supervising. The students had built some kind of physics simulation engine. They seemed pretty confident in its correctness, but in truth it hadn't been given any kind of proper testing, it merely looked about right to them. The supervisor had a suggestion: Rotate the simulated world by 19 degrees about the Y axis, run the simulation again, and compare the results. They did so. Their program showed totally different results. Oh dear.

Needless to say, not all scientific code can so easily be shown to be incorrect. All the more reason to subject it to peer review.

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points.

Why would you care? Science is about advancing the frontier of knowledge, not about avoiding invalid criticism from online communities of unqualified fools.

I sincerely hope vaccine researchers don't make publication decisions based on this sort of fear.

How exactly is this a bad thing?

> I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. I'm here to turn out science, not production ready code.

But it should be noted that what you didn't say is that you're here to turn out accurate science.

This is the software version of statistics. Imagine if someone took a random sampling of people at a Trump rally and then claimed that "98% of Americans are voting for Trump". And now imagine someone else points out that the sample is biased and therefore the conclusion is flawed, and the response was "Hey, I'm just here to do statistics".

---

Do you see the problem now? The poster above you pointed out that the conclusions of the software can't be trusted, not that the coding style was ugly. Most developers would be more than willing to say "the code is ugly, but it's accurate". What we don't want is to hear "the conclusions can't be trusted and 100 people have spent 10+ years working from those unreliable conclusions".

This is exactly the sort of pedantic cluelessness that scientists are seeking to avoid by not publishing their code.

To paraphrase what the other developer said: "I don't want my work to be checked, I'm not here for accuracy, just the act of doing science".

When I was young, the ability to invalidate was the core aspect of science, but apparently that's changed over the years.