Your tools affect the quality of your work. Work with shitty tools, get shitty results.
I'll start believing your argument when scientists actually start publishing the code to go with their papers and make it reproducible.
Right now I'm nearly done a PhD in mechanical engineering. I've worked on several computational projects of varying levels of code quality from good (e.g., NIST's Fire Dynamics Simulator: https://github.com/firemodels/fds) to bad (won't give unambiguous examples for obvious reasons...).
I see clear incentives against uncertainty quantification and high quality model validation in my field. Those topics are treated superficially if at all, because people just want their model to look good, and if they do a rigorous job, that increases the chance that their model looks bad. As far as I'm concerned, if you don't do good quality model validation, you're not doing your job as a scientist.
A more specific example: Some codes are kept for a decade or longer if the software is sufficiently complex. One code I worked with was used by multiple graduate students over several PhDs and had very little if anything in terms of documentation. What did exist was far out of date. I spent a fair amount of time trying to understand and document the software, but other people treated me as if I was crazy for wanting to understand the black box I was provided. I don't think I ever actually understood that software fully despite the many hours I spent working on it. And I'm not certain that it provided correct results either. There were almost no tests, and the existing tests were hard to run and were consequently almost never run. The software produced several PhDs, though. The first few PhDs had it fairly easy. I had to pay their technical debt, and I couldn't pay it all back. It made me look bad as far as I'm concerned and did not save time in the long run. If anything, there's a long term incentive toward good software development practices.
Ultimately I think the solution would involve having minimum standard for scientific software quality, or else grant funding will be withdrawn. It will anger many people who have so far gotten away with writing low quality software, but beyond that I don't see many downsides.
I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them. But unlike you and a lot of other posters here on HN I don't think the reason for insufficient validation is that we scientists are too stupid or too lazy. On the contrary, we would love to do more of it. But there is next to no funding to do it.
> I have actually written papers that try do to a bit of that and it was annoyingly hard to publish them.
I don't know what field you work in, but in my field (fluid dynamics) these papers are not uncommon, though often poorly done. There's a relatively new ASME journal on the subject too: http://verification.asmedigitalcollection.asme.org/journal.a...
> On the contrary, we would love to do more of it. But there is next to no funding to do it.
Perhaps this is the crux of the disagreement then. I don't need extra funding to do what I should be doing in the first place. It can be a big time sink in some cases (e.g., I probably spent 6+ months compiling existing data during my PhD), but often is not. It will probably make your model look worse than you'd like, but if everyone did it then science would progress faster.
Regarding the funding: you kind DO need that extra funding. because if you don't get funding for that you get out-competed by the other research who skips that step and the next postdoc position or grant goes to that researcher who wrote two more papers instead of testing code that ultimately didn't find any major bugs.
This seems to be a common view, but I don't think it's always true. In my own work, I've found that being careful has led me to write more papers, not less. Particularly given that I've found errors in previous work that I intend to publish or already have published. There's a delay involved (i.e., it takes time to find the problems) but I think I'm coming out ahead in total publications in the long run.
I will have to re-evaluate my thinking on this.
My knee-jerk response is to say this is wrong. I have seen too many cases where people report initial results on some elaborate model as if they explain more aspects of the data, but then the results fail to generalize, and the theory, model or implementation turns out to be flawed or fragile. It gets a publication, but the state of the art does not advance.
But I notice that some younger investigators I have worked with tend to be more careful, with things like releasing code/data openly, being serious about code/model reuse, doing more careful verification/validation, being more rigorous about test/training data. It seems to be a spectrum of better research practices rather than one thing, like “better software engineering“.
Such a more principled approach should prove itself over time, because Nature cannot be fooled.
To be clear, I don't think everyone needs to be as careful as I am. A few researchers in a particular subfield spotting errors can have large benefits for everyone. And those few may catch the vast majority of the errors. There are diminishing returns to adding more people focused on being rigorous.
The thing is: This does not mean that either of us puts a lot of effort into having nice, good, modular architecture in parts we know that we will never change. Or write a lot of code coverage in module we understand very well and can reason about using analytic calculations. Or write a lot of documentation for things that are obvious to us (be it through familiarity or whatever reason). The next researcher however might want to change exactly that part we never want to change. Or might the analytic check we always used to make sure to code was not going of the deep end is not valid for his work. And he is near certainly going to need documentation on other spots of the code (and find the documentation we wrote and needed useless, because that topic is something glaringly obvious to him).
There is usually no advantage for us to make the code nice for that researcher. And THAT is very scientific software gets the bad name from. Because we have to make the trade-off, what is useful to us (and possibly our time). If we try to make the code nice and friendly for every potential user under the sun we usually DO get outcompeted.
The thing where I seem to have a totally different view from most of HN is that I think that we make valid tradeoffs, and that undocumented function and tests that only cover 40% of the code base is a perfectly fine state of affairs, instead of being caused by scientists that are too stupif for basic software engineering.
What a lot of software people also do not get at all is that variables such as x, v and a might be perfectly descriptive variable names.
One forcing function can be if high-stakes decisions rest on the conclusions from the code. In the geosciences, you start to see some of this care for air quality measurements and for climate science. (One relevant journal here is SIAM J. UQ., https://www.siam.org/Publications/Journals/SIAM-ASA-Journal-...)
The lab where I work has spent a lot of resources on validating thermal and fluid dynamical codes for atmospheric entry of Mars landers — an engineering example rather than science. They obviously are very motivated to avoid an unpleasant surprise from a model failure.
It's very difficult to foresee what changes will be required 1-year's time, let alone 3. To that end, devs try to be extra thoughtful during the planning phase so that the end product is made up of well-behaved components. The extra time taken to come up with a sensible architecture goes a long way when business requirements eventually call for new or enhanced functionality and someone needs to understand the codebase in a reasonable amount of time. To be clear, this doesn't guarantee clean code -- bad architectural decisions happen, planning can occur without a clear requirements, no one held the quality bar during implementation, etc. -- but the outcome is always better than a laissez-faire design "strategy".
Perhaps scientific code churn is much more frequent than commercial code churn. In which case you would want to balance that by perhaps spending less time coming up with a good architecture -- and in the case of throwaway code, perhaps none at all. That being said, I would be surprised if new scientific knowledge always results in you having to discard every line and restart completely from scratch. I imagine that spending more time upfront architecting your software would instead allow you to more quickly react to new research results -- paying dividends on your time, in the long run.
https://news.ycombinator.com/item?id=18368455
This is normal. Poor sw practices lead to bugs, which lead to invalid science.
Also
https://physicstoday.scitation.org/do/10.1063/PT.6.1.2018082...
This got caught because it led to a big fight. Most other scientific software is not audited at all. Given my experience with those who write them, I tend not to trust most conjugational results coming from science.
> Given my experience with those who write them, I tend not to trust most conjugational results coming from science.
In computational fluid dynamics, there's a saying: No one believes computations aside from the person who ran them, and everyone believes experiments aside from the person who ran them.
To be fair, this is mostly because turbulence modeling can be very inaccurate, but bugs in the code are also a major concern of mine.
It was - that's where I got it from.
But yes, you're right. Perfecting things is at cross-purposes with the frenetic pace of research. Crappy untested code gets written under duress. Resist it. That needs to change, just as the whole funding model needs to reward verification more than new discovery... But back to the code, if the code is wrong, the science is likely to be as well.
I do hear this sentiment often: "Hey dudes, it's just a prototype! Why waste time making it pretty? Who cares? We're doing SCIENCE here!"
But prototypes (and beginnings in general) are precisely the time to be extra careful. Wrong turns and self-delusion are costlier, not cheaper, when you're the one paving the road for others.
In research, there are many ways to lead yourself astray, due to the inherent chaos of novelty, even without software bugs completely flipping the outcome. The idea that writing shitty code somehow "saves time" and is only worthwhile for the "17th javascript framework" has to go.
Articulating your thoughts into a sane logical structure (aka code), with sane names and motivation examples and conceptual units, saves you time even in the short run. Never mind 5 years. It also helps you avoid publishing unreliable, brittle, "SOTA" nonsense… of which there's sadly so much.
If you think he or she is only talking about pretty code, you don't understand the distinction to begin with.
Remember: This is not the 17th javascript framework, but software for problems that we don't understand going in. And often enough we have not understood the problem all that well even after a decade when we write the third code. These codes are research codes. Ongoing experiments. The main goal is NOT to produce long-term maintainable software, but to produce scientific understanding and build intuition about the systems that are modeled. The code is just another tool among experiements, analytic calculations and back-of-the-envelope discussions on a white board.
Rewriting the code every 5 years is an insane proposition in software engineering, but completely ok in some fields of science.
Would I like better language support to check SI units for me? Sure. Would I like highly performant libraries for vector fields, that work with gcc 4.6 on a top 500 machine? Sure. Would I like to be allowed to spend time on fixing yeah-I-guess-it-works code? You bet.
But would I like HN to just up about "scientists just need to learn to code"? Oh hell yes! Because -- believe it or not -- we often DO know better. But fixing code is not what the taxpayers, what YOU, pay us for. We are paided to understand nature. And until you are willing to pay higher taxes and spend more money and science and to invest more into fixing long term infrastructure you really do not get to be so damn condescending.