The reason why they can't and don't want to hire software engineers to build well designed software is because they don't have any money for it and it is very mismatched with their incentives. If a PI gets a grant for a project, they typically get a constrained budget to achieve the scientific goals of the project. There is usually no money (and it would not be approved) to engineer well designed software, because that doesn't produce scientific outcomes.
Now in the situations when there is budget for this, which are the super large institutional research initiatives, like e.g. CERN, they build incredibly well designed software. The SKA for example will have the largest storage cloud in the world.
If one wants better designed scientific software one needs to seriously overhaul the scientific funding process and quite significantly increase funding.
...please excuse this shameless plug: I'm interested in people who would like to work with me to build research software (both on-premise and AWS cloud) using best practices, in particular using Rust. If you are interesteed in machine learning, natural language processing and information retrieval and would like to work with me as a research software engineer, please do get in touch (leidner at acm dot org). Our new 2 TB RAM dev research group server running Ubuntu 22.04 LTS is waiting for you (CPU/GPU cluster ressources also available) to set up (a VM for) your CI infrastructure. Your software will be used to push forward the frontier of research in machine learning and its applications to language (summarization, question answering) and information retrieval (vertical search engines, learning to rank, unsupervised topic modeling).
Edit: academia can be poorly funded and bureaucratic. The former can be fixed by grant applications, the latter must be tolerated until politicians/activists fix it.
These people desperately need software help and have worthwhile projects that need software help. They also need to learn to collaborate over just parsing out obscure goals and being upset when what they get matches their three sentence email and not their vision in their head.
I can definitely tell you that if I talk about my work non-scientists become quickly uninterested.
This is all painting with broad strokes, but in my experience, the most interesting intellectual conversations I've had have been with people with the least amount of degrees. I always feel when I talk to someone with a Ph.D., they're afraid to let it be known someone might know more than them and if that happens, then it's suddenly something that's not important.
The book Disciplined Minds gets at this some. It's part institutionalization and other things. My gut feeling is that it's exacerbated by publish or perish culture and the push to always be right. So people that are brought up through that are intellectually risk averse.
> In my experience, scientists do not like anything that they perceive as "soft". I like to say that the hard sciences are actually the easy ones and the soft sciences are actually the hard ones. (As it turns out, Herbert Simon also said this.) The point is that when you discuss things like design, complexity, architecture, systems thinking, etc., I've found that this does not stick, so to speak. Scientists do not see beauty in form equaling function. In my experience, scientists like learning about things they consider science-y enough for their tastes.
In my experience what you say is true for engineers (and software engineers are amongst the worse in this respect), but much less so for scientists from the hard sciences (physicists, chemists, biologists). You can also see that from many famous physicists being involved in philosophy, while many engineers consider philosophy "irrelevant talking about nothing".
> This is all painting with broad strokes, but in my experience, the most interesting intellectual conversations I've had have been with people with the least amount of degrees. I always feel when I talk to someone with a Ph.D., they're afraid to let it be known someone might know more than them and if that happens, then it's suddenly something that's not important.
Maybe we converse in very different cycles, but generally for people I know the "higher up the foodchain" the more they ask even very basic questions. In my experience you can often distinguish who is the most accomplished scientists in the room by listening who is asking the most basic questions.
All this is obviously anecdotal evidence, but is reflected also by famous accounts like e.g. Feynman's account of the challenger commission.
If the requirement is that it lasts 30 years, you can choose more timeless languages like C, over say Rust.
And yes, way too much research code reads like somebody's first coding project.
Moreover, people tend to forget that scientific software evolves much more slowly than commercial software and the number of people who work on this are much fewer, so why always adopt the newest method, if the code is still working. Show me commercial projects that have had the stability of something like LAPACK or BLAS, those essentially had a stable API for > 50 years now, with very few bugs.
But to be honest, I've seen this in industry as well from software developers. So maybe it's just a law of software development that's amplified in scientific software.
They/We have probably been calling it that since before you were alive. I wouldn’t be surprised if it comes from the 60s or 70s.
Every group of people has their own terminology for stuff. There is no right or wrong.
As someone with a mathematics background, whenever I encounter someone in another community using mathematics (like economics or accounting or finance or whatever) using mathematical terminology that seems silly to me, it's a bad sign.
Why not? "A code" is a software package which solves some kind of problem computationally. Plural, codes. I'm sure it's been in use almost as long as the programmable computers. Here's[1] a link to an article from 1968 which uses the term in the singular ("The MENE Neutron Transport Code".)
[1] https://www.osti.gov/biblio/4819611-mene-neutron-transport-c...
Then you probably haven't been around physicists, or read enough physics-related material. A non-scholarly reference which can't be easily checked online is George Dyson's "Project Orion," where Freeman Dyson (author's father) is quoted as mentioning this or that "code" as significant for the design of various Orion parts. (That should be a native enough English speaker.) The time-frame is late 1950s to early 1960s.
That reference you give doesn’t appear to be available online but I’ll take your word for it.
Obviously, what word they use for it is arbitrary but it's indicative of an actual problem - they're all totally cut off from the software engineering / CS community. Lots of them think programming is like learning to juggle or ride a bike or something, a skill that anyone just learns and then they're done. Then, obviously, they make major mistakes that invalidate their conclusions and nobody notices or cares because it's all student code, all the time.
I think the point is that the physicists were among the very early adopters of computing systems. I’m not sure about the history of the term “code” used this way, but it might literally predate compiled software since Ulam and von Neumann invented the Monte Carlo method in 1948 during the Manhattan Project.
Personally I think it just sounds outdated now, but it’s pretty common to hear people call their software “a hydrodynamics code” or “a molecular dynamics code”
The tendency is to call computational science software “codes”, but I’ve never heard a scientist call, ie a web browser, a code.
There’s a different connotation. Like how theres a different connotation between a program, an application, and and app. The are largely synonymous, but have slightly different uses.
See for example Code_Aster (as in, "a code"), which is one of the largest projects for open source FEA (and it mostly rooted in academia).
As for "breaking in," I dunno. The hiring process seems very fair and international (very few locals work there in a technical capacity AFAICT), and there are plenty of people with flexible ideas. Not everyone, for sure, but then there are so many different groups there you couldn't possibly blanket everyone with that statement.
This is not really the core issue here, but yes, I've encountered this many times, and I have to admit that in some professional contexts I use "codes" as a shibboleth to identify people who aren't really in touch with the software engineering world.
Tldr: They view software as just a tool and don't see how they'll benefit from following best practices.
I've literally sat in meetings trying to get access to scientific projects I'm interested in and applied for jobs even at the university I already worked at in similar capacities, and yet, it's filled with PIs just wanting grunt labor (i.e., what they're used to from graduate students) to build things that they understand or you absolutely never hear back or the pay is a third of what you already make. And then there's pockets of software people in this niche world that only want to do things their way, because that's all they've ever done, and they can really be quite obstinate. They're like this massive pillar you must somehow work around.
I really, really, really like writing software in scientific contexts, but it's such a quagmire. I'd like to return to that world some day with the lessons I've learned outside of it. Software in support of an interesting domain or context is just really fun. I've also tried getting a job at an architecture firm that had some very neat ideas for building out a tool, but the response was basically the same as described above.