If scientists want to improve their code, the first and most important step is to get them to use descriptive verbose names for their variables and functions, and to learn the single responsibility principle.
A verbose name makes which variable it corresponds to in the paper immediately obvious, and makes the code understandable.
It may be _possible_ to have a verbose variable that makes it obvious what it corresponds to in the paper, but I've never seen it. That really rings like a No True Scotsman argument to my ear. I much prefer the creole of latex style naming that most researchers tend towards. For example, take this code/paper pair
https://www.web.stanford.edu/~boyd/l1_ls/
The first line in the main loop is
z = A*x-y;
residual = dictionary_matrix * optimization_variable - sensed_data
I don't find your second example exhausting to read at all - in fact I find it far _easier_ to read, especially to an astute reader (who is trying to make sense of a particular scientific literature).
No person will have any idea what `z = A*x-y` could mean 'in context', when it's so damn abstract of a reference.
Brevity in variable names is horrendous when cross-checking against papers. It is also a waste of time for scientists to reiterate what the legends/variable names mean, especially when their code could have used clear, unambiguous, verbose names instead.
I don't buy that in the sense that you mean. In my experience, code generally lives on no matter what, and code that remains write-once is often because no one can read it and thus update it (i.e., write it again). I've seen a decent amount of scientific code, and although it was never meant to be ran more than once by anyone other than the original author, it almost always was done so. And because it wasn't designed well, written well, or even commented well, code like that basically becomes this immutable artifact that people simultaneously don't want touched but also want it updated and keep on running.
I did an REU in mathematics back in the day where I wrote a decent amount of MAPLE code. I wasn't even a software engineer at the time, since I was a math and EE major. I honestly had little concept of what computer science even was and was not a programmer. It was not hard at all to write decent code that was commented and organized.
The problem is that most scientists, in my experience, view software and code as somewhat below them. It's like a hammer and a box of nails to them.
I inherited someone's Ph.D. dissertation Python code that had lived on across several labs. It had basically reached the "needs a rewrite" state because it was impossible to get it to run. It was stuck on Python <2.7 and basically couldn't be upgraded without replacing a lot of it because the Python distribution it was based on had become deprecated and none of the packages could be upgraded without changing a lot of the code. I'm not sure I ever saw a single comment. The person who originally wrote the code actually ended up working at one of the major scientific Python shops, somewhere I once applied to and was unceremoniously turned down. An interesting turn of events.
That is, I agree that this is better engineered code. I disagree that this is better science code.
I would understand if the list comprehension were replaced with the corresponding numpy/scipy/pandas/etc code.
At some point you’ll want to come back to code that looks more like the first example, ideally with plenty of documentation that explains how you got there. Some other comment put it really well: you create a kind of personal mapping of scientific concepts to programming abstractions. You don’t want to have colleagues (or your future self) figure out that mapping to understand your results in terms of the science you’re working on.
If only it were so!
Lots and lots of "scientific" code turned out to be re-formatting data. There were some things I wrote in Perl, then re-wrote in Java (which was the standard for non-compute-bound code at the time).
But then there was the actual computation...
Holy cow the code from the 1970s we had to learn, to transform into freaking Java code. And holy cow the poor souls that came after me, howling in torment at my code.
I think the problem here is that each iteration is solving a different and more general problem, but while the use of the flexibility at each level is discussed, it's not framed as code solving one problem and the difficulty as the same approach is applied to a broader version of the problem.
IMO, from a pedagogical point of view, this could be improved a couple of ways:
(1) First, the initial code is not, and should not be described as, “bad code”. Given a simple initial case, it is clear, straightforward, good code. It's not abstracted to handle diverse cases, but it's not intended to be. It should be held up as an excellent model of solving a simple initial problem and a good starting point.
(2) Each of the subsequent levels of abstraction enables the code to solve a more general problem more clearly than just layering on more special-case code. So, each step should expand the problem, show code to solve the expanded problem at the pre-existing level of abstraction, and then introduce the refactoring that is appropriate for it. (An extra nice thing would be to show another extension to the problem after each that fits into the new level of abstraction well, leveraging the abstraction introduced.)
We should get out of the mindset that either more or less general code is “better” in some universal sense: different levels of abstraction and generality are suitable for different problems, and often you’ll start out solving, and coding for a narrower version of the target problem than the eventual goal. Eventually, you may develop a level of intuition for the level of abstraction necessary to solve the general problem initially, but it's also easy to overabstract a solution if you don't have a keen awareness of the limits of needed generality. It's a lot easier to avoid overabstraction—which makes code harder to read and understand—if you start simple and refactor incrementally as you generalize up to just what you need.
If your scientific code is meant to be used in actual software or if you want to write tests for the functions used, a little bit more abstraction might be a good idea, however.
The initial abstraction is actually pretty good, conceptually. You have a function that returns a value, like roll(sides=6), or an object you can manipulate to get a value: Die(sides=6).roll(). You then take those samples, somehow analyze them, and get a result. That code matches people's mental model very well.
If you were going to do this in C++, I'd stop here. Things only get more complicated because Python (Matlab and R too) penalize you heavily for leaving the BLAS sandbox.
I'm not a TDD zealot, just that the unit test can provide the assurance of the computation, regardless of the spaghetti inside the computation.
So most of what gets posted to HN?
Scientific code is often write-once, run-once, done. And since mathematics/statistics is largely already formalised at the level of distributions, sampling, and so on -- we shouldn't expect to need to "software engineer" this type of code.
This article should establish a specific audience it has in mind, presumably data scientists in long-runing scientific projects who need to write generic maintable code to be shared across the organization. Then the article should establish when the first example is GOOD, and when it fails in this specific use case.