Research code, because its requirements are not fixed and it is not frequently run, doesn't need to be as "stable" as a bonafide app. For a bonafide app, requirements do not change as often and the app is run virtually 24/7.
Once the research code becomes "productionized" however - i.e. it is deployed in an online system where uptime & accuracy matter - then I think absolutely it becomes more engineering-heavy and looks quite a bit less like this code.
Would be curious to hear others' thoughts on this distinction between research vs. production code however.
I suspect it will be a downvotably unpopular opinion, but a typical researcher has a significantly larger cognitive capacity than a typical software developer. Consequently, a piece of code of certain complexity will look simpler, and be easier to manipulate for a researcher than for an average software developer.
My impression was ->
Software developers main focus is developing software so they spend a HUGE amount of time developing software, maintaining software, noticing patterns and bugs and pitfalls in software and thus they get pretty decent at writing software.
Data scientists main focus is developing models, so they spend a HUGE amount of time developing models, tweaking models, finding data, and cleaning data, and write basic software to achieve some of those goals.
I wouldn't expect Albert Einstein, Mozart, or Beyoncé to be some fantastic software developer just cause they're smart individuals. I'd expect people who spend a lot of time writing software to generally be the ones who write decent software.
I'd argue it's more about startup time, or whatever the technical term is for the time between when you first start reading code and when you've built enough of the model in your working memory to be able to touch it without breaking everything.
That heroic developer doesn't need to concern themselves with other people's experiences, and they can just write whatever weird idioms suit their brain.
That said, having used a lot of research oriented software packages over the years, researchers typically do not produce high quality software. Their objective is to produce papers, not software that other people can use. If someone else can use it too, then great, but that's not the primary motivation.
Most of the time you're just testing the viability of things and just want to fail fast if they don't work. Getting too attached to some pipeline or abstractions is usually a bad idea.
I don't think this excuses bad code, but I kinda get it when it happens, since you can't just use best practices from the beginning like in software development
Also, if your job is writing code, I'd bet you'd have less difficulty manipulating spaghetti code.
I'm genuinely curious because I understand the need to move fast but is accuracy a necessary sacrifice? (or is there a trick I don't know about)
The only way to make sure it's right is the same way you'd do full verification of other code, going through it line by line and making sure it's doing the right thing. Tests can not do that, they can only assert some forms of intent, and they are not good at catching the types of errors that result from ingesting varied data from lots of sources and getting it into modelable form. It's ETL plus a bunch of other stuff going on here.
However, "the only way to make sure it's right is to ... go through it line by line" is a bold claim. There are multiple named functions and some unnamed functions in this example that could be verified for programmer mistakes (such as typing 1000 vs 10000) and edge case handling (edge cases that often arise from messy ingested data).
But even if they were tested, I'll concede that mistakes can still be made.
Here it is not as clear what the code should do. What should the amount of excess deaths be? In what ways would we change the logic such that the test case would break? If the input data set is static, isn't it more like a mock anyway?
I think for this reason you often see more sanity checks in research code because the should case is not as clearly defined.
With several fake known inputs and there associated outputs we should be able to determine if the calculation is right.
The result on the real world data is not known but when calculating a statistic you should be able to figure out if you are calculating the right statistic or returning 42 for all inputs.
I too know first hand that students and even academics who aren't properly trained and given the right tooling will write prototype code that's not production ready. Occasionally there's even a bona fide mistake, though usually it's just very un-generalizable. It's part of my job to make some of it more production ready. That's the best use of my time and training. Conversely, I'm shit at generating useful research ideas or writing papers. Just don't have the right combination of intuition/training/experience. Good thing my academic institution has the resources to employ them and me.
With tests it is often best practice to write the tests before writing the code that gets tested. But what if you don't know ahead of time, and can't possibly ever know ahead of time?
Writing "tests" for this would be about adding stuff afterwards, like "the max and min values of this column were X and Y". But that is expected to break, if anything changes, because it's not testing anything useful.
My question is: what is one concrete test here that would be useful and actually provide confidence that the code is doing what it should be doing, and how does that test provide better sanity checking than inspecting the tables at each step of the process?
It's widely known on the "theory of testing" that data oriented systems can't be tested without immense amount of effort. If you are interested there are plenty of academic talks about the subject.
It should be said that the statistics themselves do make for some 'guide rails' .. the stats are quite demanding, require a lot of upper-division training to use correctly, and visual feedback can course-correct as the results are iteratively found.
As said in other comments, the content is often associated with a research goal that is weighted more than code-quality. In contrast, a general purpose development language has very broad application, and probably can go wrong in very broad ways. The coder is getting feedback on results, but often lots of code review and expectation of professional results.
I picked out an R ETL sequence from an article last year, and I still pull it out once in a while, to see the extensive, clever and (to my eye) really hard to read R data ingestion and manipulation. Personally I think it is a fair tradeoff to say that the expertise to use this environment is a bit of a filter, and the rigor that the results (therefore intermediate results) demand, does move the expectations on code quality ..
As for tests, there is probably no defense on not having tests also.. it is probably going to be more common as the field of R and data analysis inevitably grows.
Testing is hard. The researchers who write this code typically don't have the necessary hands-on experience to write good tests, even if they had enough time in a day/week to actually do it.
Edit: Also the code tends to be very "high level", chaining lots of high-level API functions together, and even coming up with assertions to test tends to be a bit of a challenge. Testing such code turns out to be surprisingly difficult; you might end up just rewriting big chunks of your code in the test suite.
In my data science work, I've focused on writing tests for the complicated sections (e.g. lower-level string processing routines) and just trying to focus on the keep the other stuff very clean and readable.
In general (and in my opinion), a "good test" is one that asserts that an invariant is always so, or that a property that is expected to hold under certain conditions does indeed hold under those conditions. Defining such invariants and properties for "data science code" tends to be difficult, and even when you define them it might be difficult or impossible to test them in a straightforward fashion.
And that's even before you get to the probabilistic stuff.
Similarly, even such simple test harnesses help when yourself or other go to modify the code. Having flags for "Did I break something" is very important.
It isn't. It is the evidence that calculations in the code were done correctly.
It is like the problems in high school math class. The teacher controls the inputs so that the outputs are known. The teacher then runs the test problem past the student (aka the code). If the known correct answer isn't generated then we know the student didn't do it right.
Then, why don't you apply that way to the output? You don't need tests if you only need to do it once.
Isn't that what we are discussing, how to tell if a piece of code works? You could do a formal proof that the code works (which is long and tedious) or you can test it (less long and tedious but less rigorous).
>You don't need tests if you only need to do it once.
It doesn't matter how many times you plan on running it. What counts is how much we value the output. I wouldn't bet $20 that untested code works correctly.
Good software practices don't only have to be for "production" or CI or large teams of coders. Testing for correctness could be seen as part of the work of delivering a high quality product/graph/prediction/model.
Theres a huge amount of data reshaping, mutating, and general wrangling going on here, how can one be confident without even a known input->output integration type test?