News for hackers. Stuff that matters.
Re:The Economist's excess deaths model by physPop on Tuesday May 25, 2021 @09:41AM (#27276624)
Forgive the naive question: but is code like this typical for "data science"? It seems more like something out of a masters project... Huge script files with zero testing?

Theres a huge amount of data reshaping, mutating, and general wrangling going on here, how can one be confident without even a known input->output integration type test?

Re:The Economist's excess deaths model by alexpetralia on Tuesday May 25, 2021 @09:48AM (#27276701)
In general I think data science code is more "research-oriented", meaning that it is not run continuously like a software app and its requirements often evolve as more information is learned from the data. Research code produces results maybe once or twice a day (manually triggered) while software apps potentially service hundreds of thousands of requests a day.

Research code, because its requirements are not fixed and it is not frequently run, doesn't need to be as "stable" as a bonafide app. For a bonafide app, requirements do not change as often and the app is run virtually 24/7.

Once the research code becomes "productionized" however - i.e. it is deployed in an online system where uptime & accuracy matter - then I think absolutely it becomes more engineering-heavy and looks quite a bit less like this code.

Would be curious to hear others' thoughts on this distinction between research vs. production code however.

Re:The Economist's excess deaths model by physPop on Tuesday May 25, 2021 @10:37AM (#27277344)
The purpose of testing in my example is for correctness, not continuous integration. I would argue regardless of "production" use, the fact that this is publishing a widely read newspaper makes it highly impactful and more likely to influence policy and decisionmaking, thus we want to know if its correct !
Re:The Economist's excess deaths model by whymauri on Tuesday May 25, 2021 @06:15PM (#27283417)
Well... that's partially why it's open-sourced. What fuller transparency of correctness is there an open-sourcing the entire analysis?
Re:The Economist's excess deaths model by admissionsguy on Tuesday May 25, 2021 @10:06AM (#27276904)
Many of the "good practices" in software development are primarily meant to reduce the cognitive effort on the part of the person reading the code.

I suspect it will be a downvotably unpopular opinion, but a typical researcher has a significantly larger cognitive capacity than a typical software developer. Consequently, a piece of code of certain complexity will look simpler, and be easier to manipulate for a researcher than for an average software developer.

Re:The Economist's excess deaths model by codyb on Tuesday May 25, 2021 @10:27AM (#27277185)
Huh? A researcher has greater cognitive capacity? That's a funny take.

My impression was ->

Software developers main focus is developing software so they spend a HUGE amount of time developing software, maintaining software, noticing patterns and bugs and pitfalls in software and thus they get pretty decent at writing software.

Data scientists main focus is developing models, so they spend a HUGE amount of time developing models, tweaking models, finding data, and cleaning data, and write basic software to achieve some of those goals.

I wouldn't expect Albert Einstein, Mozart, or Beyoncé to be some fantastic software developer just cause they're smart individuals. I'd expect people who spend a lot of time writing software to generally be the ones who write decent software.

Re:The Economist's excess deaths model by MengerSponge on Tuesday May 25, 2021 @11:05AM (#27277759)
As a researcher, this is flattering, but probably wrong. Reading other people's code always sucks, but reading somebody's solo project hero code that doesn't follow any typical design patterns? That's nightmarish.

I'd argue it's more about startup time, or whatever the technical term is for the time between when you first start reading code and when you've built enough of the model in your working memory to be able to touch it without breaking everything.

That heroic developer doesn't need to concern themselves with other people's experiences, and they can just write whatever weird idioms suit their brain.

Re:The Economist's excess deaths model by woeirua on Tuesday May 25, 2021 @12:08PM (#27278625)
I think most research code bases are used by a significantly smaller number of people. Hence it's easier for the researcher to know what's going on in their code because they probably wrote a significant portion of the code they use and hence understand the assumptions, and limitations that are baked into the code. Whereas if you try to use a library normally you have to kind of figure those things out on your own.

That said, having used a lot of research oriented software packages over the years, researchers typically do not produce high quality software. Their objective is to produce papers, not software that other people can use. If someone else can use it too, then great, but that's not the primary motivation.

Re:The Economist's excess deaths model by naomisperfume on Tuesday May 25, 2021 @10:28AM (#27277204)
This is a valid point, but I think in part the data science / machine learning cycle doesn't reward careful development like software engineering does.

Most of the time you're just testing the viability of things and just want to fail fast if they don't work. Getting too attached to some pipeline or abstractions is usually a bad idea.

I don't think this excuses bad code, but I kinda get it when it happens, since you can't just use best practices from the beginning like in software development

Re:The Economist's excess deaths model by patates on Tuesday May 25, 2021 @10:15AM (#27277009)
It's always easy if you're the only one writing/maintaining the code.

Also, if your job is writing code, I'd bet you'd have less difficulty manipulating spaghetti code.

Re:The Economist's excess deaths model by admissionsguy on Tuesday May 25, 2021 @09:51AM (#27276726)
This code is above average by the standards of academia I've dealt with. You've got comments announcing what each part of the code is doing, and there is only one copy of each script. Normally, you would expect to have multiple nested directories with various variations of the same code, plus multiple variations of each script with cryptic suffixes.
Re:The Economist's excess deaths model by fullshark on Tuesday May 25, 2021 @09:51AM (#27276729)
Yes, data scientists frequently are working alone and don't care about readability so much as results. Also the code isn't going into production so they don't care about optimizing it by and large.
Re:The Economist's excess deaths model by checker on Tuesday May 25, 2021 @10:00AM (#27276834)
Sure, but physPop's concerns still stands - how can one be confident about the results? Is it just eyeballing "this looks right"? If so, how many anomalies are missed due to handwaving, and how many results are inaccurate?

I'm genuinely curious because I understand the need to move fast but is accuracy a necessary sacrifice? (or is there a trick I don't know about)

Re:The Economist's excess deaths model by epistasis on Tuesday May 25, 2021 @10:29AM (#27277229)
As other comments indicate, how could you be sure the tests are testing the right thing? It's easy enough to slap on a few assert statements (or stopifnot statements, since it's R) about number of rows or something, but that's no replacement for manual inspection of the data. A code-based test can not be invented that will ever substitute for looking at the data in the raw, plotting it, and verification using your full mental capacities.

The only way to make sure it's right is the same way you'd do full verification of other code, going through it line by line and making sure it's doing the right thing. Tests can not do that, they can only assert some forms of intent, and they are not good at catching the types of errors that result from ingesting varied data from lots of sources and getting it into modelable form. It's ETL plus a bunch of other stuff going on here.

Re:The Economist's excess deaths model by checker on Tuesday May 25, 2021 @03:07PM (#27281030)
I didn't make the claim that a code-based test can be invented that will substitute for looking at the data in the raw, plotting it, and verification using your full mental capacities.

However, "the only way to make sure it's right is to ... go through it line by line" is a bold claim. There are multiple named functions and some unnamed functions in this example that could be verified for programmer mistakes (such as typing 1000 vs 10000) and edge case handling (edge cases that often arise from messy ingested data).

But even if they were tested, I'll concede that mistakes can still be made.

Re:The Economist's excess deaths model by alexpetralia on Tuesday May 25, 2021 @10:15AM (#27277000)
If I understand correctly, tests are for what the code should do. There is some business logic and you are asserting that the business logic does what it should do (the test). If you manipulate the business logic, and it no longer does what the test says the code should do, the test fails.

Here it is not as clear what the code should do. What should the amount of excess deaths be? In what ways would we change the logic such that the test case would break? If the input data set is static, isn't it more like a mock anyway?

I think for this reason you often see more sanity checks in research code because the should case is not as clearly defined.

Re:The Economist's excess deaths model by stonemetal12 on Tuesday May 25, 2021 @10:59AM (#27277660)
>What should the amount of excess deaths be?

With several fake known inputs and there associated outputs we should be able to determine if the calculation is right.

The result on the real world data is not known but when calculating a statistic you should be able to figure out if you are calculating the right statistic or returning 42 for all inputs.

Re:The Economist's excess deaths model by cinntaile on Tuesday May 25, 2021 @10:18AM (#27277043)
That's why it's a good thing that the code is open. The usual standard is the same type of code but without public access. This is definitely a step in the right direction.
Re:The Economist's excess deaths model by fullshark on Tuesday May 25, 2021 @10:06AM (#27276906)
There's definitely a greater risk of a bug leading to misleading results. There's no real unique trick other than possibly someone else trying to replicate the results and catching an error, or trying to use the code on another data set and catching a mistake.
Re:The Economist's excess deaths model by ethanbond on Tuesday May 25, 2021 @09:44AM (#27276650)
In my experience, yes. Data science is highly creative/improvisational and the resulting artifacts of work reflect that. FWIW, normal science is pretty similar. The clear step 1, step 2, ..., you see in scientific publications is effectively retcon done for the benefit of the reader's understanding.
Re:The Economist's excess deaths model by kspacewalk2 on Tuesday May 25, 2021 @09:55AM (#27276771)
As already pointed out, this isn't an engineering product. They've released the source, did some sanity checks, and moved on with their busy lives. If you find an error - cool, go ahead and report it.
Re:The Economist's excess deaths model by Glavnokoman on Tuesday May 25, 2021 @10:06AM (#27276909)
By sanity check you mean shows the numbers I wanted to see? Not saying there's necessarily something wrong in this particular code but I know first hands how little effort is put into validating the scientific codes and how much of produces just random crap.
Re:The Economist's excess deaths model by kspacewalk2 on Tuesday May 25, 2021 @10:29AM (#27277220)
Which is why releasing their code is critical. They lay it all out for you, with their pride/reputation on the line if it produces 'just random crap'. Writing exhaustive unit tests is probably not something they are good at, nor is it a requirement for them, nor is it the best use of their time.

I too know first hand that students and even academics who aren't properly trained and given the right tooling will write prototype code that's not production ready. Occasionally there's even a bona fide mistake, though usually it's just very un-generalizable. It's part of my job to make some of it more production ready. That's the best use of my time and training. Conversely, I'm shit at generating useful research ideas or writing papers. Just don't have the right combination of intuition/training/experience. Good thing my academic institution has the resources to employ them and me.

Re:The Economist's excess deaths model by epistasis on Tuesday May 25, 2021 @11:03AM (#27277732)
How can you know which numbers you want to see?

With tests it is often best practice to write the tests before writing the code that gets tested. But what if you don't know ahead of time, and can't possibly ever know ahead of time?

Writing "tests" for this would be about adding stuff afterwards, like "the max and min values of this column were X and Y". But that is expected to break, if anything changes, because it's not testing anything useful.

My question is: what is one concrete test here that would be useful and actually provide confidence that the code is doing what it should be doing, and how does that test provide better sanity checking than inspecting the tables at each step of the process?

Re:The Economist's excess deaths model by da39a3ee on Tuesday May 25, 2021 @10:02AM (#27276850)
Yes, you can see the divide in the popularity of interactive notebooks like jupyter and observable. To programmers who use a standard git-based workflow, the interactive notebooks are anathema because you can't use version control to compare code state and monitor progress, and because of the global variables. But to many quantitative researchers for whom software version control isn't a large part of their professional worldview, it seems normal to just keep adding to the same document until you hit a state that seems to work, without much emphasis on checkpointing and navigation between checkpoints, and testing at each checkpoint.
Re:The Economist's excess deaths model by lanevorockz on Tuesday May 25, 2021 @10:31AM (#27277270)
That is quite common for R language, it is a scripting language that you are suppose to explore as you use. So if you want to test it, it's very simple ... Just run the script until the part you are interested and then plot the hell out of it.

It's widely known on the "theory of testing" that data oriented systems can't be tested without immense amount of effort. If you are interested there are plenty of academic talks about the subject.

Re:The Economist's excess deaths model by mistrial9 on Tuesday May 25, 2021 @10:01AM (#27276843)
in R particularly, there is a wide range of code quality, by developer standards, yes

It should be said that the statistics themselves do make for some 'guide rails' .. the stats are quite demanding, require a lot of upper-division training to use correctly, and visual feedback can course-correct as the results are iteratively found.

As said in other comments, the content is often associated with a research goal that is weighted more than code-quality. In contrast, a general purpose development language has very broad application, and probably can go wrong in very broad ways. The coder is getting feedback on results, but often lots of code review and expectation of professional results.

I picked out an R ETL sequence from an article last year, and I still pull it out once in a while, to see the extensive, clever and (to my eye) really hard to read R data ingestion and manipulation. Personally I think it is a fair tradeoff to say that the expertise to use this environment is a bit of a filter, and the rigor that the results (therefore intermediate results) demand, does move the expectations on code quality ..

As for tests, there is probably no defense on not having tests also.. it is probably going to be more common as the field of R and data analysis inevitably grows.

Re:The Economist's excess deaths model by MattGaiser on Tuesday May 25, 2021 @09:57AM (#27276805)
For something like this, the code is more an aid to analysis rather than the product itself.
Re:The Economist's excess deaths model by nerdponx on Tuesday May 25, 2021 @09:45AM (#27276664)
Yes, sadly.

Testing is hard. The researchers who write this code typically don't have the necessary hands-on experience to write good tests, even if they had enough time in a day/week to actually do it.

Edit: Also the code tends to be very "high level", chaining lots of high-level API functions together, and even coming up with assertions to test tends to be a bit of a challenge. Testing such code turns out to be surprisingly difficult; you might end up just rewriting big chunks of your code in the test suite.

In my data science work, I've focused on writing tests for the complicated sections (e.g. lower-level string processing routines) and just trying to focus on the keep the other stuff very clean and readable.

Re:The Economist's excess deaths model by StavrosK on Tuesday May 25, 2021 @10:31AM (#27277251)
What would "good tests" look like for something like this?
Re:The Economist's excess deaths model by nerdponx on Tuesday May 25, 2021 @01:18PM (#27279565)
That's part of the problem.

In general (and in my opinion), a "good test" is one that asserts that an invariant is always so, or that a property that is expected to hold under certain conditions does indeed hold under those conditions. Defining such invariants and properties for "data science code" tends to be difficult, and even when you define them it might be difficult or impossible to test them in a straightforward fashion.

And that's even before you get to the probabilistic stuff.

Re:The Economist's excess deaths model by physPop on Tuesday May 25, 2021 @10:39AM (#27277373)
Generate synthetic data -> run model -> check expected outputs. Yes it a lot of work, but you're reaching millions of people with this model and correctness is paramount!

Similarly, even such simple test harnesses help when yourself or other go to modify the code. Having flags for "Did I break something" is very important.

Re:The Economist's excess deaths model by StavrosK on Tuesday May 25, 2021 @10:44AM (#27277442)
The deliverable here isn't the code, it's the output. If "check expected outputs" is a subjective step anyway, why not just check that the output of the code when run on the data is as expected and skip the test altogether?
Re:The Economist's excess deaths model by stonemetal12 on Tuesday May 25, 2021 @11:09AM (#27277825)
> If "check expected outputs" is a subjective step anyway

It isn't. It is the evidence that calculations in the code were done correctly.

It is like the problems in high school math class. The teacher controls the inputs so that the outputs are known. The teacher then runs the test problem past the student (aka the code). If the known correct answer isn't generated then we know the student didn't do it right.

Re:The Economist's excess deaths model by StavrosK on Tuesday May 25, 2021 @11:19AM (#27277978)
How do you tell that the calculations were done correctly? Presumably you have some way of doing that.

Then, why don't you apply that way to the output? You don't need tests if you only need to do it once.

Re:The Economist's excess deaths model by stonemetal12 on Wednesday May 26, 2021 @01:48PM (#27293755)
>How do you tell that the calculations were done correctly? Presumably you have some way of doing that.

Isn't that what we are discussing, how to tell if a piece of code works? You could do a formal proof that the code works (which is long and tedious) or you can test it (less long and tedious but less rigorous).

>You don't need tests if you only need to do it once.

It doesn't matter how many times you plan on running it. What counts is how much we value the output. I wouldn't bet $20 that untested code works correctly.

Re:The Economist's excess deaths model by pigeonhole123 on Tuesday May 25, 2021 @11:57AM (#27278480)
Checking that the code outputs what you think it does is a pretty low bar to clear. The fact that you can think of higher bars doesn't invalidate the value of checking this.
Re:The Economist's excess deaths model by woeirua on Tuesday May 25, 2021 @12:12PM (#27278676)
Synthetics only take you so far though. Real data almost always has unusual data (outliers), or the data will violate various assumptions that you've made along the way. In many cases there is no way to know ahead of time how the data will break your model. Which is why monitoring for model drift is so important.
Re:The Economist's excess deaths model by omginternets on Tuesday May 25, 2021 @10:18AM (#27277042)
This is pretty good by data-science (and other research) standards.
Re:The Economist's excess deaths model by physPop on Tuesday May 25, 2021 @10:41AM (#27277389)
Would love some commentary on why this is an unpopular question.

Good software practices don't only have to be for "production" or CI or large teams of coders. Testing for correctness could be seen as part of the work of delivering a high quality product/graph/prediction/model.