News for hackers. Stuff that matters.
Re:The Economist's excess deaths model by physPop on Tuesday May 25, 2021 @10:39AM (#27277373)
Generate synthetic data -> run model -> check expected outputs. Yes it a lot of work, but you're reaching millions of people with this model and correctness is paramount!

Similarly, even such simple test harnesses help when yourself or other go to modify the code. Having flags for "Did I break something" is very important.

Re:The Economist's excess deaths model by StavrosK on Tuesday May 25, 2021 @10:44AM (#27277442)
The deliverable here isn't the code, it's the output. If "check expected outputs" is a subjective step anyway, why not just check that the output of the code when run on the data is as expected and skip the test altogether?
Re:The Economist's excess deaths model by stonemetal12 on Tuesday May 25, 2021 @11:09AM (#27277825)
> If "check expected outputs" is a subjective step anyway

It isn't. It is the evidence that calculations in the code were done correctly.

It is like the problems in high school math class. The teacher controls the inputs so that the outputs are known. The teacher then runs the test problem past the student (aka the code). If the known correct answer isn't generated then we know the student didn't do it right.

Re:The Economist's excess deaths model by StavrosK on Tuesday May 25, 2021 @11:19AM (#27277978)
How do you tell that the calculations were done correctly? Presumably you have some way of doing that.

Then, why don't you apply that way to the output? You don't need tests if you only need to do it once.

Re:The Economist's excess deaths model by stonemetal12 on Wednesday May 26, 2021 @01:48PM (#27293755)
>How do you tell that the calculations were done correctly? Presumably you have some way of doing that.

Isn't that what we are discussing, how to tell if a piece of code works? You could do a formal proof that the code works (which is long and tedious) or you can test it (less long and tedious but less rigorous).

>You don't need tests if you only need to do it once.

It doesn't matter how many times you plan on running it. What counts is how much we value the output. I wouldn't bet $20 that untested code works correctly.

Re:The Economist's excess deaths model by pigeonhole123 on Tuesday May 25, 2021 @11:57AM (#27278480)
Checking that the code outputs what you think it does is a pretty low bar to clear. The fact that you can think of higher bars doesn't invalidate the value of checking this.
Re:The Economist's excess deaths model by woeirua on Tuesday May 25, 2021 @12:12PM (#27278676)
Synthetics only take you so far though. Real data almost always has unusual data (outliers), or the data will violate various assumptions that you've made along the way. In many cases there is no way to know ahead of time how the data will break your model. Which is why monitoring for model drift is so important.