News for hackers. Stuff that matters.
Re:What if writing tests was a joyful experience? by avgcorrection on Saturday January 14, 2023 @05:02AM (#34378936)
> I think you’re supposed to write some nonsense, like assert fibonacci(15) == 8, then when the test says “WRONG! Expected 8, got 610”, you’re supposed to copy and paste the 610 from your terminal buffer into your editor.

> This is insane!

The sane approach is presumably to either expand the call tree and verify all the unique subsolutions. Or to do every step with a calculator if you can’t expand the call tree.

> The %expect block starts out blank precisely because you don’t know what to expect. You let the computer figure it out for you. In our setup, you don’t just get a build failure telling you that you want 610 instead of a blank string. You get a diff showing you the exact change you’d need to make to your file to make this test pass; and with a keybinding you can “accept” that diff. The Emacs buffer you’re in will literally be overwritten in place with the new contents [1]:

Oh okay. The non-insane approach is to do the first thing but Emacs copies the result on your behalf.

Re:What if writing tests was a joyful experience? by eru on Saturday January 14, 2023 @05:47AM (#34379168)
Well, the non-insane thing is to do property-based testing. Instead of testing only a handful of examples.
Re:What if writing tests was a joyful experience? by c-cube on Saturday January 14, 2023 @06:57AM (#34379565)
They also do that, the post refers to their Quickcheck library. But how do you property test the Fibonacci function ? There isn't much to say about it...
Re:What if writing tests was a joyful experience? by angio on Saturday January 14, 2023 @07:10AM (#34379646)
I believe the way to test it is to have a property like `n in integer | fib(n+2) == fib(n+1) + fib(n)`. This is close to the naive (but obviously correct?) implementation of fib and can be used to test the optimized version of the function.

You can also test that the sequence is increasing like `fib(n+2) > fib(n+1) > fib(n)`.

Re:What if writing tests was a joyful experience? by avgcorrection on Saturday January 14, 2023 @07:07AM (#34379626)
You use the naive implementation as a test oracle, limit `n` to something small (through the property tester), and use the test oracle on your efficient implementation.

Unit testing elegant functions has no value.

(fib is often used as an example. But you asked how to test it.)

Re:What if writing tests was a joyful experience? by davidgrenier on Saturday January 14, 2023 @08:02AM (#34379996)
In combinatorics the adjusted fibonacci numbers start with 1 instead and is more commonly used as it aligns with many other results. One might want to document in the code, via a test, which sequence is of interest.

This is just an example of course but elegant functions might need to be tested.

Re:What if writing tests was a joyful experience? by bottled_poe on Saturday January 14, 2023 @07:45AM (#34379867)
Actually, Fibonacci results can be fairly precisely tested with the golden ratio ratio.
Re:What if writing tests was a joyful experience? by davidgrenier on Saturday January 14, 2023 @07:55AM (#34379940)
You could compare against a closed-form solution:

    let fib2 n =
        let sq5 = sqrt(5.0)
        ((1.0 + sq5)/2.0)**(float n)/sq5
        |> round |> int
Re:What if writing tests was a joyful experience? by MereInterest on Saturday January 14, 2023 @08:43AM (#34380263)
The problem is that the closed-form solution is vulnerable to floating-point error. If the calculations are done in float32 (including all intermediate steps), then the 32nd fibonacci number is erroneously given as 2178310, instead of the correct value of 2178309. Using float64 does better, but still has an error at the 71st fibonacci number. (I made a quick plot of the error as a function of N at https://i.imgur.com/bbc9OFC.png. As soon as the error crossed ±0.5, the rounding results in the wrong result.)

These are fine for property-based testing, so long as you restrict yourself to the range in which you have a correct value. But at that point, you might as well just hard-code the first 93 fibonacci numbers (the most that will fit in a uint64_t) and be done with it.

Re:What if writing tests was a joyful experience? by travisjungroth on Saturday January 14, 2023 @08:32AM (#34380201)
Properties of the Fibonacci function:

It is non-decreasing monotonic. fib(n) <= fib(n+1)

It is increasing monotonic after 1. fib(n) < fib(n+1)

Its domain and codomain are non-negative integers.

fib(n) + fib(n+1) == fib(n+2) Notice this is like the recursive solution except going the other way (addition not subtraction) and is missing the base case.

Re:What if writing tests was a joyful experience? by xtagon on Saturday January 14, 2023 @01:48PM (#34382806)
It also converges closer and closer to the golden ratio. Implementing a property test for that would be interesting.
Re:What if writing tests was a joyful experience? by travisjungroth on Saturday January 14, 2023 @01:55PM (#34382875)

    a, b, c = fib(n), fib(n+1), fib(n+2)
    assert abs(c / b - phi) < abs(b / a - phi)
Re:What if writing tests was a joyful experience? by eru on Sunday January 15, 2023 @02:28AM (#34387459)
In addition to what travisjungroth said, you can also check against a reference implementation.

Eg if you coded up an O(n) version of the Fibonacci calculation, you can check against the naive recursive one (or if you are feeling confident, you can check against the O(log n) solution via repeated squaring of matrices.)

Re:What if writing tests was a joyful experience? by postalrat on Saturday January 14, 2023 @11:13PM (#34386654)
I prefer "code it twice and hope you get it right once" testing.

Complex systems use that system everywhere. Why aren't we doing it for our code?

Re:What if writing tests was a joyful experience? by eru on Sunday January 15, 2023 @02:26AM (#34387449)
Comparing the output of your system against an oracle is one property you can test.

But you don't always have an oracle. So other properties still make sense.

As a simple example: if you code up a quantum mechanics simulator, that's hard, and I wouldn't be able to code up an oracle for you straight away. But I can tell you that you probably want to check that things like momentum and energy better be conserved.

Re:What if writing tests was a joyful experience? by thaumasiotes on Saturday January 14, 2023 @05:47AM (#34379171)
Yes, I have difficulty understanding the point of a test-writing system that relies on your explicit assumption that whatever the code already does is correct.

What are you testing? Why?

Re:What if writing tests was a joyful experience? by sesm on Saturday January 14, 2023 @06:24AM (#34379378)
This looks similar to snapshot testing in UI, where you save an output of UI components and test system notifies you when the output changes. This can be useful to detect changes in components that you didn’t intend to change.
Re:What if writing tests was a joyful experience? by Temporary_31337 on Saturday January 14, 2023 @07:06AM (#34379618)
Do you simply mean regression testing?
Re:What if writing tests was a joyful experience? by avgcorrection on Saturday January 14, 2023 @07:52AM (#34379923)
Yeah, weird to see all these variations

- Aha, an expect test!

- Oh, you mean a snapshot test!

- This here is akin to UI testing framework X where the test framework can compare an expected screenshot of the UI to a screenshot of the actual UI!

The last one basically requires automation if you want anyone to make use of it. The regression testing automation described in the OP is a nice-to-have, not a so-good-that-it-gets-a-new-name.

Re:What if writing tests was a joyful experience? by avgcorrection on Saturday January 14, 2023 @10:46AM (#34381150)
… And apparently also “change detector test” https://news.ycombinator.com/item?id=34379175
Re:What if writing tests was a joyful experience? by drothlis on Saturday January 14, 2023 @12:10PM (#34381844)
...and my favourite term, "characterization test": https://en.wikipedia.org/wiki/Characterization_test

"Regression test" means something else, at least at the companies I've worked at: It means a test that was written after a defect was found in production, to ensure that the same defect doesn't happen again (that the fix doesn't "regress"). It can be a manual test or an automated test. https://en.wikipedia.org/wiki/Regression_testing

Re:What if writing tests was a joyful experience? by avgcorrection on Saturday January 14, 2023 @12:31PM (#34382024)
That’s fine and I have no objective argument against it. But I don’t see much reason to need two different names for tests that do the same thing merely based on how they were introduced. Sometimes I add a regression test because I fixed a bug, and sometimes I add a regression test because I just implemented a feature that I don’t want my future self to ruin: six months from now they will co-exist in the same suite and serve the same function.

One reason to call bug fix tests for “regression tests” (and only those kinds of tests) is that someone might regress the code base through a merge conflict (maybe they effectively undo a commit?). So that’s one argument I suppose.

Re:What if writing tests was a joyful experience? by drothlis on Monday January 16, 2023 @05:32AM (#34399145)
"Regression testing" can also refer to a process: When the QA team says they're doing regression testing, it means they're testing that existing functionality hasn't regressed (as opposed to testing a new feature).

I'm not particularly wedded to any of these terms, I'm just pointing out that "regression testing" has an established meaning, and it isn't snapshot testing (outside of certain industries, at least). I do find it amusing that one implementation of snapshot testing (https://pypi.org/project/pytest-regtest/) links to https://en.wikipedia.org/wiki/Regression_testing but that article doesn't describe snapshot testing at all! Maybe the article changed? Oh well, language changes too. ¯\_(ツ)_/¯

Re:What if writing tests was a joyful experience? by avgcorrection on Wednesday January 18, 2023 @11:12AM (#34428552)
Good points.
Re:What if writing tests was a joyful experience? by MichaelBurge on Saturday January 14, 2023 @09:31AM (#34380567)
A regression test is checking causality: Changes in new code, updating dependencies, updating the OS the software is running on, updating shared libraries, porting the code to a new platform, etc. aren't supposed to change the test results.

"I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"

Re:What if writing tests was a joyful experience? by thaumasiotes on Saturday January 14, 2023 @05:47PM (#34384897)
> "I may not know what cos(x) means, but whatever it is shouldn't depend on what OS version I'm running"

Cosine is a terrible example to use for that idea. It's pretty likely to change, for certain x, in similar circumstances to your examples of "when test results should never change".

Re:What if writing tests was a joyful experience? by MichaelBurge on Sunday January 15, 2023 @07:15AM (#34388830)
If it's likely to change, then you especially want the regression test so you can decide how to handle the divergence during your port. Maybe one library preserves the signal on NaNs and the other doesn't. Or maybe the CPU's default rounding mode is different when called in this context, and you're off by 1 ulp.

In either case, if the behavior is to change, it should change as an informed decision and not because nobody noticed.

Re:What if writing tests was a joyful experience? by gavinray on Saturday January 14, 2023 @09:45AM (#34380671)
Lol I came here to post this but you beat me.