News for hackers. Stuff that matters.
Re:There are only four billion floats, so test them all (2014) by seanmcdirmid on Sunday October 11, 2020 @12:25PM (#24746900)
The only high fidelity mantra in testing is a bit misguided. If you only test your system in as close to production state as possible, you are never going to encounter stress states that would only very rarely appear in production. Seeing how your components independently deal with less ideal components in the system can tell you a lot about how they might handle rare but bad cases in production, and allow you to over engineer them a bit more to avoid that.

> which makes it even more likely that introducing never-used-in-production behaviour unexpectedly perturbs the system.

Isn’t that a good thing? Isn’t the goal to deal with the unexpected in the first case?

Re:There are only four billion floats, so test them all (2014) by jniedrauer on Sunday October 11, 2020 @12:40PM (#24747026)
Tests come with a non-zero cost. They're code that needs to be maintained. Often complex code, with poor documentation. I don't think the value provided by testing states that you will never actually encounter in production outweighs the cost of maintaining the tests.
Re:There are only four billion floats, so test them all (2014) by tikhonj on Sunday October 11, 2020 @01:18PM (#24747383)
This approach works well until you do encounter a situation like that in production :). It's worth testing conservatively because our understanding of our systems is imperfect and because assumptions that may be valid now are likely to change in the future. It's especially true for code that might operate at scale or has security considerations, because both of those can magnify the effects of conditions that occur with arbitrarily low probabilities.

Personally, I see tests as not just a way to prevent bugs, but as a way to learn about the system being tested—both how it behaves now and, continually, how it behaves in future iterations. In this specific case, understanding what happens if two URIs hash to the same value even if they don't "really" do that is an important bit of information: it tells us not only what would happen with a real hash collision but also how our system would behave if there were a bug either in the hashing algorithm itself or in the code that connects the hashing algorithm to the component that uses the hashes.

What happens if we introduce a caching layer between where the hash is generated and where it's used, and there's a cache invalidation bug? I don't know how realistic that is in this specific hypothetical, but I'm sure there are other performance optimizations with the same bug potential that I'm not even thinking of. Changes like that can lead to absolute debugging nightmares! Testing even "impossible" edge cases helps catch this in a systematic way, without needing full knowledge of which conditions might matter.

Of course, none of this talks to the trade-off you pointed out: test code does have a real cost. That is absolutely something to consider. I just don't think the answer is to draw a hard line at "don't test situations that can't come up in production"—and, perhaps naively, I think the "real" solution is less about deciding what is and isn't worth testing and more about reducing the costs of testing more. Write application code in a way that's easy to test and treat your test code as code—keep it clean, include comments, and invest time in setup and tooling to make your tests as easy to write and maintain as possible.

Re:There are only four billion floats, so test them all (2014) by seanmcdirmid on Sunday October 11, 2020 @12:54PM (#24747155)
To achieve “defense in depth,” testing components in degenerate non-production situations (and then over engineering them to gracefully deal with those situations) is the only way to get there. Of course, we aren’t building bridges, so maybe it’s less important.
Re:There are only four billion floats, so test them all (2014) by TickleSteve on Sunday October 11, 2020 @01:09PM (#24747300)
Thats not really a "defense in depth" situation.

Defense in depth is regarding layering different types of security, not targeted testing.

https://en.wikipedia.org/wiki/Defense_in_depth_(computing)

Re:There are only four billion floats, so test them all (2014) by nitrogen on Sunday October 11, 2020 @01:15PM (#24747341)
It's an analogy. Multiple layers of defense against bugs.
Re:There are only four billion floats, so test them all (2014) by kortilla on Sunday October 11, 2020 @02:19PM (#24747842)
> The only high fidelity mantra in testing is a bit misguided.

That’s understating it. It’s incompetence because it’s an excuse to not test a huge surface area of edge cases.