Re:Lean Testing or Why Unit Tests Are Worse Than You Think
by ajb on Monday July 16, 2018 @12:01PM
(#17542346)
It's also worth looking at all-pairs testing.
The insight is that you can cover all pairs of a set of combinations in only O(N^2) tests rather than O(N^D) irrespective of the number of dimensions. The number of tests ends up being the product of the size of the two largest dimensions. So in your example you can cover all combinations of each pair:
* (lib A, lib B)
* (lib A, language runtime)
* (lib A, browser)
* (lib B, language runtime)
* (lib A, browser)
* (language runtime, browser)
Re:Lean Testing or Why Unit Tests Are Worse Than You Think
by Jach on Monday July 16, 2018 @12:24PM
(#17542523)
This is clever... is the N^2 behavior just due to nCr(n, 2) = n^2/2 - n/2 growing quadratically?
Re:Lean Testing or Why Unit Tests Are Worse Than You Think
by AllegedAlec on Monday July 16, 2018 @09:07AM
(#17540971)
Fair enough; in this case unit testing is not very useful. I was more thinking about business logic when I wrote that example.
In practice I've also run into a lot of combinatorial explosions for which unit testing is worse than useless. E.g. say you have a library that has two dependencies and interacts with browsers you can have:
* 30 different versions of library A
* 50 different versions of library B
* 25 different versions of the language runtime
* 30 different browsers
That's over a million different potential variations for every test. In practice though, there's a sharp power law. You don't have to test all of those variations, you just have to randomly throw variations at the code until bugs stop cropping up. 99% of the bugs you find probably be found in the first 1,000 iterations.