News for hackers. Stuff that matters.
Re:Lean Testing or Why Unit Tests Are Worse Than You Think by AllegedAlec on Monday July 16, 2018 @07:53AM (#17540536)
> Unit test thus get written for part that we absolutely want to be maximum certain of that it works correctly.

I think this is a partial answer.

Yes, I write unit tests for such pieces of software. However, what I also write a lot of unit tests for a pieces of software which are hard to write good integration test for because of combinatorial explosion. If I have a part for integration method which consists of a chain of 4 methods, each of which can have 5 different results, that'd mean I'd have to write 4^5 integration tests, which is clearly not possible. If I were to write unit + integration tests, however, I could do with 4x5 unit tests plus a couple of integration test which prove that the overall combination of those methods work well together.

Re:Lean Testing or Why Unit Tests Are Worse Than You Think by crdoconnor on Monday July 16, 2018 @08:19AM (#17540683)
Combinatorial explosions are best mitigated by doing property based testing, design by contract and creating clean, low level abstractions which you test against.

In practice I've also run into a lot of combinatorial explosions for which unit testing is worse than useless. E.g. say you have a library that has two dependencies and interacts with browsers you can have:

* 30 different versions of library A

* 50 different versions of library B

* 25 different versions of the language runtime

* 30 different browsers

That's over a million different potential variations for every test. In practice though, there's a sharp power law. You don't have to test all of those variations, you just have to randomly throw variations at the code until bugs stop cropping up. 99% of the bugs you find probably be found in the first 1,000 iterations.

Re:Lean Testing or Why Unit Tests Are Worse Than You Think by ajb on Monday July 16, 2018 @12:01PM (#17542346)
It's also worth looking at all-pairs testing. The insight is that you can cover all pairs of a set of combinations in only O(N^2) tests rather than O(N^D) irrespective of the number of dimensions. The number of tests ends up being the product of the size of the two largest dimensions. So in your example you can cover all combinations of each pair:

  * (lib A, lib B) 
  * (lib A, language runtime)
  * (lib A, browser)
  * (lib B, language runtime)
  * (lib A, browser)
  * (language runtime, browser)
in 50*30 = 1500 tests (or a little more to make the analysis tractable)
Re:Lean Testing or Why Unit Tests Are Worse Than You Think by Jach on Monday July 16, 2018 @12:24PM (#17542523)
This is clever... is the N^2 behavior just due to nCr(n, 2) = n^2/2 - n/2 growing quadratically?
Re:Lean Testing or Why Unit Tests Are Worse Than You Think by AllegedAlec on Monday July 16, 2018 @09:07AM (#17540971)
Fair enough; in this case unit testing is not very useful. I was more thinking about business logic when I wrote that example.