If you have something like this:
if condition1:
do_something1()
if condition2:
do_something2()
if condition3:
do_something3()
There are 8 possible paths for the code to follow here, but you can cover 100% of lines of code in one test. If your code coverage measure tells you that testing the case where conditions 1, 2, and 3 are all true achieves 100% coverage, your measure is worthless. Covering this code cannot be done in less than eight tests.But in reality, if those branches have any interaction, you would need to write eight test cases for every combination of branches being run and not being run.
Take this simplified code:
if condition1:
do_something()
You need one test for code coverage and two for branch coverage, but if there's a bug in the code as written you'll find that you actually need more. Assume that we want to run do_something() when, and only when, condition2 is true.Now we have four cases:
- condition1 && condition2: do_something() runs, which is correct.
- condition1 && !condition2: do_something() runs, which is incorrect.
- !condition1 && !condition2: do_something() does not run, which is correct.
- !condition1 && condition2: do_something() does not run, which is incorrect.
If you happened to write tests for the first and third of those options, your tests will make it look like your code works, and your (official) coverage metrics will look comprehensive. But because your coverage metrics (in reality) are actually terrible, your code is secretly not working.
The big problem we've just run into is that the number of test cases we need depends on the number of bugs in our code. If the goal of writing tests is to prove that we don't have bugs, this is a terrible result; it means our goal is fundamentally impossible to achieve. We don't know whether we have bugs, so we don't know how many tests we need.
Your point is a good one, but not where I was going.
That’s because code coverage doesn’t find the bugs that result from code you didn’t write, but should have. Code coverage is but one measure, and to treat it as the measure is folly.
(But, yes, I have heard a test manager at a large software company we’ve all heard of declare that test team was done because 100% coverage.)