Most of us aren't working on life or death code like this. My React app doesn't need 100% code coverage but you put it well when you said "There should be just enough to feel safe"
But you're not saying the right words here. 100% doesn't mean you've covered every edge case (although I suppose 60% necessarily means you _haven't_). I can hit 100% without actually asserting anything.
I think it's harmful to talk about 100% without also considering mutation testing, boundary analysis, manual mutation testing...
The senior engineers were not supportive of other forms of testing in any way (even the user testing). They flat out refused my proposal to attempt to create an integration test suite to cover it's communication to other services as well as other services consuming it's data (this I started over the weekend and was told to proceed no longer).
tl;dr: You're absolutely correct but I still think my testing procedures were better than nothing.
I am sure they were, but you also said:
>take the time to understand why they want 100% coverage
which to me indicates that they are under the false impression that this gives some sort of completeness, i.e. "make sure this plastic fence covers the entire stretch of mountain road".
We have a coverage threshold of 90% and it is helpful because it works as a reminder if you try to check in code that isn't covered by tests. I think that pushing that to 100% would add little value but a lot of overhead. Never tried it though so I fully admit that it's just my hunch.
Exactly. You are in the tiny minority writing literally life-and-death code, and for which a lot of other common advice given regarding general software development likely does not apply either (like "move fast and break things".) Also, for your type of application I would probably want 100% state space coverage too.
Keep adding tests as long as they tell us something new, give us more confidence, document some regression, etc.
Keep adding tests if they're exposing edge cases (e.g. MAX_INT), even if those code paths are already covered.
Keep adding tests if they're asserting useful things about the results, even if those code paths are already covered.
Stop adding tests when you're only trying to make the coverage number increase.
- 0-80% was adding useful tests.
- 80-98% was mostly useless.
- 98-100% found some really interesting bugs and forced refactoring that made testing easier, and was definitely worth it.
100% coverage still does not mean 100% of all cases tested - it just means that every line has been run with SOME data and every conditional statement was run once in both directions.
It also depends on the service. Some services are critical infrastructure and should have their edge cases tested. Some services provide non essential functions and can get away with less.
I agree that there is no magic number, but 100% coverage is clearly overkill for the average engineering team. In fact, broadly demanding any code coverage percentage is probably over simplifying the issue. It just shouldn't be 0 :)
The problem with 100% coverage that I think of is that the last 30% is often boilerplate code (generated getters/setters in Java and that kind of thing).
But what if an engineer started with the boilerplate and only then progressed to tests that are actually important? You might get to 30% without testing anything useful at all. Then you might test half of the important stuff and hit an arbitrary metric of 70%.
If someone were ruled by the sonarqube score, they might even do this.
Yeah, thinking about the why matters more than the %.
As others mentioned it also doesn't say anything about state space coverage or input argument space coverage or all combinatorial paths through your code. Only that some line/branch has been hit at least once.
The decidability/halting problem hints that perfect testing is an impossibility for systems of any complexity, as in not just too big of a big-O... it's flat out not possible on a turing machine of any power.
That doesn't mean give up all testing, but there is a LOT of dogma in testing philosophy.
I hope many managers and programmers out there don't take this the wrong way. I've been an engineer on a project that was attempting to get 100% code coverage on a piece of software I was writing. I heard constant remarks during this period that were similar to "You don't need 100% code coverage, it doesn't do anything!" These engineers who I was working with had only read articles like this and didn't stop to think about what the article was trying to say. From my experience there is no safe rule of thumb for for how many tests should be implemented for a project. There should be just enough to feel safe (as hathawsh has said). If you're recommending to engineers on your team to stop implementing tests when they say "I'm at 60% coverage and it'll take ~2 days to get to 100%" I'd really hope you take the time to understand why they want 100% coverage.
The software I was working on when my coworkers started telling me to stop writing tests was code designed to trigger alarms when patient's vital signs met certain criteria set by doctors. I am very thankful that I did hit 100% coverage because between 60% and 100% there were many small edge cases that, had they caused a death, I wouldn't have been able to sleep well. Had they said I was explicitly disallowed from working on the tests I would have come in on a weekend (or taking a PTO) and implemented them then. It's our ethical responsibility to know when and where paranoia is worth the marginal time penalty.