All you need is a bit hyperbolic, because you also need to quarantine flaky tests and other things, but coverage as a whole I think is useful only if you have an engineering organization that doesn't see the point in tests - but that's going to be an uphill battle.
In other words, first bug-fix PR should be hacky af to fix the bug. No refactoring, nothing controversial (other than the hacky af fix). After you verify the fix in production, then, and only then, do you open a PR to refactor the code. Finally, after that is verified in production, close the bug ticket.
I think coverage stats are always useful as they help find the edge cases that people forgot to test. A common culprit I've seen is error handling code where a bunch of tests target the happy path, but nothing tests the error logging when something breaks.
Another comment here mentioned mutation tests which could be a solution to increase quality of unit testing, but I've never seen anyone to actually use it in enterprise development. Same story with test driven development concept.
It is technically correct. But, it is only meaningful if you assume a bad actor in the team who knowingly games the system, and a team who tolerates it.
At that point, your problem has nothing to to with code quality, nor is coverage meant to be a solution for it.
That being said, you don't need to assume a bad actor in the team to encounter the situation of having 100% code coverage with meaningless tests. Even the most well-meaning engineer can accidentally write tests that boil down to asserting "true == true" without proving anything about the code paths it touches. This isn't necessarily a cultural issue.
I'd even assert that, due to the languages in common use, it's very common to see these sort of "touch all the code paths but assert nothing about the correctness of the code" tests. Standard OOP languages like TypeScript or Java for example have relatively limited type systems and allow mutable variables, so you end up with implementations of algorithms and data structures that don't lend themselves to property-based testing. This leads to tests which basically just duplicate the implementation and assert "I wrote what I wrote" or in other words "true == true".
Indeed. The thing is, I was replying to a comment that referred _explicitly_ to a bad actor. The situation @ponector describes (team's coverage indicator is rendered useless because have a bad actor games it), THEN the cause is not coverage. It says nothing about coverage beyond "a tool only works in certain conditions".
You bring 2 more cases that break the indicator, and I agree with both (there are more!). We have (1): We all make mistakes and write dumb tests. (2): Coverage is not useful for _some_ practises / scopes of testing.
I agree on both. But we're back on the same place. I'm not saying those problems don't exist. I'm saying that those are not problems coverage ever claimed to solve. Making a sweeping dismissal of a tool because it doesn't solve problems it never claimed to solve is throwing away the baby with the dirty water.
* Coverage does not claim to be a tool to fix bad actor, (1) or (2)! There are other tools to cover those risks (e.g. managers, code reviews, pair programming, etc.).
* Discussions about those tools (code reviews, etc.) tend to make the same mistake. Find problems the tool doesn't claim to solve to dismiss the tool.
* This all happens because people pretend to treat tools like coverage, tests, DORA metrics, as silver bullets. They are not. They are all meant to be a toolbox that engineers evaluate and use where they yield value.
And this is why yes, a lot of the "$tool is useless because $situation_where_it_doesnt_work" conversations are fundamentally about cultural issues. If your team uses a tool without knowing what problem is trying to solve, you have a cultural issue. If your tool has an actual purpose, and yet engineers are intentionally working around it, you have a cultural issue. Etc.
> 100% tells you nothing - but that _any other percentage_ does tell you something.
100% doesn't tell you it is meaningful coverage, but less than 100% tells you for sure that uncovered part doesn't have meaningful coverage.
Once you get there you already fucked up. In Java covering 100% lines means in average case testing Lombok and every equals hash. If you're doing that you fucked up.
My other favourite of trivial code that's broken: returning the same Iterator instance in Iterable.iterator()
I remember hibernate recommending using static hash. In order to prevent saving entities changing hash values.
What I also do, is ensure test coverage is over 100%¹ for important parts. Important is designated through churn (if a file or class os changed on every second commit, it must be important) and through domain knowledge (building a recipe app, then likely the Recipe is important).
¹ covered by unit tests, AND (partially) covered by integration AND by E2E tests.
That’s a stupid rule and not measuring what you think it is.
It fails if you just delete tested code.
It fails if you remove some code thats tested and add some tested code, but not enough.
These are all extremely common for any refactor. Here is a simplistic example but imagine the same principles applied to a large set of changes. Dozens of files, thousands of lines. You cant just manually account for that.
50%: 100 lines of code. 50 covered, 50 uncovered.
remove some function and its 44% from 40 covered, 50 uncovered. Failed.
Or remove some function and replace it with something better thats not as long. 44 covered, 50 uncovered. 47% coverage. Failed.
A stack overflow post about this.
https://softwareengineering.stackexchange.com/questions/4007...
This inevitably leads to worthless tests to increase coverage and avoiding optionL refactors.
If I had to pick one rule that is great to always follow, is when you get hit with a bug, write tests that catch it and then write code to fix the test.
Aside from preventing said bugs from resurfacing, it forces coverage of code that is likely complex enough to be buggy.
This rule has been a problem for me when deleting code. On my precious check we had an automated checker that wouldn't let us merge (easily) if we broke this rule. We had mediocre coverage. Some parts of the code had lots of tests, some parts were totally uncovered. Many times I had to delete code in the tested parts, and the checker would complain that total coverage went down because I deleted a tested line without adding more tested lines.
But I agree it's a good rule in general, which is why we kept it despite the occasional hiccup.
Various people objected to this, pointing out that 100% test coverage tells you nothing about whether the tests are any good. Our lead (wisely, IMO) responded that they were correct - 100% tells you nothing - but that _any other percentage_ does tell you something.