Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it.
I highly recommend investing in mutation and fuzzy testing.
[1]:https://pitest.orgFor Python we use mutmut [1].
I think it's pretty important that we who know about MT and have used it keep talking about it. It's a pretty great tool that more people should have in their tool belt.
This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program.
Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives 100% code coverage but it covers at the very best only 25% of the code behaviors.
> Full path coverage, of the type described above, is usually impractical or impossible. ... a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[14] Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage."
Coverage can prove that code is not tested but cannot prove that code is tested.
Tests can prove that bugs exist but cannot prove that bugs do not exist.
#1 source will be the code that doesn't appear at all (because the test build system didn't even include it) #2 will be all those red lines.
Right on point. High CC% doesn't tell application is well tested but is an indicator of what isn't covered
While there are uncovered lines, you have a TODO list that has been generated by the machine, telling you what needs more attention. If you go to that section intending to test it well, you’ll get a lot of lines of test code for a modest increase in coverage. If you’re just trying to boost coverage, you’ll check off the todo without accomplishing much.
The only place these two metrics line up at all is when you can keep the number of conditions in each method close to one, and the functions pure. Otherwise you have a bunch of little functions that accumulate state and you’ve just moved the combinatorics where it’s even harder to reason about.
I think the most you can say with any certainty is that coverage percentage should not go down when adding code. But staying flat doesn’t prevent things from getting worse.
Being forced into stupid tricks like adding five useless log statements so that the CI tool doesn't block your change is ... annoying.
Probably not the test you want to have though.
The same way you should make an expected “no such object” result a 200 status code, so that she does not have to stand up and justify her error rate.
My point is rzimmerman never contextualized this as a "how to navigate the workspace" argument, they are arguing it from a pure technical perspective.
Absolutely correct. But at the same time, <100% coverage means you’ve got code where you can’t be sure there are no bugs because it doesn’t even get tested.
Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.
You're asking this rhetorically, but I often find interesting problems once I start really exercising the latter, like:
- Non-global collectors we forgot to register, or fail to get registered on some code paths.
- Collectors that get registered twice or overlap with other collectors only in some conditions (e.g. connection addresses that are sometimes the same and sometimes not depending on how your service gets distributed).
- Collectors that are not really safe scraping concurrently with the rest of the program; our test suites include race detectors.
- Metrics that don't follow idiomatic naming conventions (can be found via e.g. `promtool check metrics`).
- Metrics with legitimate difficult-to-notice bugs. We had an underflow counting the size of a connection pool because `Close()`ing one twice was explicitly allowed.
> Testing trivial code brings negative value
Agreed in the abstract, but if we were good judges of what code was trivial or not, we'd write a lot less bugs in the first place!
That being said, this is also a discussion about coverage, not assertions. Even if I'm not actually checking any of those things, those code paths (and logging) are still getting exercised via any decent E2E/functional/behavioral test suite.
For example, here's a case where I incremented the wrong metric:
https://github.com/jrockway/alertmanager-status/commit/fccae...
I noticed the bug when I went to look at a dashboard with that metric on it, and noticed it had the wrong name. If go's Prometheus library had an easy way to run "metric.CurrentValue()", I would have tested it... but it didn't, so I didn't. And then had to patch it, release it, and update all my servers. Writing the test (with that API) would have taken less than a second. Finding the bug in production and fixing it took an hour.
(That codebase has 100% line coverage, but of course, I know that 100% test coverage is a meaningless metric. That the code ran is a prerequisite to the program behaving correctly, but it also has to do the right thing.)
A small tip for the future: https://pkg.go.dev/github.com/prometheus/client_golang/prome....
Sure, why not? All that requires is running tests with the log level turned up. And it’s not unheard of that e.g. somewhere in a log statement someone forgot to check that the object they’re calling toString() on is not null.
A test that calls code (which counts as code coverage) does not have to verify that something specific happened. Often, this is a weakness of a test. The test does not verify what you think it would for various reasons...visibility, access to resources, etc.
They can stay red until the end of time for all I care.
Integration tests for the most common paths, unit tests for more complex logic.
Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort involved in mocking/wrapping/abstracting sometimes isn’t worth the effort for some code, like simple calls to common libraries, reading a file, CLI help strings, etc. This stuff generally is better served by end-to-end or integration tests. Or just gets exercised naturally as part of development. Or can crash without much consequence.
My point is coverage is a tool that tells you where to look, not the one and only metric that tells you how reliable your code is.