News for hackers. Stuff that matters.
Re:The Myth of Code Coverage by rzimmerman on Friday December 04, 2020 @02:20PM (#25306485)
Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code.

Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort involved in mocking/wrapping/abstracting sometimes isn’t worth the effort for some code, like simple calls to common libraries, reading a file, CLI help strings, etc. This stuff generally is better served by end-to-end or integration tests. Or just gets exercised naturally as part of development. Or can crash without much consequence.

My point is coverage is a tool that tells you where to look, not the one and only metric that tells you how reliable your code is.

Re:The Myth of Code Coverage by g14i on Friday December 04, 2020 @02:29PM (#25306584)
Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs.

Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it.

I highly recommend investing in mutation and fuzzy testing.

Re:The Myth of Code Coverage by whymarrh on Friday December 04, 2020 @02:32PM (#25306614)
I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together.

  [1]:https://pitest.org
Re:The Myth of Code Coverage by TeaVMFan on Friday December 04, 2020 @09:11PM (#25311000)
Mutation coverage definitely improves the testing process. It is much harder to accidentally write bad tests once you are using a tool like PIT: https://frequal.com/java/HighQualityTestsWithPitMutationTest...
Re:The Myth of Code Coverage by g14i on Friday December 04, 2020 @02:44PM (#25306788)
whymarrh, great that you mentioned PIT. This is exactly what we use for Java as well.

For Python we use mutmut [1].

[1] - https://pypi.org/project/mutmut/

Re:The Myth of Code Coverage by boxed on Friday December 04, 2020 @04:05PM (#25307970)
Mutmut author here. Great to hear you're using mutmut!

I think it's pretty important that we who know about MT and have used it keep talking about it. It's a pretty great tool that more people should have in their tool belt.

Re:The Myth of Code Coverage by ay on Friday December 04, 2020 @03:45PM (#25307668)
test coverage is calculated as % of the locations seen by profiler during the test runs, right ?

This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program.

Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives 100% code coverage but it covers at the very best only 25% of the code behaviors.

Re:The Myth of Code Coverage by dalke on Friday December 04, 2020 @04:17PM (#25308135)
There are many types of test coverage measurement. I believe path coverage ("Has every possible route through a given part of the code been executed?") is your latter example. Quoting https://en.wikipedia.org/wiki/Code_coverage which also says:

> Full path coverage, of the type described above, is usually impractical or impossible. ... a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[14] Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage."

Re:The Myth of Code Coverage by Mindless2112 on Friday December 04, 2020 @02:45PM (#25306810)
I like this parallelism between coverage and tests:

Coverage can prove that code is not tested but cannot prove that code is tested.

Tests can prove that bugs exist but cannot prove that bugs do not exist.

Re:The Myth of Code Coverage by boxed on Saturday December 05, 2020 @01:46AM (#25312398)
Mutation testing can prove that the tests check all the behavior of the code, but it can't prove that the code shouldn't have more behavior, or that this behavior is correct.
Re:The Myth of Code Coverage by breck on Friday December 04, 2020 @03:57PM (#25307835)
It's a really helpful tool for as you say telling "you where to look". One trick I've learned: if you want to forecast where the bugs will popup next month, take a quick look at the code coverage report.

#1 source will be the code that doesn't appear at all (because the test build system didn't even include it) #2 will be all those red lines.

Re:The Myth of Code Coverage by jameson on Friday December 04, 2020 @03:45PM (#25307667)
> Coverage is more useful for finding chunks of code that aren’t exercised at all by tests.

Right on point. High CC% doesn't tell application is well tested but is an indicator of what isn't covered

Re:The Myth of Code Coverage by hinkley on Friday December 04, 2020 @07:24PM (#25310244)
And that’s why you shouldn’t chase after code coverage percentages.

While there are uncovered lines, you have a TODO list that has been generated by the machine, telling you what needs more attention. If you go to that section intending to test it well, you’ll get a lot of lines of test code for a modest increase in coverage. If you’re just trying to boost coverage, you’ll check off the todo without accomplishing much.

The only place these two metrics line up at all is when you can keep the number of conditions in each method close to one, and the functions pure. Otherwise you have a bunch of little functions that accumulate state and you’ve just moved the combinatorics where it’s even harder to reason about.

I think the most you can say with any certainty is that coverage percentage should not go down when adding code. But staying flat doesn’t prevent things from getting worse.

Re:The Myth of Code Coverage by zlynx on Friday December 04, 2020 @08:02PM (#25310544)
Except when handling an error condition from a hard to test system call that was previously ignored.

Being forced into stupid tricks like adding five useless log statements so that the CI tool doesn't block your change is ... annoying.

Re:The Myth of Code Coverage by voodootrucker on Saturday December 05, 2020 @12:18AM (#25311994)
You can hit 100% with zero assertions if you like.

Probably not the test you want to have though.

Re:The Myth of Code Coverage by regularfry on Saturday December 05, 2020 @01:02AM (#25312194)
Zero assertions means that you know there aren't any lurking exceptions triggered by just running the code. That's a good thing to know, but I'd hope like hell that it's not the only thing you want to know.
Re:The Myth of Code Coverage by joshschreuder on Saturday December 05, 2020 @01:35AM (#25312348)
It may not even prove that depending on your mock/fake strategy though.
Re:The Myth of Code Coverage by closeparen on Friday December 04, 2020 @06:18PM (#25309630)
When “hold people accountable to metrics” is not just your company’s management philosophy but the gospel of its entire surrounding culture, this is not an argument that gets anywhere. It is always better to just cover the damn lines than make your Director explain to the VP why he has worse coverage numbers than a sister org.

The same way you should make an expected “no such object” result a 200 status code, so that she does not have to stand up and justify her error rate.

Re:The Myth of Code Coverage by capableweb on Friday December 04, 2020 @06:46PM (#25309856)
Note that you're now discussing corporate politics, not just creating software, even though creating software often has to deal with corporate politics.

My point is rzimmerman never contextualized this as a "how to navigate the workspace" argument, they are arguing it from a pure technical perspective.

Re:The Myth of Code Coverage by xienze on Friday December 04, 2020 @03:29PM (#25307394)
> Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code.

Absolutely correct. But at the same time, <100% coverage means you’ve got code where you can’t be sure there are no bugs because it doesn’t even get tested.

Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.

Re:The Myth of Code Coverage by Tomis02 on Friday December 04, 2020 @04:18PM (#25308144)
Really? You want to strive to test that logging logs and observability observes? You want to test constructors, getters and setters? Testing trivial code brings negative value, why would you do that?
Re:The Myth of Code Coverage by morelisp on Friday December 04, 2020 @04:51PM (#25308615)
> You want to strive to test that logging logs and observability observes?

You're asking this rhetorically, but I often find interesting problems once I start really exercising the latter, like:

- Non-global collectors we forgot to register, or fail to get registered on some code paths.

- Collectors that get registered twice or overlap with other collectors only in some conditions (e.g. connection addresses that are sometimes the same and sometimes not depending on how your service gets distributed).

- Collectors that are not really safe scraping concurrently with the rest of the program; our test suites include race detectors.

- Metrics that don't follow idiomatic naming conventions (can be found via e.g. `promtool check metrics`).

- Metrics with legitimate difficult-to-notice bugs. We had an underflow counting the size of a connection pool because `Close()`ing one twice was explicitly allowed.

> Testing trivial code brings negative value

Agreed in the abstract, but if we were good judges of what code was trivial or not, we'd write a lot less bugs in the first place!

That being said, this is also a discussion about coverage, not assertions. Even if I'm not actually checking any of those things, those code paths (and logging) are still getting exercised via any decent E2E/functional/behavioral test suite.

Re:The Myth of Code Coverage by jrockway on Friday December 04, 2020 @05:14PM (#25308935)
I prefer to test everything. The parts you don't test are the parts that break.

For example, here's a case where I incremented the wrong metric:

https://github.com/jrockway/alertmanager-status/commit/fccae...

I noticed the bug when I went to look at a dashboard with that metric on it, and noticed it had the wrong name. If go's Prometheus library had an easy way to run "metric.CurrentValue()", I would have tested it... but it didn't, so I didn't. And then had to patch it, release it, and update all my servers. Writing the test (with that API) would have taken less than a second. Finding the bug in production and fixing it took an hour.

(That codebase has 100% line coverage, but of course, I know that 100% test coverage is a meaningless metric. That the code ran is a prerequisite to the program behaving correctly, but it also has to do the right thing.)

Re:The Myth of Code Coverage by morelisp on Friday December 04, 2020 @06:37PM (#25309785)
> If go's Prometheus library had an easy way to run "metric.CurrentValue()"...

A small tip for the future: https://pkg.go.dev/github.com/prometheus/client_golang/prome....

Re:The Myth of Code Coverage by jrockway on Friday December 04, 2020 @06:55PM (#25309961)
Thanks! I've seen that before, but didn't realize that ToFloat64() was what I was looking for.
Re:The Myth of Code Coverage by tcbasche on Saturday December 05, 2020 @04:10AM (#25312996)
Arguably you should be testing behaviour rather than getters and setters. If a codepath requires a name formatted correctly, then that forms the basis of a test and an assertion. Otherwise you're introducing fragility by coupling your 'under the hood' stuff with your 'API' (so to speak).
Re:The Myth of Code Coverage by xienze on Friday December 04, 2020 @05:08PM (#25308844)
> You want to strive to test that logging logs and observability observes?

Sure, why not? All that requires is running tests with the log level turned up. And it’s not unheard of that e.g. somewhere in a log statement someone forgot to check that the object they’re calling toString() on is not null.

Re:The Myth of Code Coverage by nickcox on Friday December 04, 2020 @04:31PM (#25308335)
You probably don't need to test any of those things but I would still expect them to be exercised. You can't very well test a class without constructing it.
Re:The Myth of Code Coverage by Supermancho on Friday December 04, 2020 @05:53PM (#25309406)
> You want to strive to test that logging logs and observability observes?

A test that calls code (which counts as code coverage) does not have to verify that something specific happened. Often, this is a weakness of a test. The test does not verify what you think it would for various reasons...visibility, access to resources, etc.

Re:The Myth of Code Coverage by theshrike79 on Saturday December 05, 2020 @02:27AM (#25312597)
I'm pretty sure all those IDE generated getters and setters will work flawlessly even if I don't add unit tests for them.

They can stay red until the end of time for all I care.

Integration tests for the most common paths, unit tests for more complex logic.

Re:The Myth of Code Coverage by 1337shadow on Friday December 04, 2020 @06:01PM (#25309485)
Another advantage that does not get the love it deserves in my opinion, is how useful coverage becomes when it comes to upgrading a dependency, or even the language version. But your project will have to stick around for a while to appreciate that.