There was a technique that existed already where you could use `go test -cover` and the `-o` flag to produce a binary from `go test` rather than actually running tests. So you could build a binary that had coverage enabled. Then when you exercised code paths by running that binary with whatever inputs, coverage would be collected and dumped into some local file.

Here's an example of the build step: https://github.com/multiprocessio/datastation/blob/main/runn....

And here's how I'd analyze coverage after using the binary: https://github.com/multiprocessio/datastation/blob/main/runn.... This analysis actually combined both unit test coverage files with integration test coverage so it gave a pretty good overall indication of code paths covered by any tests.

I can't remember where I found this technique but it's been around for a while.

This new option is the same thing but a way to `go build` with `-cover` instead of `go test -cover -o $out`? Do I have that right?

Another example of doing this from 2019: https://stackoverflow.com/a/55640874/1507139. I'm sure I learned about this from somebody's blog but I can't remember where.

No idea if the technique was original, it’s been too long, but I wrote about it on the Cloudflare blog in 2016. https://blog.cloudflare.com/go-coverage-with-external-tests/

Anyway, very happy to see the team ship first class support for it. It immediately made coverage support in testscript much more natural, too. https://github.com/rogpeppe/go-internal/pull/201

Elastic wrote about that in 2015: https://www.elastic.co/blog/code-coverage-for-your-golang-sy.... I guess multiple people figured it out independently.

Yes, technically you could wrap every `main` in a test and build everything just right, but it's nice to be able to do this without any hacks if you already have an integration test suite and you want to know what coverage it gets you.

In case you have never used it, https://app.codecov.io/ can watch your GitHub repo and run tests + coverage. There's a nice UI to then see the coverage % for each folder and also whether a PR or commit is increasing or decreasing coverage.

If I recall correctly, there's zero configuration needed for Go projects. You can be up and running in minutes.

I think Codecov is free for open source projects and paid for private repos, someone more knowledgeable can chime in.

♥ thank you! If y’all have feedback for what we can do better with Codecov let us know.

I wish your “redesigns” did not remove useful features: https://app.codecov.io/gh/feedback/p/view-source-with-flags

This is a really cool feature, thanks very much for adding it. I'd love to see this in more languages. I'd love to use it in my Rust integration tests.

A bit more convoluted, but it's also possible in Rust

https://github.com/pldubouilh/rust-coverage-integration-test

Go tooling keeps getting better

Cool. I definitely felt the lack of this.

I find the value of code coverage goes down when the size of the instrumented codebase goes up. It's helpful for smaller tests as you can eyeball where you may be missing cases but on integration tests you end up getting a high coverage percentage just by bringing up the system and not even asserting anything (I guess it doesn't crash :) ).

If you want to methodically measure integration test coverage it seems more valuable to enumerate all your critical features or user flows and attribute test cases to them.

Proper testing is your issue, not coverage. Coverage is to know where it’s missing. Proper testing (positive AND negative cases) is extremely valuable. Not only to ensure your code works properly, but also as documentation on how to use your APIs.

Integration testing is where most testing cycles fail. It’s also more difficult for the developer because now they need an emulator, a container, or some sandbox to test the integration.

Feature testing is one thing but I’d like to know if the foundation works, that the plumbing is there.

A colleague of mine wrote a tool to make the process of dealing with coverage data a bit simpler: https://github.com/gotesttools/limgo

It is (non-sarcastically) good to see the Go team changing their opinion on the usefulness of it: https://github.com/golang/go/issues/16537

I guess this can be used to eliminate dead code from your projects as well. Cases that you thought were executed but, in reality, are not.

I am curious if the overhead of this feature allows for running it in production for some time, let’s say a day.

A few years ago, I used goc to collect coverage from multiple services at the same time when they were used in E2E tests. I was very happy with it. Check it out!

https://github.com/qiniu/goc

Oh, interesting. Full on integration testing for Go CLI apps is an area in which I am sorely lacking.

Does anyone have recommendations of tooling to generally help instrument testing CLI tools? Hopefully better than what I’ve done thus far which is basically a shell script of

    ./mybinary <<< $INPUT > output.txt 2> error.txt
    diff expected-output.txt output.txt
    diff expected-error.txt error.txt

Could we hack this to better understand what your code is doing in prod?

Let me describe a scenario:

Let's say you just inherited some service that is already running in production but it don't have a great documentation nor a good test suite. To make matters worse you don't really know how this service is being used and what are the critical paths inside it.

What I'm thinking is if it would be possible to use this feature to collect some real world usage data for this application. And this data could give you a better understanding about what is actually important in the service.

For APIs we already have a part of this data, as most server in Go have the number of requests for each endpoint. But I think in many cases a more granular data could be really helpful.

Yes, I know that the number of times some function was called isn't a reliable indicator of the importance of said function. But it should be better than nothing.

And I also know that something like this would have a negative performance effect that would not be acceptable for some situations.