runtime/coverage: coverage from other binaries is not collected in tests #60182

FiloSottile · 2023-05-14T09:22:19Z

What version of Go are you using (`go version`)?

$ go version
go version go1.20.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes, with devel go1.21-91b8cc0d.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/filippo/Library/Caches/go-build"
GOENV="/Users/filippo/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/filippo/pkg/mod"
GONOPROXY="github.com/FiloSottile/*,filippo.io/*"
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/filippo"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/opt/homebrew/Cellar/go/1.20.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.4/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/Users/filippo/src/filippo.io/litetlog/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_j/hq4ytn1n4b94fhrpvvb9tktr0000gn/T/go-build1323402631=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Within a Go test, built a binary from a different package in the same module with -cover and executed it.

Ran the test with -coverpkg=./....

What did you expect to see?

Coverage of the command invocation included in the test summary.

What did you see instead?

coverage: 0.0% of statements in ./...

I expected this to work because there is explicit support for it in the testing package, which propagates the temporary GOCOVERDIR to the environment.

go/src/cmd/go/internal/test/test.go

Lines 1326 to 1330 in 91b8cc0

    
           // Even though we are passing the -test.gocoverdir option to 
        
           // the test binary, also set GOCOVERDIR as well. This is 
        
           // intended to help with tests that run "go build" to build 
        
           // fresh copies of tools to test as part of the testing. 
        
           addToEnv = "GOCOVERDIR=" + gcd

Indeed, rogpeppe/go-internal#201 takes advantage of this to run the test binary as a subprocess and collect coverage from that invocation with the coverage of the main test run.

However, there is then code to exclude coverage from other binaries, introduced for #57924.

go/src/runtime/coverage/testsupport.go

Lines 84 to 96 in 91b8cc0

    
           // Generate the expected hash string based on the final meta-data 
        
           // hash for this test, then look only for pods that refer to that 
        
           // hash (just in case there are multiple instrumented executables 
        
           // in play). See issue #57924 for more on this. 
        
           hashstring := fmt.Sprintf("%x", finalHash) 
        
           for _, p := range podlist { 
        
           	if !strings.Contains(p.MetaFile, hashstring) { 
        
           		continue 
        
           	} 
        
           	if err := ts.processPod(p); err != nil { 
        
           		return err 
        
           	} 
        
           }

I think running integration tests within a Go test is a more idiomatic way to do it than rigging a script with go tool covdata invocations, and it's nice that GOCOVERDIR gets already set up and propagated automatically, but the pattern then breaks when the coverage data is not collected.

As an aside, it should be documented somewhere how test coverage is propagated and collected.

The text was updated successfully, but these errors were encountered:

seankhliao · 2023-05-14T10:39:57Z

cc @thanm

ankitjain1510 · 2023-05-17T05:43:11Z

@seankhliao @thanm My use case is that I need to generate the code-coverage from a containerised Go application running inside Kubernetes Pod after running the integration test. This application (server) is long running and doesn't terminate.
I need a way to emit the coverage out after running my integration test.

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

FiloSottile · 2023-05-17T14:29:54Z

@ankitjain1510 That sounds unrelated to filtering of coverage collected by the testing package, so I suggest you open a separate issue for that, or ask on golang-nuts.

thanm · 2023-05-17T15:19:24Z

@ankitjain1510 collecting snapshots of coverage from server binaries is a bit different as @FiloSottile mentioned. We have APIs for doing this, but the documentation for them is not written/published (please bear with me on that). Thanks.

thanm · 2023-05-17T16:59:41Z

Hi @FiloSottile

Thanks for the report. It might be helpful for me to understand your high-level goals here in a bit more detail.

From your description it sounds like you have a module with that has a package P with tests, and in the test code you use "go build -cover" to build some sort of auxiliary binary (which might import P), then you would like to see the coverage effects of running the binary reflected in the coverage percentage numbers for P, something like that?

Is your main item of interest the coverage percentages or are you looking at text profiles as well?

It seems important to ensure that when a "-cover" test binary executes and writes out coverage stats or profiles, the raw material for that report comes only from running the test.

So for example, if you were to do something like

$ mkdir /tmp/xyz
$ go build -o myapp.exe -cover myapp
$ GOCOVERDIR=/tmp/xyz ./myapp.exe
$ GOCOVERDIR=/tmp/xyz go test -cover somePackage
...
$

it is important that the coverage numbers reported for "somePackage" don't incorporate any of the data files that might be in /tmp/xyz. These sorts of concerns are what motivated the testsupport.go code that you hlghlighted.

ankitjain1510 · 2023-05-24T05:42:36Z

@ankitjain1510 collecting snapshots of coverage from server binaries is a bit different as @FiloSottile mentioned. We have APIs for doing this, but the documentation for them is not written/published (please bear with me on that). Thanks.

Thanks for the response @thanm. Do we have any timeline available by when the documentation will be available?

enrichman · 2023-05-28T00:33:27Z

@seankhliao @thanm My use case is that I need to generate the code-coverage from a containerised Go application running inside Kubernetes Pod after running the integration test. This application (server) is long running and doesn't terminate. I need a way to emit the coverage out after running my integration test.

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

We have the same need, and at the moment we are using the same trick of pre-1.20 version. Our server is checking the GOCOVERDIR env var, if set we are adding a /exit route that will be used to quit the server.

Not sure if the new APIs will simplify this workflow but it will be neat.

FiloSottile · 2023-05-30T16:22:07Z

From your description it sounds like you have a module with that has a package P with tests, and in the test code you use "go build -cover" to build some sort of auxiliary binary (which might import P), then you would like to see the coverage effects of running the binary reflected in the coverage percentage numbers for P, something like that?

Is your main item of interest the coverage percentages or are you looking at text profiles as well?

Yep pretty much. I have testscript tests that build a set of commands from my module and use them all together in test scripts (for example, use foo-keygen to make a key and then run a server with that key and then use fooctl to inspec tthe database) and I would like to get aggregate coverage profiles.

I understand wanting to ignore spurious entries in GOCOVERDIR. Maybe that can be achieved by taking a snapshot of existing files before executing the test and then only aggregating new ones?

Anyway it's weirdly inconsistent that sub-invocations of the same test binary leads to profiles that get aggregated but sub-invocations of other binaries doesn't. Even invocations of the same test binary might be from separate test runs that shouldn't be aggregated, too.

sjuyal5 · 2023-07-11T09:07:17Z

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

@ankitjain1510 did you find solution to this problem?

ankitjain1510 · 2023-07-11T09:11:21Z

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

@ankitjain1510 did you find solution to this problem?

@sjuyal5 No. I am also stuck at the same place with the same issue. I started using the alternative tool i.e. https://github.com/qiniu/goc

sjuyal5 · 2023-07-11T09:14:18Z

Thanks for the prompt response Ankit. Regards Savita Juyal

…

On Tue, 11 Jul 2023 at 2:41 PM, Ankit Jain ***@***.***> wrote: Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available? @ankitjain1510 <https://github.com/ankitjain1510> did you find solution to this problem? @sjuyal5 <https://github.com/sjuyal5> No. I am also stuck at the same place with the same issue. I started using the alternative tool i.e. https://github.com/qiniu/goc — Reply to this email directly, view it on GitHub <#60182 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BADQKOZHISRONVQJCAOGVQTXPUKEPANCNFSM6AAAAAAYBBJITE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

davidmdm · 2023-10-22T16:11:50Z

I have a very similar use case which led me to create an issue that was closed as duplicate #63660

To sum it up, I wrote a package that handles context cancelation around signals. Signals cannot be tested as a unit test, and so I created a small testing acceptance binary in my tests that I can run, send signals to, and assert the correct output.

When I detect that a -test.gocoverdir flag is set because the go test command is being run in -cover , I build the acceptance binary with the -cover flag.

However, only the coverage of unit tests is reported using the go test command. If I however set the gocoverdir to some local non-temporary folder, I can see that all of the covdata has been emitted into it and that if I run the go tool covdata percent command that I have 100% coverage and everything is there.

This seems like a bug. I have noticed that go test -cover creates a new temporary directory for coverage on each run, and that the directory is removed before exiting. If that's the case, why aren't we taking into account all of the coverage that go test creates including the subprocesses that the tests execute? Why are we only reporting the covdata that go test -cover creates within its own process rather than all the coverage that was triggered by it?

As of today to get full coverage of you application you need to:

run go test with a local gocoverdir
- go test -cover -test.gocoverdir=./coverage ./...
view an incorrect report of your coverage
run covdata percentage yourself
- go tool covdata percent -i ./coverage
run cvdata to create your coverage profile
- go tool covdata textfmt -i ./coverage -o ./coverage.out
- go tool cover -html ./coverage.out

When really it should just work:

go test -coverprofile ./coverage.out
go tool cover -html ./coverage.out

mitar · 2023-11-19T07:16:21Z

I hit the same issue. In fact, I didn't know that GOCOVERDIR is magically set. So I was running my tests with:

GOCOVERDIR=coverage go test ./... -race -timeout 10m -cover -covermode atomic -args -test.gocoverdir=coverage

It took me quite some time to figure out why things do not work. Why when I then internally call go run from a test, coverage output does not end up in coverage dir. I checked and GOCOVERDIR was present. Only when I checked the value I realized something is strange.

So my proposal would be (and or or):

GOCOVERDIR should not be set automatically if it is already set.
All content from temporary gocovedir should be copied to original gocoverdir.

The issue is now that I even cannot manually set GOCOVERDIR because the value is lost because it is overridden with a temporary one.

hugelgupf · 2024-01-27T07:28:53Z

GOCOVERDIR should not be set automatically if it is already set.

All content from temporary gocovedir should be copied to original gocoverdir.

Or it should be merged into the cover profile provided with -coverprofile.

Anything but discarding my data would be acceptable... I just ran into this as well. Very frustrating as it took me a few days to figure out this is why my data was missing.

When running VM tests with `GOCOVERDIR=$x go test -cover`, Go overrides the GOCOVERDIR for the test process, but does not share the coverage data in any way. Upstream issue is golang/go#60182 Signed-off-by: Chris Koch <chrisko@google.com>

seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 14, 2023

thanm self-assigned this May 17, 2023

thanm added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 20, 2023

FiloSottile removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 30, 2023

seankhliao mentioned this issue Oct 22, 2023

test coverage: subprocess coverage of binaries built with -cover not included with default coverage. #63660

Closed

hugelgupf mentioned this issue Jan 27, 2024

Share VMTEST_GOCOVERDIR with VM instead of GOCOVERDIR hugelgupf/vmtest#98

Merged

thanm mentioned this issue Mar 14, 2024

runtime/coverage: confusing go test -cover + GOCOVERDIR behavior #66225

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime/coverage: coverage from other binaries is not collected in tests #60182

runtime/coverage: coverage from other binaries is not collected in tests #60182

FiloSottile commented May 14, 2023

seankhliao commented May 14, 2023

ankitjain1510 commented May 17, 2023

FiloSottile commented May 17, 2023

thanm commented May 17, 2023

thanm commented May 17, 2023

ankitjain1510 commented May 24, 2023

enrichman commented May 28, 2023 •

edited

FiloSottile commented May 30, 2023

sjuyal5 commented Jul 11, 2023

ankitjain1510 commented Jul 11, 2023

sjuyal5 commented Jul 11, 2023 via email

davidmdm commented Oct 22, 2023 •

edited

mitar commented Nov 19, 2023 •

edited

hugelgupf commented Jan 27, 2024

runtime/coverage: coverage from other binaries is not collected in tests #60182

runtime/coverage: coverage from other binaries is not collected in tests #60182

Comments

FiloSottile commented May 14, 2023

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

seankhliao commented May 14, 2023

ankitjain1510 commented May 17, 2023

FiloSottile commented May 17, 2023

thanm commented May 17, 2023

thanm commented May 17, 2023

ankitjain1510 commented May 24, 2023

enrichman commented May 28, 2023 • edited

FiloSottile commented May 30, 2023

sjuyal5 commented Jul 11, 2023

ankitjain1510 commented Jul 11, 2023

sjuyal5 commented Jul 11, 2023 via email

davidmdm commented Oct 22, 2023 • edited

mitar commented Nov 19, 2023 • edited

hugelgupf commented Jan 27, 2024

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

enrichman commented May 28, 2023 •

edited

davidmdm commented Oct 22, 2023 •

edited

mitar commented Nov 19, 2023 •

edited