Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go test -cover & go test -coverprofile should always output a coverage #24570

Closed
Tracked by #383 ...
gustafj opened this issue Mar 27, 2018 · 34 comments
Closed
Tracked by #383 ...
Assignees
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@gustafj
Copy link

gustafj commented Mar 27, 2018

Description

Now in 1.10 when go test -cover supports multiple packages, I would expect it to print out a percentage for all packages (including those missing tests).
And for go test -coverprofile, I would expect all packages to be included in the calculated total.

Currently only packages that have at least one test (can be a *_test.go with only the package declaration) is included, see pkg2 below.

What version of Go are you using (go version)?

go version go1.10 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

Linux, amd64

What did you do?

go test ./... -cover
go test ./... -coverprofile cover.out; go tool cover -func cover.out

What did you expect to see?

?      path/to/pkg1   0.001s  coverage:   0.0% of statements [no test files]
ok     path/to/pkg2   0.019s  coverage:   0.0% of statements [no tests to run]
ok     path/to/pkg3   0.371s  coverage: 100.0% of statements
path/to/pkg1/pkg1.go:5: String 0.0%
path/to/pkg2/pkg2.go:5: String 0.0%
path/to/pkg3/pkg3.go:5: String 100.0%
total: (statements) 33.3%

What did you see instead?

?      path/to/pkg1   [no test files]
ok     path/to/pkg2   0.019s  coverage:   0.0% of statements [no tests to run]
ok     path/to/pkg3   0.371s  coverage: 100.0% of statements
path/to/pkg2/pkg2.go:5: String 0.0%
path/to/pkg3/pkg3.go:5: String 100.0%
total: (statements) 50.0%
@ianlancetaylor ianlancetaylor added the NeedsFix The path to resolution is known, but the work has not been done. label Mar 27, 2018
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Mar 27, 2018
@gopherbot
Copy link

Change https://golang.org/cl/115095 mentions this issue: cmd/go/internal/test: always output a coverage

@mvdan
Copy link
Member

mvdan commented May 30, 2018

@kyroy noticed that this issue is very similar to #25492. It seems to me like both should be fixed at once, as they suggest changes in opposite directions. The other issue wants to change a 0.0% with a [no statements], and this one wants to change [no test files] with 0.0%.

/cc @bcmills @egonk

@bcmills
Copy link
Contributor

bcmills commented May 30, 2018

they suggest changes in opposite directions.

#25492 is about the case where there are no statements to cover (and thus the percentage is undefined). That's materially different from the case here, where pkg1 does have a statement and it is not covered: 0.0% coverage is well-defined and correct for a package with one statement and no tests.

@mvdan
Copy link
Member

mvdan commented May 30, 2018

My reading of this issue is that go test -cover should always output a percentage if it succeeded. The other issue is precisely about removing a 0.0% output in favor of something else.

@bcmills
Copy link
Contributor

bcmills commented May 30, 2018

My reading of this issue is that go test -cover should always output a percentage if it succeeded.

That's the current issue title, but that behavior seems clearly wrong if there are no statements to cover: 0/0 is not a well-defined percentage.

(I guess we could output NaN%, but that seems strictly less helpful than [no statements].)

@kyroy
Copy link
Contributor

kyroy commented May 30, 2018

I like the idea of a consistent output that would be created by just implementing this issue.

Another approach is, as in #25492, for all statements it is true that they are covered by the tests. So one could argue for 100% which is obviously mathematically incorrect.

I am also the creator of the CL. Happy to implement a solution that results from the discussion :)

sdboyer added a commit to sdboyer/dep that referenced this issue Jul 2, 2018
This should fix test failures that were being caused by golang/go#24570,
which caused test binaries to be built even when there are no test
files. The poor interaction there arose from the setting of flags in our
init() function, it seems.
sdboyer added a commit to sdboyer/dep that referenced this issue Jul 3, 2018
This should fix test failures that were being caused by golang/go#24570,
which caused test binaries to be built even when there are no test
files. The poor interaction there arose from the setting of flags in our
init() function, it seems.
@gopherbot
Copy link

Change https://golang.org/cl/122518 mentions this issue: cmd/go: revert "output coverage report even if there are no test files"

gopherbot pushed a commit that referenced this issue Jul 9, 2018
Original CL description:

    When using test -cover or -coverprofile the output for "no test files"
    is the same format as for "no tests to run".

Reverting because this CL changed cmd/go to build test binaries for
packages that have no tests, leading to extra work and confusion.

Updates #24570
Fixes #25789
Fixes #26157
Fixes #26242

Change-Id: Ibab1307d39dfaec0de9359d6d96706e3910c8efd
Reviewed-on: https://go-review.googlesource.com/122518
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
@ianlancetaylor
Copy link
Contributor

CL 115095 was reverted, so reopening this issue.

@agnivade
Copy link
Contributor

FYI, f7248f0 brought back the pkg{1,2,3} folders inside cmd/go/testdata/testcover that was originally reverted by 7254cfc. Those files are unused as of now.

@Nerzal
Copy link

Nerzal commented Feb 10, 2019

Any news here?

@rabadin
Copy link

rabadin commented Apr 1, 2019

One can use -coverpkg=./... to get accurate results:

$ tree .
.
├── coverage.out
├── package0.go
├── package0_test.go
├── package1
│   ├── package1.go
│   └── package1_test.go
└── package2
    └── package2.go

2 directories, 6 files

Without -coverpkg=./...:

$ go test -v -coverprofile=coverage.out ./...
=== RUN   TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 100.0% of statements
ok  	github.com/rabadin/brokencoverage	0.007s	coverage: 100.0% of statements
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 100.0% of statements
ok  	github.com/rabadin/brokencoverage/package1	0.006s	coverage: 100.0% of statements
?   	github.com/rabadin/brokencoverage/package2	[no test files]

$ go tool cover -func=coverage.out
github.com/rabadin/brokencoverage/package0.go:3:		min		100.0%
github.com/rabadin/brokencoverage/package1/package1.go:3:	add		100.0%
total:								(statements)	100.0%

With -coverpkg=./...:

$ rm coverage.out

$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
=== RUN   TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok  	github.com/rabadin/brokencoverage	0.006s	coverage: 33.3% of statements in ./...
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok  	github.com/rabadin/brokencoverage/package1	0.006s	coverage: 33.3% of statements in ./...
?   	github.com/rabadin/brokencoverage/package2	[no test files]

$ go tool cover -func=coverage.out
github.com/rabadin/brokencoverage/package0.go:3:		min		100.0%
github.com/rabadin/brokencoverage/package1/package1.go:3:	add		100.0%
github.com/rabadin/brokencoverage/package2/package2.go:3:	mult		0.0%
total:								(statements)	66.7%

@puellanivis
Copy link

puellanivis commented Jun 7, 2019

Using -coverpkg does not actually help if the package is never called in any test at all. -coverpkg is a cover-up that allows packages from a different package to provide coverage for this package.

So, say package a has no tests, and thus no coverage. But package b has tests, and those tests call into a. As a result, if you start using a -coverpkg that covers both packages, the tests from package b will report its coverage of package a as coverage for a.

However, if we have a package c that is never called by either a or b then that package will still be left out in the dry, even if it would match the -coverpkg given.

project$ find . -name "*_test.go"
project$ go test -coverpkg=./... -coverprofile=coverage.out ./...
?   	project	[no test files]
?   	project/sub/dir1	[no test files]
?   	project/sub/dir2	[no test files]
project$ cat coverage.out
mode: set
project$

Additionally, it is misleading to say that because b has called into a that that coverage of a counts as tests on a, because they are not unit tests of a’s functionality, and b is fundamentally not responsible for testing of a.

@rabadin
Copy link

rabadin commented Jun 7, 2019

@puellanivis:

Using -coverpkg does not actually help if the package is never called in any test at all.

I'm not sure this is true. Here is the code from the example above:

find . -name "*.go" -exec echo '******** {}' \; -exec cat {} \;
******** ./package2/package2.go
package package2

func mult(a int, b int) int {
	return a * b
}
******** ./package0_test.go
package brokencoverage

import "testing"

func TestMin(t *testing.T) {
	if min(3,5) != 3-5 {
		t.Errorf("min is broken")
	}
}
******** ./package1/package1_test.go
package package1

import "testing"

func TestAdd(t *testing.T) {
	if add(3,5) != 3+5 {
		t.Errorf("add is broken")
	}
}
******** ./package1/package1.go

package package1

func add(a int, b int) int {
	return a + b
}
******** ./package0.go
package brokencoverage

func min(a int, b int) int {
	return a - b
}

as you can see nothing is calling package2/package2.go:mult.

@puellanivis
Copy link

puellanivis commented Jun 11, 2019

I don’t know exactly what is different, but even with -coverpkg=./... my project with no tests still reports no packages:

project-with-no-tests$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
?   	project-with-no-tests	[no test files]
?   	project-with-no-tests/subpackage	[no test files]
project-with-no-tests$ go tool cover -func=coverage.out
total:	(statements)	0.0%

While adding just a single _test.go with just the package name yields coverage details for a package.

Meanwhile, if I add a reference to package2.Mult in package0_test.go suddenly, with -coverpkg=./... we report that there is coverage of package2.Mult, even though it is not actually covered by any unit tests. (It would make sense to include such coverage in integration tests though, as there you are testing the system as a whole, but unit tests should never count coverage provided by any other package.)

broken-coverage$ cat package0_test.go 
package brokencoverage

import (
	"testing"
	"github.com/puellanivis/broken-coverage/package2"
)

func TestMin(t *testing.T) {
	if min(3,5) != package2.Mult(-1, 2) {
		t.Errorf("min is broken")
	}
}
broken-coverage$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
=== RUN   TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 66.7% of statements in ./...
ok  	github.com/puellanivis/broken-coverage	0.001s	coverage: 66.7% of statements in ./...
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok  	github.com/puellanivis/broken-coverage/package1	0.004s	coverage: 33.3% of statements in ./...
?   	github.com/puellanivis/broken-coverage/package2	[no test files]
broken-coverage$ go tool cover -func=coverage.out
github.com/puellanivis/broken-coverage/package0.go:3:		min		100.0%
github.com/puellanivis/broken-coverage/package1/package1.go:3:	add		100.0%
github.com/puellanivis/broken-coverage/package2/package2.go:3:	Mult		100.0%
total:								(statements)	100.0%

@devleesch001
Copy link

Je me demande quelle est la bonne façon de forcer la sortie de 0.0%Go 1.20 ?

PS : La meilleure solution de contournement (qui a été mentionnée par d'autres) : ajoutez <foo>_test.godes fichiers factices avec une seule ligne package <foo>_test.

It's not very optimal but it's the only solution that seems to work for the moment. thx

@simonkotwicz
Copy link
Contributor

simonkotwicz commented Apr 26, 2023

You can use export GOEXPERIMENT=nocoverageredesign as a workaround in Go 1.20 until this issue is fixed as per #58770

@thanm thanm self-assigned this Apr 27, 2023
@gopherbot
Copy link

Change https://go.dev/cl/495452 mentions this issue: cmd/go: fix percent covered problems with -coverpkg

@gopherbot
Copy link

Change https://go.dev/cl/495446 mentions this issue: cmd/cover: add new "emit meta file" mode for packages without tests

@gopherbot
Copy link

Change https://go.dev/cl/495447 mentions this issue: cmd/go: improve handling of no-test packages for coverage

SOF3 added a commit to SOF3/kelemetry that referenced this issue Aug 2, 2023
gopherbot pushed a commit that referenced this issue Sep 14, 2023
Introduce a new mode of execution for instrumenting packages that have
no test files. Instead of just skipping packages with no test files
(during "go test -cover" runs), the go command will invoke cmd/cover
on the package passing in an option in the config file indicating that
it should emit a coverage meta-data file directly for the package (if
the package has no functions, an empty file is emitted). Note that
this CL doesn't actually wire up this functionality in the Go command,
that will come in a later patch.

Updates #27261.
Updates #58770
Updates #24570.

Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-windows-amd64-longtest
Change-Id: I01e8a3edb62441698c7246596e4bacbd966591c3
Reviewed-on: https://go-review.googlesource.com/c/go/+/495446
Reviewed-by: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
gopherbot pushed a commit that referenced this issue Sep 30, 2023
This patch fixes some problems with how "go test -cover" was handling
tests involving A) multiple package tests and B) multiple packages
matched by "-coverpkg". In such scenarios the expectation is that the
percent statements covered metric for each package needs to be
relative to all statements in all packages matched by the -coverpkg
arg (this aspect of the reporting here was broken as part of
GOEXPERIMENT=coverageredesign).

The new scheme works as follows.  If -coverpkg is in effect and is
matching multiple packages, and we have multiple test targets, then:

  - each time a package is built for coverage, capture a meta-data
    file fragment corresponding to just the meta-data for that package.

  - create a new "writeCoverMeta" action, and interpose it between the
    build actions for the covered packages and the run actions. The
    "writeCoverMeta" action at runtime will emit a file
    "metafiles.txt" containing a table mapping each covered package
    (by import path) to its corresponding meta-data file fragment.

  - pass in the "metafiles.txt" file to each run action, so that
    when the test finishes running it will have an accurate picture
    of _all_ covered packages, permitting it to calculate the correct
    percentage.

Concrete example: suppose we have a top level directory with three
package subdirs, "a", "b", and "c", and from the top level, a user
runs "go test -coverpkg=./... ./...". This will result in (roughly)
the following action graph:

  build("a")       build("b")         build("c")
      |               |                   |
  link("a.test")   link("b.test")     link("c.test")
      |               |                   |
  run("a.test")    run("b.test")      run("c.test")
      |               |                   |
    print          print              print

With the new scheme, the action graph is augmented with a
writeCoverMeta action and additional dependence edges to form

  build("a")       build("b")         build("c")
      |   \       /   |               /   |
      |    v     v    |              /    |
      | writecovmeta<-|-------------+     |
      |         |||   |                   |
      |         ||\   |                   |
  link("a.test")/\ \  link("b.test")      link("c.test")
      |        /  \ +-|--------------+    |
      |       /    \  |               \   |
      |      v      v |                v  |
  run("a.test")    run("b.test")      run("c.test")
      |               |                   |
    print          print              print

A note on init functions: prior to GOEXPERIMENT=coverageredesign
the "-coverpkg=..." flag was implemented by force-importing
all packages matched by "-coverpkg" into each instrumented package.
This meant that for the example above, when executing "a.test",
the init function for package "c" would fire even if package "a"
did not ordinarily import package "c".  The new implementation
does not do this sort of forced importing, meaning that the coverage
percentages can be slightly different between 1.21 and 1.19 if
there are user-written init funcs.

Fixes #58770.
Updates #24570.

Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-windows-amd64-longtest
Change-Id: I7749ed205dce81b96ad7f74ab98bc1e90e377302
Reviewed-on: https://go-review.googlesource.com/c/go/+/495452
Reviewed-by: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@shuLhan
Copy link
Contributor

shuLhan commented Oct 3, 2023

@thanm The commit d1cb5c0 cause regression which makes "go tool cover -html=cover.out -o cover.html" return an error if a package called "internal" in root module does not have test files.

This can be reproduced by running "make" on these Go repository: https://github.com/shuLhan/share/

Using go tip 1e69040 (one commit before d1cb5c0),

(ins) 1 $ make                                                                                                                       [132/234]
CGO_ENABLED=1 go test -failfast -timeout=1m -race -coverprofile=cover.out ./...
?       github.com/shuLhan/share        [no test files]
...
?       github.com/shuLhan/share/cmd/xtrk       [no test files]
?       github.com/shuLhan/share/internal/bytes [no test files]
?       github.com/shuLhan/share/lib/contact    [no test files]
...
ok      github.com/shuLhan/share/lib/xmlrpc     1.017s  coverage: 64.1% of statements
go tool cover -html=cover.out -o cover.html
go vet ./...
fieldalignment ./...
shadow ./...
revive ./...
mkdir -p _bin/
go build -ldflags "-s -w -X 'github.com/shuLhan/share.Version=v0.49.1-99-g7cf89da5'" -o _bin/ ./cmd/...

Using go tip d1cb5c0,

(ins) 1 $ make
CGO_ENABLED=1 go test -failfast -timeout=1m -race -coverprofile=cover.out ./...
?       github.com/shuLhan/share        [no test files]
...
        github.com/shuLhan/share/cmd/totp               coverage: 0.0% of statements
        github.com/shuLhan/share/internal/bytes         coverage: 0.0% of statements
        github.com/shuLhan/share/cmd/smtpcli            coverage: 0.0% of statements
...
ok      github.com/shuLhan/share/lib/xmlrpc     1.017s  coverage: 61.5% of statements
go tool cover -html=cover.out -o cover.html
cover: package e/internal/bytes is not in std (/home/ms/opt/go/src/e/internal/bytes)

Using go tip 638e0d3,

 23:35:29 ~/go/src/github.com/shuLhan/share
(ins) 1 $ go version
go version devel go1.22-638e0d36d2 Tue Oct 3 16:01:10 2023 +0000 linux/amd64

 23:47:11 ~/go/src/github.com/shuLhan/share
(ins) 1 $ make
CGO_ENABLED=1 go test -failfast -timeout=1m -race -coverprofile=cover.out ./...
?       github.com/shuLhan/share        [no test files]
...
        github.com/shuLhan/share/cmd/xtrk               coverage: 0.0% of statements
        github.com/shuLhan/share/internal/bytes         coverage: 0.0% of statements
        github.com/shuLhan/share/lib/contact            coverage: 0.0% of statements
...
ok      github.com/shuLhan/share/lib/xmlrpc     1.018s  coverage: 61.5% of statements
go tool cover -html=cover.out -o cover.html
cover: no required module provides package thub.com/shuLhan/share/cmd/bcrypt; to add it:
        go get thub.com/shuLhan/share/cmd/bcrypt
make: *** [Makefile:28: test] Error 1

@thanm
Copy link
Contributor

thanm commented Oct 3, 2023

Thanks for the report. Could you possibly open another issue for this? Thanks.

@thanm
Copy link
Contributor

thanm commented Oct 3, 2023

Update: I will go ahead and file an issue for this, I think I see what the problem is.

@thanm
Copy link
Contributor

thanm commented Oct 3, 2023

#63356 filed, please follow along on that issue if you want to track the fix.

@gopherbot
Copy link

Change https://go.dev/cl/532555 mentions this issue: cmd/go: fix objdir for run actions for -cover no-test packages

gopherbot pushed a commit that referenced this issue Oct 4, 2023
As of CL 495447 we now synthesize coverage data (including coverage
profiles) for packages that have no tests, if they are included in a
"go test -cover" run. The code that set up the "run" actions for such
tests wasn't setting the objdir for the action, which meant that the
coverage profile temp file fragment ("_cover_.out") was being created
in the dir where the test was run, and in addition the same fragment
could be written to by more than one package (which could lead to a
corrupted file). This CL updates the code to properly set the objdir,
and to create the dir when needed.

Updates #24570.
Fixes #63356.

Change-Id: Iffe131cf50f07ce91085b816a039308e0ca84776
Reviewed-on: https://go-review.googlesource.com/c/go/+/532555
Reviewed-by: Russ Cox <rsc@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@bcmills bcmills modified the milestones: Unplanned, Go1.22 Dec 5, 2023
@mfridman
Copy link

mfridman commented Feb 9, 2024

This change is a bit unfortunate, because with <go1.22 one could run -cover and see which packages don't have test files, without running go test twice, with and without -cover.

image

This used to be consistent, where "no test files" regardless of -cover and regardless of function bodies literally meant just that: "no test files". But now if I have function bodies and run -cover there's no way to bubble to a user that a given package doesn't have test files. The property seems more useful than just displaying 0.0%.

@thanm
Copy link
Contributor

thanm commented Feb 12, 2024

I can see your point, however there are other easy ways of determining that a package has no tests, e.g. go list -f '{{.TestGoFiles}}' <package> is quick and easy to use.

If on the other hand I run

$ go test -coverprofile=profile.txt ./...
...
$ go tool cover -func=profile.txt

to identify code that isn't covered, I would argue that what most folks want is to see all of their uncovered functions, not just the functions in subdirectories where someone wrote tests.

shahzadlone added a commit to shahzadlone/defradb that referenced this issue Feb 23, 2024
Resolves sourcenetwork#672 

**Description:**

Ensures the empty packages aren't left out (because previously packages with 0 tests won't be included in the coverage). This should help avoid seeing the random coverage drops we see in PRs that are actually introducing tests.

__Moreover__
- Removes the make rule `make test:coverage-quick`.
- Change make rule `make test:coverage-full` to `make test:coverage`
- Adds the ability to also provide `path=...` to `make test:coverage` like we had for `make test:coverage-html`.
- Account for integration test coverage for the rule `make test:coverage-html`.

__Limitation__
Using `-coverpkg` does not actually help if the package is never called in any test at all. `-coverpkg` is a cover-up that allows packages from a different package to provide coverage for this package.
So, say package `a` has no tests, and thus no coverage. But package `b` has tests, and those tests call into `a`. As a result, if you start using a `-coverpkg` that covers both packages, the tests from package `b` will report its coverage of package `a` as coverage for `a`.
However, if we have a package `c` that is never called by either `a` or `b` then that package will still be left out in the dry, even if it would match the `-coverpkg` given.
Reference: golang/go#24570
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests