Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: test -cover reports 100.0% without full coverage #39168

Open
egonk opened this issue May 20, 2020 · 8 comments
Open

cmd/go: test -cover reports 100.0% without full coverage #39168

egonk opened this issue May 20, 2020 · 8 comments
Labels
NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@egonk
Copy link

egonk commented May 20, 2020

What version of Go are you using (go version)?

go version go1.14.3 windows/amd64

Does this issue reproduce with the latest release?

Yes (1.14.3)

What operating system and processor architecture are you using (go env)?

go env Output
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\egon\AppData\Local\go-build
set GOENV=C:\Users\egon\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\egon\Desktop
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\egon\AppData\Local\Temp\go-build906217222=/tmp/go-build -gno-record-gcc-switches

What did you do?

The problem came up while developing a binary protocol library with many generated functions.

Generate many tests with a script:

package main

import (
	"html/template"
	"os"
)

const n = 50000

func main() {
	{
		f, err := os.Create("cov.go")
		p(err)
		_, err = f.Write([]byte(`package cov`))
		p(err)
		for i := 0; i < n; i++ {
			p(template.Must(template.New("").Parse(`
func F{{.N}}() int { return {{.N}} }`)).Execute(f, map[string]interface{}{"N": i}))
		}
		p(f.Close())
	}
	{
		f, err := os.Create("cov_test.go")
		p(err)
		_, err = f.Write([]byte(`package cov
import (
	"testing"
)
`))
		p(err)
		for i := 0; i < n-1; i++ {
			p(template.Must(template.New("").Parse(`
func TestF{{.N}}(t *testing.T) { F{{.N}}() }`)).Execute(f, map[string]interface{}{"N": i}))
		}
		p(f.Close())
	}
}

func p(err error) {
	if err != nil {
		panic(err)
	}
}

Run test coverage:

>go test -cover cov
ok      cov     0.238s  coverage: 100.0% of statements

What did you expect to see?

99.9% or something similar.

Reducing n to 1000 produces 99.9% on my machine.

What did you see instead?

100.0% (which implies full coverage, not true!)

@egonk
Copy link
Author

egonk commented May 20, 2020

Workaround for the issue using -coverprofile output:

$ egrep ' 0$' /c/Users/egon/AppData/Local/Temp/vscode-govqeSQk/go-code-cover
<dir>/<file>.go:77.47,79.3 1 0

@robpike
Copy link
Contributor

robpike commented May 20, 2020

Isn't this just floating point rounding for an extreme corner case? Even if it printed 2 digits after the decimal point, you might still get 100.00%, depending on exactly how many statements are not executed.

I don't see a good solution, and don't feel the problem is important enough to do something special. The shorthand report makes no guarantees but there are other output forms that might help.

@egonk
Copy link
Author

egonk commented May 20, 2020

It's not pretty, but a simple hack strings.HasPrefix(pct, "100.") would work...

@egonk
Copy link
Author

egonk commented May 20, 2020

Maybe something like this?

package main

import (
	"fmt"
	"strings"
)

func coveragePct(active, total int64) string {
	pn := 100 * float64(active) / float64(total)
	var pct string
	for f := []byte("%.1f%%"); f[2] <= '8'; f[2]++ {
		pct = fmt.Sprintf(string(f), pn)
		if active == total || !strings.HasPrefix(pct, "100.") {
			break
		}
	}
	return pct
}

func main() {
	for n := int64(10); n <= 100_000_000_000; n *= 10 {
		a := n - 1
		fmt.Printf("%v/%v => %v\n", a, n, coveragePct(a, n))
	}
}

Output:

9/10 => 90.0%
99/100 => 99.0%
999/1000 => 99.9%
9999/10000 => 99.99%
99999/100000 => 99.999%
999999/1000000 => 99.9999%
9999999/10000000 => 99.99999%
99999999/100000000 => 99.999999%
999999999/1000000000 => 99.9999999%
9999999999/10000000000 => 99.99999999%
99999999999/100000000000 => 100.00000000%

@cagedmantis cagedmantis changed the title go test -cover 100.0% without full coverage cmd/go: test -cover reports 100.0% without full coverage May 20, 2020
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 20, 2020
@cagedmantis cagedmantis added this to the Backlog milestone May 20, 2020
@cagedmantis
Copy link
Contributor

/cc @bcmills @jayconrod @matloob

@bcmills
Copy link
Contributor

bcmills commented May 20, 2020

I agree with @robpike. 100.0% of statements does not imply “full coverage”, it implies ≥
99.95% coverage. (I would assume that only the printed digits are significant, as I was taught in high school science classes.)

If you want to analyze coverage down to the exact line, I would expect that you could build a standalone tool for that on top of the -coverprofile output. Is that not the case?

@bcmills bcmills modified the milestones: Backlog, Unplanned May 20, 2020
@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 20, 2020
@egonk
Copy link
Author

egonk commented May 21, 2020

@bcmills

If you want to analyze coverage down to the exact line, I would expect that you could build a standalone tool for that on top of the -coverprofile output. Is that not the case?

That's true of course. I've been thinking about the default experience, this is how it went for me:

  1. add some tests
  2. click "Run package tests" in VS Code
  3. it says 100.0%, great!
  4. write doc comment for the tested function
  5. there's a red line in the function, oh right, floating point

I agree with @robpike. 100.0% of statements does not imply “full coverage”, it implies ≥
99.95% coverage. (I would assume that only the printed digits are significant, as I was taught in high school science classes.)

Yes, but it also means that the output cannot be reliably used to check for full coverage if the package is slightly bigger. I've just checked, only 2000 lines! The code above increases the precision around 100.0% but it still stays in the high school mathematics.

There's another simpler solution which I use for progress reports (I hate seeing 100% when the software is not done):

func coveragePct(active, total int64) string {
	return fmt.Sprintf("%.1f%%", math.Trunc(1000*float64(active)/float64(total))/10)
}

@gburt
Copy link

gburt commented Jan 7, 2021

I'd be in favor of go test -cover outputting 100.0% if and only if every line is covered, and otherwise cap it out at 99.9%.
Thoughts on that @robpike?

For context, we have a script which parses that output to check that a pkg's coverage meets a specified threshold (which is often 100%). I understand we could (and perhaps should) be parsing some other, more robust output, but I think between our use-case and the general use-case of a CLI or editor/IDE user seeing 100% when it's not really (and having expectation that 100% means every-line-is-covered), that it's worth making this rounding exception.

@ALTree ALTree added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jun 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

No branches or pull requests

6 participants