Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/cover: should support -coverprofile unifying tests from multiple packages #6909

Closed
wadey opened this issue Dec 6, 2013 · 57 comments
Closed
Milestone

Comments

@wadey
Copy link
Contributor

wadey commented Dec 6, 2013

What steps will reproduce the problem?
If possible, include a link to a program on play.golang.org.
1. run `go test -coverprofile=coverage.out ./...`

What is the expected output?

coverage.out contains coverage all of the specified packages

What do you see instead?

`cannot use test profile flag with multiple packages`

Which compiler are you using (5g, 6g, 8g, gccgo)?

6g

Which operating system are you using?

OS X 10.9

Which version are you using?  (run 'go version')

go version go1.2 darwin/amd64

Please provide any additional information below.

If I manually construct a coverage.out file from multiple packages, it seems to work
fine:

1. go test -coverprofile a.part ./a
2. go test -coverprofile b.part ./b
3. echo "mode: set" >coverage.out
4. grep -h -v "mode: set" *.part >>coverage.out
5. go tool cover -html=coverage.out

Why can't `go test -coverprofile` do this automatically?
@minux
Copy link
Member

minux commented Dec 6, 2013

Comment 1:

Labels changed: added release-go1.3maybe, repo-tools.

Status changed to Accepted.

@rsc
Copy link
Contributor

rsc commented May 9, 2014

Comment 3:

Labels changed: added release-none, removed release-go1.3maybe.

@hgfischer
Copy link
Contributor

Comment 4:

With the 1.3 release, any idea when this will be fixed?

@gopherbot
Copy link

Comment 5 by phuna24:

I'm also facing same problem today with 1.3.1. Is there plan for fixing this soon?

@gopherbot
Copy link

Comment 6 by hwang.dev:

It is not an urgent issue. The workaround is simple, please check the shell script:
https://gist.github.com/hailiang/0f22736320abe6be71ce

@gopherbot
Copy link

Comment 7 by phuna24:

Hi,
Thanks for the link. Very helpful.

@gopherbot
Copy link

Comment 8:

CL https://golang.org/cl/159910044 mentions this issue.

@dlsniper
Copy link
Contributor

dlsniper commented Feb 8, 2015

Can this be included in 1.5. I don't see any technical reason for it to not be. Thank you.

@mikioh mikioh changed the title go.tools/cmd/cover: should support -coverprofile for multiple packages cmd/cover: should support -coverprofile for multiple packages Feb 8, 2015
@minux
Copy link
Member

minux commented Feb 8, 2015 via email

@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@rsc rsc changed the title cmd/cover: should support -coverprofile for multiple packages x/tools/cmd/cover: should support -coverprofile for multiple packages Apr 14, 2015
@rsc rsc modified the milestones: Unreleased, Unplanned Apr 14, 2015
@rsc rsc removed the repo-tools label Apr 14, 2015
@andrewchambers
Copy link

I am interested in getting total test coverage across a DAG of packages. All the solutions online do naive merges of cover profiles via concatenation (including comment 6 here), but this does not take into account lines covered across package boundaries. +1 for anyone who can work on this.

Edited My original phrasing was bad.

@adg
Copy link
Contributor

adg commented May 13, 2015

I wouldn't say there's "no interest" but rather nobody with time to work on it.

@andrewchambers
Copy link

With regard to the solution I can see two approaches:

  • Instrumenting every package in go list ./... for each run and then merging cover profiles.
  • Generating a single instrumented test binary with all tests combined. This probably means go test -c ./... produces one combined test binary.

@josharian
Copy link
Contributor

I personally want go test -c to support producing a single binary. In particular, I want go test -c std to work, so that I can easily benchmark compiler changes. Right now I have to play games to generate and manage a slew of test binaries.

However, this would require a bit of thought. Right now, all tests and benchmarks are implicitly namespaced by the package. Take for example go test -c encoding/.... Several packages have a TestEncode function. If you did ./encoding.test -test.run=Encode, should it run all of them? How do you specify just one of them? There's a similar question about output. Run e.g. go test -v -run=^TestEncode$ encoding/... and observe that the go tool is responsible for providing the output that differentiates the different packages. Same thing for benchmarks.

This is not to say it can't/shouldn't be done, just that there are some careful decisions to be made.

@tv42
Copy link

tv42 commented May 13, 2015

@josharian Tests also can define their own command-line flag parsing. I don't see how you could make that work right.

@josharian
Copy link
Contributor

I would think that you would deal with that the same way you do now when combining packages that define their own command-line flag parsing in a non-test program—by manually fixing those collisions when they occur.

@andrewchambers
Copy link

@josharian wrt name spacing, The full package path could be part of the test name, or there could be a seperate flag -runpkg that works in conjunction with -run.

@josharian
Copy link
Contributor

Another challenge (just writing them down here as I think of them) -- tests are supposed to be executed from the directory in which they are written. This lets them find fixtures easily. However, if multiple tests are bundled into a single executable, there is no single correct working directory. Again, not a showstopper, just something to be thought through.

@tv42
Copy link

tv42 commented May 14, 2015

@josharian Please don't rely on that. go test -c github.com/jdoe/foo is a thing, and so is cross-compiling & rsyncing the test binary.

@rasky
Copy link
Member

rasky commented Nov 6, 2017

@rsc now that the build cache is in, are we closer to fix this bug? I think people are waiting here for some guidance on how to fix it. I can attempt something myself, but it would be great if you could provide some guidance. Thanks!

@dlsniper
Copy link
Contributor

dlsniper commented Nov 6, 2017

Maybe a different way to see the problem is that we should have a different covermode for which we can apply this feature.

For example, running go test -coverprofile=coverage.out ./... should detect that the user wants to run in a recursive mode thus set the covermode to a recursive (or better named mode) in which, regardless of how many times a line is executed, it counts only as a simple boolean value (yes / no)?

@dlsniper
Copy link
Contributor

dlsniper commented Nov 6, 2017

Of course, this could also be done in a more complex way, by storing which test hit which line and then, on a per-line basis, and then offer for each line of code touched by tests a slice of test names. It would then be up to the tools to read that information and display it to the users, in the most simplistic way doing this as described above with a simple yes / no mode.

Sorry for the double message.

@andradei
Copy link

andradei commented Nov 7, 2017

The first suggestions is a great way to get this started. But the more powerful and resourceful go test -converprofile is , the better.

@gopherbot
Copy link

Change https://golang.org/cl/76875 mentions this issue: cmd/go: allow -coverprofile with multiple packages being tested

@rsc rsc modified the milestones: Unreleased, Go1.10 Nov 10, 2017
@rsc rsc changed the title x/tools/cmd/cover: should support -coverprofile unifying tests from multiple packages cmd/cover: should support -coverprofile unifying tests from multiple packages Nov 10, 2017
dhui added a commit to dhui/passhash that referenced this issue Nov 1, 2018
with multiple packages has been supported since Go 1.10
    - golang/go#6909
@golang golang locked and limited conversation to collaborators Nov 10, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests