Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/pprof: CPU profile writer falls behind in big program #7949

Closed
btracey opened this issue May 7, 2014 · 8 comments
Closed

runtime/pprof: CPU profile writer falls behind in big program #7949

btracey opened this issue May 7, 2014 · 8 comments

Comments

@btracey
Copy link
Contributor

btracey commented May 7, 2014

Running pprof on a program that has O(10^6) cpu-bound goroutines loses ~50% of samples.

Code is github.com/btracey/stackmc/examples/paper/rosen_unif

Svg of pprof output: http://stanford.edu/~btracey/gobench/stackmc/graph.svg

Working now on a shorter reproducer

[btracey@zion ~]$ go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/ADL/btracey/mygo"
GORACE=""
GOROOT="/ADL/btracey/gover/go_tip/go"
GOTOOLDIR="/ADL/btracey/gover/go_tip/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
@btracey
Copy link
Contributor Author

btracey commented May 7, 2014

Comment 1:

Here is a shorter reproducing code. This takes a couple minutes on my machine, but
changing the inner loop computation to size 1000 rather than 100000 did not reproduce
the problem
http://play.golang.org/p/O-ixYe9BdY

@ianlancetaylor
Copy link
Contributor

Comment 2:

I'm pretty sure this was fixed earlier today by https://golang.org/cl/96060043
.  Closing this bug assuming that I am correct;  please re-open if I am not.

Status changed to Fixed.

@btracey
Copy link
Contributor Author

btracey commented May 7, 2014

Comment 3:

I don't think it was fixed by that revision.
[btracey@zion ~]$ go version
go version devel +ad0ee2227396 Wed May 07 18:49:13 2014 +0400 linux/amd64
I downloaded the source after that revision was committed, though I don't know if there
is a lag, and I'm not sure how to check that that CL is included in my version.

@ianlancetaylor
Copy link
Contributor

Comment 4:

You're right, you do have that change (use hg log to see the changes you have).

Labels changed: added repo-main, release-go1.3maybe.

Status changed to New.

@btracey
Copy link
Contributor Author

btracey commented May 7, 2014

Comment 5:

From Dmitry:
The profile is written by a normal goroutine, so because you lots of
runnable goroutine, the profiler goroutine is executed very
episodically.

@rsc
Copy link
Contributor

rsc commented May 9, 2014

Comment 6:

I don't expect to change this for Go 1.3 at this point. I'm not really sure what we
should do short of giving extra priority to the writer goroutine (but even that is not
clear).

Labels changed: added release-go1.4, removed release-go1.3maybe.

Status changed to Accepted.

@btracey
Copy link
Contributor Author

btracey commented May 9, 2014

Comment 7:

What about not counting LostProfileData samples, and instead just report a smaller
number of total samples and renormalize the percentages (or having a pprof option to do
so)?

@rsc
Copy link
Contributor

rsc commented Sep 18, 2014

Comment 8:

I don't see any good way to fix this. Unless it comes up more often I'm just going to
live with it.
If you want to just ignore the lost profile data, you can give pprof the option
--ignore=LostProfileData.

Status changed to Unfortunate.

@rsc rsc added this to the Go1.4 milestone Apr 14, 2015
@rsc rsc removed the release-go1.4 label Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 25, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants