Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: runtime: provide access to info about recent GC cycles #28623

Open
prashantv opened this issue Nov 6, 2018 · 8 comments
Open

proposal: runtime: provide access to info about recent GC cycles #28623

prashantv opened this issue Nov 6, 2018 · 8 comments

Comments

@prashantv
Copy link
Contributor

Currently, runtime.parsedebugvars is called from schedinit which is part of binary bootstrap. This means it's not possible to dynamically enable GODEBUG features once a binary has started.

Typically, we don't want gc traces to flood our logs, but when debugging specific memory issues, it would be helpful if we could enable features (such as gctrace=1 and scavenge=1) dynamically.

While it doesn't have to be via os.Setenv, it;d be useful to have some way to dynamically enable GODEBUG features.

What version of Go are you using (go version)?

$ go version
go version go1.11.2 linux/amd64

Does this issue reproduce with the latest release?

Yes, I'm using the latest release.

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/prashant/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/prashant/go"
GOPROXY=""
GORACE=""
GOROOT="/home/prashant/bin/go1.11.2"
GOTMPDIR=""
GOTOOLDIR="/home/prashant/bin/go1.11.2/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build849457703=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Set GODEBUG to gctrace=1,scavenge=1 dynamically using os.Setenv.

What did you expect to see?

Expected to see GC traces once the variable was set.

What did you see instead?

No GC traces.

@agnivade agnivade changed the title runtime: Allow GODEBUG flags to be enabled dynamically proposal: runtime: Allow GODEBUG flags to be enabled dynamically Nov 7, 2018
@gopherbot gopherbot added this to the Proposal milestone Nov 7, 2018
@rsc
Copy link
Contributor

rsc commented Nov 28, 2018

In general we've assumed that these are constants for the life of the program. It's unclear what would happen if some of these "run-time constants" started changing underfoot, even if we did all the wiring you suggest. It would probably be better to identify things that users really need to change and add explicit functions instead of reparsing $GODEBUG.

For example GODEBUG lets you set memprofilerate but there's a direct way to set it too.

/cc @aclements @RLH for whether it makes sense to be able to toggle gctrace, scavenge.

@rsc rsc added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Dec 12, 2018
@rsc rsc changed the title proposal: runtime: Allow GODEBUG flags to be enabled dynamically proposal: runtime: allow GODEBUG flags to be enabled dynamically Dec 19, 2018
@prashantv
Copy link
Contributor Author

Explicit functions to enable GC logging, or recording the last N logs in-memory without printing them, and making them visible on /debug/pprof/heap would be good enough.

@rsc rsc changed the title proposal: runtime: allow GODEBUG flags to be enabled dynamically proposal: runtime: provide access to info about recent GC cycles Jan 9, 2019
@rsc
Copy link
Contributor

rsc commented Jan 9, 2019

OK, it sounds like this issue should be narrowed from arbitrary GODEBUG toggling to really how to get better info about recent GCs. I apologize for the terrible-ness of the current MemStats - the GC and everything else was much simpler back then. Retitled this issue to be about GC info, some kind of extra helper alongside ReadMemStats for GC info. Or maybe even other runtime info.

The runtime team discussed this earlier this week but no progress on anything concrete yet.

@aclements
Copy link
Member

I think the bigger issue here is how we can export stats from the runtime that may change from version to version. I would love to expose programmatic access to gctrace, but we haven't because those stats change between releases, often substantially. And simply exposing the string formatted gctrace and forcing people to parse what they want out of it would make users' lives harder without addressing the compatibility problem.

We've been talking about a more general mechanism to expose "unstable" runtime stats. Probably these would be named, with some way to enumerate the current names and fetch the value of a given stat. The set of names would not be guaranteed between versions, but monitoring systems could enumerate and collect them, and applications could use them with the understanding that they may change in the future. Probably the API would fetch a single stat so that expensive stats could be computed only when fetched (versus, for example, exposing all stats at once in a map). And probably the API should have explicit support for "streaming" stats that have a meaningful history, like in the gctrace.

@ianlancetaylor
Copy link
Contributor

The problem is vaguely similar to MIBs used by SNMP, where the tool that examines the stats can not know the exact nature of the stats provided by a specific device.

@agnivade agnivade removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 25, 2019
@rsc
Copy link
Contributor

rsc commented May 28, 2019

On hold for the runtime team.

@prashantv
Copy link
Contributor Author

If this data cannot be exposed through a library function, could it be exposed through a HTTP endpoint on /debug/pprof?

@prashantv
Copy link
Contributor Author

Bumping this, can pprof be used to export the GC data?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants