Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/pprof: inline frames may not use combined location #37446

Closed
prattmic opened this issue Feb 25, 2020 · 5 comments
Closed

runtime/pprof: inline frames may not use combined location #37446

prattmic opened this issue Feb 25, 2020 · 5 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@prattmic
Copy link
Member

prattmic commented Feb 25, 2020

What version of Go are you using (go version)?

$ go version
go version devel +a50c3ffbd4 linux/amd64

Does this issue reproduce with the latest release?

Regression in Go 1.14

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/mpratt/.cache/go-build"
GOENV="/usr/local/google/home/mpratt/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/mpratt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/google-golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build076660292=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Collect a heap profile from a profile with stacks truncated in the middle of inline frames: https://play.golang.org/p/6--6a10WPCa

What did you expect to see?

Consistent profile output.

What did you see instead?

http://golang.org/cl/204636 attempts to group inlined frames into the same Location, as required by the pprof format. Once a location is created, it is cached for reuse by future identical frames.

However, the runtime may truncate stacks, potentially truncating in between inlined frames. If the truncated frame is encountered first, then separate locations will be created for the two frames. If a complete frame is encountered first, then a combined location will be use for both frames.

This behavior is nondeterministic, so similar profiles may different:

case1
case2

This may be particularly problematic for tools looking at diffs between two profiles.

The former has location:

    30: 0x4bb6df M=1 main.depth1 /usr/local/google/home/mpratt/Downloads/alloc.go:18 s=0
             main.main /usr/local/google/home/mpratt/Downloads/alloc.go:312 s=0

The latter has locations:

    31: 0x4bb6df M=1 main.depth1 /usr/local/google/home/mpratt/Downloads/alloc.go:18 s=0
    33: 0x4bb6da M=1 main.main /usr/local/google/home/mpratt/Downloads/alloc.go:312 s=0

cc @hyangah @heschik

@hyangah
Copy link
Contributor

hyangah commented Feb 25, 2020

@gopherbot please consider this for backport to 1.14, it's a regression.

@gopherbot
Copy link

Backport issue(s) opened: #37447 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@prattmic
Copy link
Member Author

To clarify, there are two related issues here:

  1. Go 1.14 will nondeterministically generate profiles with either combined or split stacks (the two screenshots above). 1.13 always generated split stacks in this case.
  2. When the stacks are split, 1.14 loses the inline information on depth1. (1.13 never had inline information).

@gopherbot
Copy link

Change https://golang.org/cl/220377 mentions this issue: runtime/pprof: deterministic handling of truncated stacks

@cagedmantis cagedmantis added this to the Backlog milestone Feb 28, 2020
@cagedmantis cagedmantis added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 28, 2020
@gopherbot
Copy link

Change https://golang.org/cl/221577 mentions this issue: runtime/pprof: expand final stack frame to avoid truncation

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

4 participants