Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/debug: viewcore tool don't support dumps with go1.13.3+ #38638

Closed
bozaro opened this issue Apr 24, 2020 · 7 comments
Closed

x/debug: viewcore tool don't support dumps with go1.13.3+ #38638

bozaro opened this issue Apr 24, 2020 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@bozaro
Copy link

bozaro commented Apr 24, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.2 linux/amd64
go version go1.13.3 linux/amd64
go version go1.14.2 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bozaro/.cache/go-build"
GOENV="/home/bozaro/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB="github.com/joomcode/*"
GOOS="linux"
GOPATH="/home/bozaro/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/bozaro/sdk/go-1.13.3"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/bozaro/sdk/go-1.13.3/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/bozaro/github/debug/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build072895212=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Make some dumps with steps from https://github.com/golang/debug/blob/master/internal/gocore/testdata/README

With go 1.13.2 it's works as expected:

> go build -o test . && GOTRACEBACK=crash ./test || viewcore core --exe test histogram
[1]    19749 abort (core dumped)  GOTRACEBACK=crash ./test
 count size bytes type
     5  384  1920 runtime.g
     1 1152  1152 [70+2?]string
     1  896   896 runtime.m
     1  480   480 [15]internal/cpu.option
     1  112   112 unk112
     1   96    96 [12]*runtime.p
     1   96    96 hchan<bool>
     1   64    64 runtime.timer
     1   32    32 []*runtime.moduledata
     1   32    32 [4]*runtime.g
     1   16    16 [4]uint32
     1   16    16 string

But with go 1.13.3 I got panic:

[1]    23671 abort (core dumped)  GOTRACEBACK=crash ./test
panic: check failed for heap: 67108864 vs 720896

goroutine 1 [running]:
golang.org/x/debug/internal/gocore.(*Process).readSpans.func2(0xc0006d61e0)
	/home/bozaro/github/debug/internal/gocore/process.go:457 +0x1e3
golang.org/x/debug/internal/gocore.(*Process).readSpans.func2(0xc0006d60f0)
	/home/bozaro/github/debug/internal/gocore/process.go:460 +0x8f
golang.org/x/debug/internal/gocore.(*Process).readSpans(0xc000158000, 0xc000158000, 0x5717c0, 0xc0003b1040, 0xc000791a70, 0x1, 0x1)
	/home/bozaro/github/debug/internal/gocore/process.go:463 +0x1bd0
golang.org/x/debug/internal/gocore.(*Process).readHeap(0xc000158000)
	/home/bozaro/github/debug/internal/gocore/process.go:262 +0x77b
golang.org/x/debug/internal/gocore.Core(0xc00020e000, 0x4, 0x0, 0x0)
	/home/bozaro/github/debug/internal/gocore/process.go:156 +0x2c4
main.readCore(0xc00020a300, 0x8bfb3d, 0x3, 0x0)
	/home/bozaro/github/debug/cmd/viewcore/main.go:265 +0xd2
main.runHistogram(0xc50120, 0xc0001263c0, 0x0, 0x2)
	/home/bozaro/github/debug/cmd/viewcore/main.go:457 +0xdd
github.com/spf13/cobra.(*Command).execute(0xc50120, 0xc000126360, 0x2, 0x2, 0xc50120, 0xc000126360)
	/home/bozaro/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xc4f7a0, 0x8bf822, 0x1, 0xc000207f00)
	/home/bozaro/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2ea
github.com/spf13/cobra.(*Command).Execute(...)
	/home/bozaro/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
	/home/bozaro/github/debug/cmd/viewcore/main.go:243 +0x104

With go 1.14.2 I got another panic:

[1]    23046 abort (core dumped)  GOTRACEBACK=crash ./test
panic: bad uint8 type runtime.mSpanStateBox

goroutine 1 [running]:
golang.org/x/debug/internal/gocore.region.Uint8(0xc0000be9a0, 0x7fb61e18516b, 0xc000433720, 0x8c019d)
	/home/bozaro/github/debug/internal/gocore/region.go:100 +0xc6
golang.org/x/debug/internal/gocore.(*Process).readSpans(0xc0000be9a0, 0xc0000be9a0, 0x4e7280, 0xc000706550, 0xc000620300, 0x1, 0x1)
	/home/bozaro/github/debug/internal/gocore/process.go:349 +0x566
golang.org/x/debug/internal/gocore.(*Process).readHeap(0xc0000be9a0)
	/home/bozaro/github/debug/internal/gocore/process.go:262 +0x77b
golang.org/x/debug/internal/gocore.Core(0xc0000f2000, 0x4, 0x0, 0x0)
	/home/bozaro/github/debug/internal/gocore/process.go:156 +0x2c4
main.readCore(0xc0000ee300, 0x8bfb3d, 0x3, 0x0)
	/home/bozaro/github/debug/cmd/viewcore/main.go:265 +0xd2
main.runHistogram(0xc50120, 0xc00000e3c0, 0x0, 0x2)
	/home/bozaro/github/debug/cmd/viewcore/main.go:457 +0xdd
github.com/spf13/cobra.(*Command).execute(0xc50120, 0xc00000e360, 0x2, 0x2, 0xc50120, 0xc00000e360)
	/home/bozaro/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xc4f7a0, 0x8bf822, 0x1, 0xc0000ebf00)
	/home/bozaro/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2ea
github.com/spf13/cobra.(*Command).Execute(...)
	/home/bozaro/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
	/home/bozaro/github/debug/cmd/viewcore/main.go:243 +0x104

I try update golang.org/x/sys vendor to v0.0.0-20200420163511-1957bb5e6d1f but behavior has not changed.
@gopherbot gopherbot added this to the Unreleased milestone Apr 24, 2020
@ianlancetaylor
Copy link
Contributor

CC @randall77

@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 25, 2020
@ianlancetaylor ianlancetaylor modified the milestones: Unreleased, Go1.15 Apr 25, 2020
@gopherbot
Copy link

Change https://golang.org/cl/232158 mentions this issue: cmd/viewcore: account for partially allocated arenas

@gopherbot
Copy link

Change https://golang.org/cl/232160 mentions this issue: cmd/viewcore: handle typedef'd mspan.state

@gopherbot
Copy link

Change https://golang.org/cl/232161 mentions this issue: cmd/viewcore: handle 1.14 free spans

bozaro pushed a commit to bozaro/debug that referenced this issue May 6, 2020
@randall77 @bozaro
cmd/viewcore: account for partially allocated arenas
7b28f3b 
Arenas used to be allocated completely (or at least, have a span
allocated to represent the unused space). Since 1.13.3, the latest
arena only has spans up to its highest allocation point.

Fixes viewcore for 1.13.3.

This change was CL 189957, backported to 1.13 at CL 198485.

Update golang/go#38638

Change-Id: Ia8cb14925cf6f4226a80e7ff4f011aa93e8b41d1
bozaro pushed a commit to bozaro/debug that referenced this issue May 6, 2020
@randall77
cmd/viewcore: account for partially allocated arenas
b62320d 
Arenas used to be allocated completely (or at least, have a span
allocated to represent the unused space). Since 1.13.3, the latest
arena only has spans up to its highest allocation point.

Fixes viewcore for 1.13.3.

This change was CL 189957, backported to 1.13 at CL 198485.

Update golang/go#38638

Change-Id: Ia8cb14925cf6f4226a80e7ff4f011aa93e8b41d1
bozaro pushed a commit to bozaro/debug that referenced this issue May 6, 2020
@randall77
cmd/viewcore: handle typedef'd mspan.state
a1fc89a 
mspan.state used to (go1.13) be a uint8, now (go1.14) it is a struct
with a uint8 field.

Update golang/go#38638

Change-Id: I042787a61479808d37afa0a7f8caed06b5a18567
bozaro pushed a commit to bozaro/debug that referenced this issue May 6, 2020
@randall77
cmd/viewcore: handle 1.14 free spans
50bc1c4 
Free spans are now implicit. Derive the free span information
by finding all the memory in the inuse address ranges that isn't
attributed to any span.

Fixes viewcore for all versions of 1.14.
Also works for tip.

Fixes golang/go#38638

Change-Id: Ied283f59391aad8ed5b1b657d2fd6e89e9e9ae6c
@ppggff
Copy link

ppggff commented May 7, 2020

With commits in bozaro's repo, test.go works. But my go program(with cgo) still panic:

go version go1.14.1 linux/amd64

panic: check failed for heap: 1598029824 vs 1282605056

goroutine 1 [running]:
golang.org/x/debug/internal/gocore.(*Process).readSpans.func2(0xc0037ea180)
	/home/gpadmin/go/src/golang.org/x/debug/internal/gocore/process.go:515 +0x1e3
golang.org/x/debug/internal/gocore.(*Process).readSpans.func2(0xc0037ea090)
	/home/gpadmin/go/src/golang.org/x/debug/internal/gocore/process.go:518 +0x8f
golang.org/x/debug/internal/gocore.(*Process).readSpans(0xc0011cc000, 0xc0011cc000, 0x16f1e20, 0xc0013fa550, 0xc000104000, 0x18, 0x20)
	/home/gpadmin/go/src/golang.org/x/debug/internal/gocore/process.go:521 +0x237e
golang.org/x/debug/internal/gocore.(*Process).readHeap(0xc0011cc000)
	/home/gpadmin/go/src/golang.org/x/debug/internal/gocore/process.go:262 +0x775
golang.org/x/debug/internal/gocore.Core(0xc0000f0000, 0xa, 0x0, 0x0)
	/home/gpadmin/go/src/golang.org/x/debug/internal/gocore/process.go:156 +0x2c4
main.readCore(0x49d8a9, 0xc0000e9ba8, 0x0, 0xc000089520)
	/home/gpadmin/go/src/golang.org/x/debug/cmd/viewcore/main.go:265 +0xd2
main.runRoot(0xc55ee0, 0xc000092320, 0x0, 0x2)
	/home/gpadmin/go/src/golang.org/x/debug/cmd/viewcore/main.go:287 +0x45
github.com/spf13/cobra.(*Command).execute(0xc55ee0, 0xc00009c020, 0x2, 0x2, 0xc55ee0, 0xc00009c020)
	/home/gpadmin/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xc55ee0, 0x8d3ce3, 0x1, 0xc0000e9f00)
	/home/gpadmin/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2ea
github.com/spf13/cobra.(*Command).Execute(...)
	/home/gpadmin/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
	/home/gpadmin/go/src/golang.org/x/debug/cmd/viewcore/main.go:243 +0x111

gopherbot pushed a commit to golang/debug that referenced this issue May 7, 2020
@randall77
cmd/viewcore: account for partially allocated arenas
416347b 
Arenas used to be allocated completely (or at least, have a span
allocated to represent the unused space). Since 1.13.3, the latest
arena only has spans up to its highest allocation point.

Fixes viewcore for 1.13.3.

This change was CL 189957, backported to 1.13 at CL 198485.

Update golang/go#38638

Change-Id: Ia8cb14925cf6f4226a80e7ff4f011aa93e8b41d1
Reviewed-on: https://go-review.googlesource.com/c/debug/+/232158
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/debug that referenced this issue May 7, 2020
@randall77
cmd/viewcore: handle typedef'd mspan.state
cdf388b 
mspan.state used to (go1.13) be a uint8, now (go1.14) it is a struct
with a uint8 field.

Update golang/go#38638

Change-Id: I042787a61479808d37afa0a7f8caed06b5a18567
Reviewed-on: https://go-review.googlesource.com/c/debug/+/232160
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@ianlancetaylor
Copy link
Contributor

@randall77 Is there anything else to do here?

@ianlancetaylor ianlancetaylor modified the milestones: Go1.15, Unreleased Jun 26, 2020
@randall77
Copy link
Contributor

Yes, just waiting on a +2. I'll ping the CL.

@jordanlewis jordanlewis mentioned this issue May 6, 2021
Open
@golang golang locked and limited conversation to collaborators Jun 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@bozaro @ppggff @ianlancetaylor @randall77 @gopherbot