Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/debug/cmd/viewcore: viewcore panics: bad int32 type uint32 #45984

Open
jordanlewis opened this issue May 6, 2021 · 4 comments
Open

x/debug/cmd/viewcore: viewcore panics: bad int32 type uint32 #45984

jordanlewis opened this issue May 6, 2021 · 4 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@jordanlewis
Copy link

What version of Go are you using (go version)?

1.16.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
ubuntu@jordan-test-0001:~$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1750380533=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I attempted to use viewcore on a core dump from CockroachDB. I ran the CockroachDB binary, killed it with SIGSEGV, and collected the resultant dump. Then, I ran viewcore with the overview argument.

./go/bin/viewcore /mnt/data1/cores/core.cockroach.13188.jordan-test-0001.1620268383 --exe ./cockroach overview

What did you expect to see?

Viewcore output, which could hopefully help me track down some issues with memory usage. 😄

What did you see instead?

A panic:

ubuntu@jordan-test-0001:~$ ./go/bin/viewcore /mnt/data1/cores/core.cockroach.13188.jordan-test-0001.1620268383 --exe ./cockroach overview
panic: bad int32 type uint32

goroutine 1 [running]:
golang.org/x/debug/internal/gocore.region.Int32(0xc0004fa000, 0x777bdec, 0xc000182550, 0x86201a)
        /home/ubuntu/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/region.go:82 +0xc6
golang.org/x/debug/internal/gocore.(*module).readFunc(0xc00f4cee70, 0xc0004fa000, 0x777bdd8, 0xc000aa86e0, 0xc0004fa000, 0x80ee0e8, 0xc000183d60, 0xc000182280)
        /home/ubuntu/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/module.go:62 +0x22e
golang.org/x/debug/internal/gocore.(*Process).readModule(0xc0004fa000, 0xc0004fa000, 0x80ee080, 0xc000a58a50, 0x80ee080)
        /home/ubuntu/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/module.go:45 +0x4cc
golang.org/x/debug/internal/gocore.(*Process).readModules(0xc0004fa000)
        /home/ubuntu/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/module.go:26 +0x1a7
golang.org/x/debug/internal/gocore.Core(0xc000162000, 0x41, 0x0, 0x0)
        /home/ubuntu/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/process.go:156 +0x2c8
main.readCore(0x78f037, 0xc00015c700, 0xc00011fd80, 0xc00011fda8)
        /home/ubuntu/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/cmd/viewcore/main.go:266 +0xdb
main.runOverview(0xb05b80, 0xc000034260, 0x0, 0x2)
        /home/ubuntu/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/cmd/viewcore/main.go:378 +0x34
github.com/spf13/cobra.(*Command).execute(0xb05b80, 0xc000034200, 0x2, 0x2, 0xb05b80, 0xc000034200)
        /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xb05200, 0x0, 0xac76e0, 0xc00007a058)
        /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2fe
github.com/spf13/cobra.(*Command).Execute(...)
        /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
        /home/ubuntu/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/cmd/viewcore/main.go:244 +0x125

Here is a link to the core dump (334 MB): https://drive.google.com/file/d/1KlwSvP-5xxghBRHprxsmtFFQtsK9zAmg/view?usp=sharing
Here is a link to the binary used (178 MB): https://drive.google.com/file/d/147h4wTi7WloFWPjIXm4kZoIGPNvu6QNz/view?usp=sharing

I know this is a bit of a heavy repro, but I'm not quite sure how else to track down the issue. Any ideas would be appreciated for how to produce a smaller case for you to look at.

@randall77, would you have an idea about this issue? I saw #38638, but this appears to be a different issue.

Thank you!
@gopherbot gopherbot added this to the Unreleased milestone May 6, 2021
@jordanlewis jordanlewis mentioned this issue May 9, 2021
Closed
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 10, 2021
@cagedmantis
Copy link
Contributor

/cc @hyangah @randall77

@cagedmantis cagedmantis changed the title x/debug: viewcore panics: bad int32 type uint32 x/debug/cmd/viewcore: viewcore panics: bad int32 type uint32 May 10, 2021
@jordanlewis
Copy link
Author

This happens even with a trivial program, which I should have checked first.

package main

func main() {
    panic("no")
}

If I run this with GOTRACEBACK=crash, the resultant core crashes:

[22:24:2]% viewcore core
panic: bad int32 type uint32

goroutine 1 [running]:
golang.org/x/debug/internal/gocore.region.Int32(0xc00016a580, 0x4b50fc, 0xc00014a2d0, 0x86201a)
        /home/jordan/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/region.go:82 +0xc6
golang.org/x/debug/internal/gocore.(*module).readFunc(0xc0003cdb00, 0xc00016a580, 0x4b50e8, 0xc000260aa0, 0xc00016a580, 0x4c8888, 0xc0000c5810, 0xc00014a0a0)
        /home/jordan/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/module.go:62 +0x22e
golang.org/x/debug/internal/gocore.(*Process).readModule(0xc00016a580, 0xc00016a580, 0x4c8820, 0xc0000c5ea0, 0x4c8820)
        /home/jordan/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/module.go:45 +0x4cc
golang.org/x/debug/internal/gocore.(*Process).readModules(0xc00016a580)
        /home/jordan/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/module.go:26 +0x1a7
golang.org/x/debug/internal/gocore.Core(0xc000192000, 0x4, 0x0, 0x0)
        /home/jordan/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/internal/gocore/process.go:156 +0x2c8
main.readCore(0x747c25, 0x7f3820, 0xc00011afc0, 0x862026)
        /home/jordan/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/cmd/viewcore/main.go:266 +0xdb
main.runRoot(0xb05200, 0xb3c0d0, 0x0, 0x0)
        /home/jordan/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/cmd/viewcore/main.go:288 +0x55
github.com/spf13/cobra.(*Command).execute(0xb05200, 0xc000138010, 0x0, 0x0, 0xb05200, 0xc000138010)
        /home/jordan/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xb05200, 0x0, 0xac76e0, 0xc00010e058)
        /home/jordan/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2fe
github.com/spf13/cobra.(*Command).Execute(...)
        /home/jordan/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
        /home/jordan/go/pkg/mod/golang.org/x/debug@v0.0.0-20210309195612-26ed8fdb6258/cmd/viewcore/main.go:244 +0x125

@jordanlewis
Copy link
Author

jordanlewis commented May 15, 2021 •
edited

The problem here seems to be that the types of npcdata and pscp were changed in _func to be uint32.

https://github.com/golang/go/blame/3b304ce7fe35b9d1e8cf0b0518ed2550c361a010/src/runtime/runtime2.go#L876

Fixing that locally seems to improve matters! But, there are still some things broken even after patching debug to expect uint32 instead of int32 on those slots.

diff --git a/internal/gocore/module.go b/internal/gocore/module.go
index 2f5e632..95c10b7 100644
--- a/internal/gocore/module.go
+++ b/internal/gocore/module.go
@@ -59,23 +59,23 @@ func (m *module) readFunc(r region, pcln region) *Func {
        f := &Func{module: m, r: r}
        f.entry = core.Address(r.Field("entry").Uintptr())
        f.name = r.p.proc.ReadCString(pcln.SliceIndex(int64(r.Field("nameoff").Int32())).a)
-       f.frameSize.read(r.p.proc, pcln.SliceIndex(int64(r.Field("pcsp").Int32())).a)
+       f.frameSize.read(r.p.proc, pcln.SliceIndex(int64(r.Field("pcsp").Uint32())).a)

        // Parse pcdata and funcdata, which are laid out beyond the end of the _func.
        a := r.a.Add(int64(r.p.findType("runtime._func").Size))
-       n := r.Field("npcdata").Int32()
-       for i := int32(0); i < n; i++ {
+       n := r.Field("npcdata").Uint32()
+       for i := uint32(0); i < n; i++ {
                f.pcdata = append(f.pcdata, r.p.proc.ReadInt32(a))
                a = a.Add(4)
        }
        a = a.Align(r.p.proc.PtrSize())

        if nfd := r.Field("nfuncdata"); nfd.typ.Size == 1 { // go 1.12 and beyond, this is a uint8
-               n = int32(nfd.Uint8())
+               n = uint32(nfd.Uint8())
        } else { // go 1.11 and earlier, this is an int32
-               n = nfd.Int32()
+               n = uint32(nfd.Int32())
        }
-       for i := int32(0); i < n; i++ {
+       for i := uint32(0); i < n; i++ {
                f.funcdata = append(f.funcdata, r.p.proc.ReadPtr(a))
                a = a.Add(r.p.proc.PtrSize())

Some cores give messages like this:

warning: giving up on backtrace: cannot read frame size at pc=0x465661: can't find pctab entry for offset 0x21
warning: giving up on backtrace: cannot read frame size at pc=0x4354c5: can't find pctab entry for offset 0xc1
warning: giving up on backtrace: cannot read frame size at pc=0x4354c5: can't find pctab entry for offset 0xc1
warning: giving up on backtrace: cannot read frame size at pc=0x4354c5: can't find pctab entry for offset 0xc1
warning: giving up on backtrace: cannot read frame size at pc=0x4354c5: can't find pctab entry for offset 0xc1

Some cores fail to load altogether:

warning: giving up on backtrace: cannot find func for pc=0x99e0ffffffffff
panic: address 0 is not mapped in the core file

goroutine 1 [running]:
golang.org/x/debug/internal/core.(*Process).ReadUint32(0xc0001b2000, 0x0, 0xc0003c5020)
        /home/jordan/go/src/github.com/golang/debug/internal/core/read.go:63 +0x24c
golang.org/x/debug/internal/core.(*Process).ReadInt32(...)
        /home/jordan/go/src/github.com/golang/debug/internal/core/read.go:107
golang.org/x/debug/internal/gocore.region.Int32(0xc00018e840, 0x0, 0xc0000c7a90, 0x861ab7)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/region.go:84 +0x4e
golang.org/x/debug/internal/gocore.(*Process).readFrame(0xc00018e840, 0xc000084f90, 0x430705, 0x430705, 0x2, 0xc00018e840)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/process.go:662 +0x7f6
golang.org/x/debug/internal/gocore.(*Process).readG(0xc00018e840, 0xc00018e840, 0xc000000600, 0xc0000c7e50, 0xc000000600)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/process.go:603 +0x595
golang.org/x/debug/internal/gocore.(*Process).readGs(0xc00018e840)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/process.go:550 +0x12d
golang.org/x/debug/internal/gocore.Core(0xc0001b2000, 0x32, 0x0, 0x0)
        /home/jordan/go/src/github.com/golang/debug/internal/gocore/process.go:158 +0x2e5
main.readCore(0x747c25, 0x7f3820, 0xc0000d2fc0, 0x862026)
        /home/jordan/go/src/github.com/golang/debug/cmd/viewcore/main.go:266 +0xdb
main.runRoot(0xb04200, 0xb3b0d0, 0x0, 0x0)
        /home/jordan/go/src/github.com/golang/debug/cmd/viewcore/main.go:288 +0x55
github.com/spf13/cobra.(*Command).execute(0xb04200, 0xc000020050, 0x0, 0x0, 0xb04200, 0xc000020050)
        /home/jordan/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xb04200, 0x0, 0xac66e0, 0xc0000c8058)
        /home/jordan/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2fe
github.com/spf13/cobra.(*Command).Execute(...)
        /home/jordan/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
        /home/jordan/go/src/github.com/golang/debug/cmd/viewcore/main.go:244 +0x125

I'm wondering, perhaps there are some more updates that need to be applied for a more recent version of Go here?

@jordanlewis
Copy link
Author

I have submitted a PR to fix this problem and another. Please take a look: golang/debug#7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@jordanlewis @cagedmantis @gopherbot