Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/debug/cmd/viewcore: error while trying to run command "objects" #64527

Open
chenk008 opened this issue Dec 4, 2023 · 4 comments
Open

x/debug/cmd/viewcore: error while trying to run command "objects" #64527

chenk008 opened this issue Dec 4, 2023 · 4 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@chenk008
Copy link

chenk008 commented Dec 4, 2023 •
edited

Go version

go version go1.20.11 linux/amd64

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.11"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/debug/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build559913390=/tmp/go-build -gno-record-gcc-switches"

What did you do?

  1. user gcore to dump kubelet , saved to core.70006
  2. run viewcore ~/core.70006 --exe /usr/bin/kubelet and objects
Error while trying to run command "objects": address 3b79634f6c6c754e is not mapped in the core file
Stack: goroutine 1 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
main.capturePanic.func1()
	/root/debug/cmd/viewcore/main.go:372 +0x45
panic({0x7e3a00, 0xc01cd31600})
	/usr/local/go/src/runtime/panic.go:884 +0x213
golang.org/x/debug/internal/core.(*Process).ReadUint64(0x80d3e0?, 0x9b0000c0265230f0?)
	/root/debug/internal/core/read.go:81 +0x1b3
golang.org/x/debug/internal/core.(*Process).ReadUintptr(0x30?, 0xc01450aeb8?)
	/root/debug/internal/core/read.go:120 +0x36
golang.org/x/debug/internal/gocore.region.Uintptr({0xc008ce6000?, 0xc019637410?, 0xc0000c8000?})
	/root/debug/internal/gocore/region.go:40 +0x36
golang.org/x/debug/internal/gocore.(*Process).runtimeType2Type(0xc008ce6000, 0x3b79634f6c6c754e, 0x852ff4?)
	/root/debug/internal/gocore/type.go:148 +0x1aa
golang.org/x/debug/internal/gocore.(*Process).typeObject(0xc008ce6000, 0xc0014b42c8, 0xc00c5e0a50, {0x8f56d0, 0xc008cdc000}, 0xc026523590)
	/root/debug/internal/gocore/type.go:630 +0x98f
golang.org/x/debug/internal/gocore.(*Process).typeObject(0xc008ce6000, 0xc0014b42a0, 0xc00c5e0d20, {0x8f56d0, 0xc008cdc000}, 0xc026523590)
	/root/debug/internal/gocore/type.go:740 +0x51e
golang.org/x/debug/internal/gocore.(*Process).typeHeap.func1()
	/root/debug/internal/gocore/type.go:523 +0x3f4
sync.(*Once).doSlow(0xc01cd28d20?, 0x4100c5?)
	/usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:65
golang.org/x/debug/internal/gocore.(*Process).typeHeap(0x413464?)
	/root/debug/internal/gocore/type.go:377 +0x47
golang.org/x/debug/internal/gocore.(*Process).Type(0xc008ce6000, 0xc011ecc030?)
	/root/debug/internal/gocore/object.go:242 +0x25
main.typeName(0xc008ce6000, 0xc000000000)
	/root/debug/cmd/viewcore/main.go:768 +0xac
main.runObjects.func1(0x7eae40?)
	/root/debug/cmd/viewcore/main.go:612 +0x2c
golang.org/x/debug/internal/gocore.(*Process).ForEachObject(0xc008ce6000, 0xc026523a08)
	/root/debug/internal/gocore/object.go:202 +0xfa
main.runObjects(0xb5c100?, {0xb926c0?, 0x0?, 0x0?})
	/root/debug/cmd/viewcore/main.go:611 +0x87
github.com/spf13/cobra.(*Command).execute(0xb5c100, {0xb926c0, 0x0, 0x0})
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0xc000c8a000)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2e5
github.com/spf13/cobra.(*Command).Execute(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.runRoot.func2()
	/root/debug/cmd/viewcore/main.go:361 +0x8a
main.capturePanic(0x8f3e20?)
	/root/debug/cmd/viewcore/main.go:376 +0x62
main.runRoot(0xb5aba0, {0xc000038240?, 0x2?, 0x2?})
	/root/debug/cmd/viewcore/main.go:358 +0x6e5
github.com/spf13/cobra.(*Command).execute(0xb5aba0, {0xc0000240a0, 0x2, 0x2})
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0xb5aba0)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2e5
github.com/spf13/cobra.(*Command).Execute(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
	/root/debug/cmd/viewcore/main.go:244 +0x12a

What did you expect to see?

viewcore can analyze this coredump

What did you see instead?

  1. viewcore panic and fail
  2. I have tried multiple golang versions and kubelet versions, it always paniced on address 3b79634f6c6c754e
@chenk008
Copy link
Author

chenk008 commented Dec 4, 2023

Maybe it is related to how kubelet is built?

@chenk008
Copy link
Author

chenk008 commented Dec 4, 2023

Similar issue #64442.

After run object, the next command object works.

@chenk008
Copy link
Author

chenk008 commented Dec 4, 2023 •
edited

initTypeHeap run once at the first command is running.
When the next command runs, gocore.Process is reused because of coreCache. The panic is absence.

The type which failed to parse is KindIface k8s.io/kubernetes/pkg/kubelet/volumemanager/cache.DesiredStateOfWorld, it is an interface.

@seankhliao seankhliao changed the title viewcore: Error while trying to run command "objects": address 3b79634f6c6c754e is not mapped in the core file x/debug/cmd/viewcore: error while trying to run command "objects": address 3b79634f6c6c754e is not mapped in the core file Dec 4, 2023
@gopherbot gopherbot added this to the Unreleased milestone Dec 4, 2023
@cagedmantis cagedmantis changed the title x/debug/cmd/viewcore: error while trying to run command "objects": address 3b79634f6c6c754e is not mapped in the core file x/debug/cmd/viewcore: error while trying to run command "objects" Dec 4, 2023
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 4, 2023
@cagedmantis
Copy link
Contributor

cc @golang/runtime

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@cagedmantis @chenk008 @gopherbot