Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: unexpected return pc for runtime.systemstack_switch #45159

Closed
kkasila opened this issue Mar 22, 2021 · 6 comments
Closed

runtime: unexpected return pc for runtime.systemstack_switch #45159

kkasila opened this issue Mar 22, 2021 · 6 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@kkasila
Copy link

kkasila commented Mar 22, 2021

What version of Go are you using (go version)?

go1.15.2 
go1.15.8
go1.16

Does this issue reproduce with the latest release?

go1.16 is the latest we see this error, with kubectl v1.21.0-beta.1

What operating system and processor architecture are you using (go env)?

x86_64 GNU/Linux

What did you do?

We upgraded kubernetes to 1.19. version. The kubernetes client programs kubectl and helm started crashing occasionally to "runtime: unexpected return pc for runtime.systemstack_switch". We have tried with different kubectl and helm versions , and it looks that the error starts to appear with go 1.15.x . If we downgrade to kubectl 1.18.x (go 13.x) the error does not appear.

We are running this on VM with 8 virtual CPUs. Error comes on both OpenStack and VMWare infras, but error does not come on bare metal. We can reproduce this simply by running "kubectl get nodes "or "helm list" in a busy loop, the crash comes in a few minutes.

crash.txt

What did you expect to see?

kubectl and helm working correctly without go exceptions.

What did you see instead?

Go exception "runtime: unexpected return pc for runtime.systemstack_switch"
@cagedmantis
Copy link
Contributor

crash.txt contains:

runtime: unexpected return pc for runtime.systemstack_switch called from 0xc00034e000
stack: frame={sp:0xc000b415f0, fp:0xc000b415f8} stack=[0xc000b22000,0xc000b42000)
000000c000b414f0:  0000000000000000  0000000000000000
000000c000b41500:  0000000000000000  0000000000000000
000000c000b41510:  0000000000000000  0000000000000000
000000c000b41520:  0000000000000000  0000000000000000
000000c000b41530:  0000000000000000  0000000000000000
000000c000b41540:  0000000000000000  0000000000000000
000000c000b41550:  0000000000000000  0000000000000000
000000c000b41560:  0000000000000000  0000000000000000
000000c000b41570:  0000000000000000  0000000000000000
000000c000b41580:  0000000000000000  0000000000000000
000000c000b41590:  0000000000000000  0000000000000000
000000c000b415a0:  0000000000000000  0000000000000000
000000c000b415b0:  0000000000000000  0000000000000000
000000c000b415c0:  0000000000000000  0000000000000000
000000c000b415d0:  0000000000000000  0000000000000000
000000c000b415e0:  0000000000000000  0000000000000000
000000c000b415f0: <000000c00034e000 >000000c00034e900
000000c000b41600:  0000000000000000  0000000000000000
000000c000b41610:  0000000000000000  0000000000000000
000000c000b41620:  000000c00060eaa0  000000c000038a80
000000c000b41630:  0000000000000000  0000000000000000
000000c000b41640:  000000c0002e8900  000000c0002e9200
000000c000b41650:  0000000000000000  0000000000000000
000000c000b41660:  000000c0002e9b00  000000c0002ea240
000000c000b41670:  0000000000000000  0000000000000000
000000c000b41680:  0000000000000000  0000000000000000
000000c000b41690:  00007ff6cfa67000  0000000000203000
000000c000b416a0:  00000adf45d7e1f3  0000000000000000
000000c000b416b0:  0000000000000000  00007ff6cf8ef0c8
000000c000b416c0:  000000000301f780  000000c000000180
000000c000b416d0:  000000c00093f700  0000000000450b8c <runtime.makeslice+108>
000000c000b416e0:  0000000000000478  0000000001ac5100
000000c000b416f0:  0000000000000001
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x1df797b, 0x11)
	/usr/local/go/src/runtime/panic.go:1117 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000000180, 0x0, 0x0, 0x7fffffff, 0x1ee22d8, 0xc00058daf8, 0x0, ...)
	/usr/local/go/src/runtime/traceback.go:261 +0x1a56
runtime.copystack(0xc000000180, 0x20000)
	/usr/local/go/src/runtime/stack.go:908 +0x2f1
runtime.shrinkstack(0xc000000180)
	/usr/local/go/src/runtime/stack.go:1180 +0x13d
runtime.scanstack(0xc000000180, 0xc000056698)
	/usr/local/go/src/runtime/mgcmark.go:720 +0x58e
runtime.markroot.func1()
	/usr/local/go/src/runtime/mgcmark.go:233 +0xc6
runtime.markroot(0xc000056698, 0x14)
	/usr/local/go/src/runtime/mgcmark.go:206 +0x33e
runtime.gcDrain(0xc000056698, 0x7)
	/usr/local/go/src/runtime/mgcmark.go:1014 +0x118
runtime.gcBgMarkWorker.func2()
	/usr/local/go/src/runtime/mgc.go:2003 +0x17e
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:379 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1246

goroutine 21 [GC worker (idle)]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:339 fp=0xc000065f60 sp=0xc000065f58 pc=0x46de20
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1967 +0x1c7 fp=0xc000065fe0 sp=0xc000065f60 pc=0x41dc47
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x46fb21
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1835 +0x37

goroutine 1 [GC assist marking (scan)]:
runtime: unexpected return pc for runtime.systemstack_switch called from 0xc00034e000
stack: frame={sp:0xc000b415f0, fp:0xc000b415f8} stack=[0xc000b22000,0xc000b42000)
000000c000b414f0:  0000000000000000  0000000000000000
000000c000b41500:  0000000000000000  0000000000000000
000000c000b41510:  0000000000000000  0000000000000000
000000c000b41520:  0000000000000000  0000000000000000
000000c000b41530:  0000000000000000  0000000000000000
000000c000b41540:  0000000000000000  0000000000000000
000000c000b41550:  0000000000000000  0000000000000000
000000c000b41560:  0000000000000000  0000000000000000
000000c000b41570:  0000000000000000  0000000000000000
000000c000b41580:  0000000000000000  0000000000000000
000000c000b41590:  0000000000000000  0000000000000000
000000c000b415a0:  0000000000000000  0000000000000000
000000c000b415b0:  0000000000000000  0000000000000000
000000c000b415c0:  0000000000000000  0000000000000000
000000c000b415d0:  0000000000000000  0000000000000000
000000c000b415e0:  0000000000000000  0000000000000000
000000c000b415f0: <000000c00034e000 >000000c00034e900
000000c000b41600:  0000000000000000  0000000000000000
000000c000b41610:  0000000000000000  0000000000000000
000000c000b41620:  000000c00060eaa0  000000c000038a80
000000c000b41630:  0000000000000000  0000000000000000
000000c000b41640:  000000c0002e8900  000000c0002e9200
000000c000b41650:  0000000000000000  0000000000000000
000000c000b41660:  000000c0002e9b00  000000c0002ea240
000000c000b41670:  0000000000000000  0000000000000000
000000c000b41680:  0000000000000000  0000000000000000
000000c000b41690:  00007ff6cfa67000  0000000000203000
000000c000b416a0:  00000adf45d7e1f3  0000000000000000
000000c000b416b0:  0000000000000000  00007ff6cf8ef0c8
000000c000b416c0:  000000000301f780  000000c000000180
000000c000b416d0:  000000c00093f700  0000000000450b8c <runtime.makeslice+108>
000000c000b416e0:  0000000000000478  0000000001ac5100
000000c000b416f0:  0000000000000001
runtime: unexpected return pc for runtime.systemstack_switch called from 0xc00034e000
stack: frame={sp:0xc000b415f0, fp:0xc000b415f8} stack=[0xc000b22000,0xc000b42000)
000000c000b414f0:  0000000000000000  0000000000000000
000000c000b41500:  0000000000000000  0000000000000000
000000c000b41510:  0000000000000000  0000000000000000
000000c000b41520:  0000000000000000  0000000000000000
000000c000b41530:  0000000000000000  0000000000000000
000000c000b41540:  0000000000000000  0000000000000000
000000c000b41550:  0000000000000000  0000000000000000
000000c000b41560:  0000000000000000  0000000000000000
000000c000b41570:  0000000000000000  0000000000000000
000000c000b41580:  0000000000000000  0000000000000000
000000c000b41590:  0000000000000000  0000000000000000
000000c000b415a0:  0000000000000000  0000000000000000
000000c000b415b0:  0000000000000000  0000000000000000
000000c000b415c0:  0000000000000000  0000000000000000
000000c000b415d0:  0000000000000000  0000000000000000
000000c000b415e0:  0000000000000000  0000000000000000
000000c000b415f0: <000000c00034e000 >000000c00034e900
000000c000b41600:  0000000000000000  0000000000000000
000000c000b41610:  0000000000000000  0000000000000000
000000c000b41620:  000000c00060eaa0  000000c000038a80
000000c000b41630:  0000000000000000  0000000000000000
000000c000b41640:  000000c0002e8900  000000c0002e9200
000000c000b41650:  0000000000000000  0000000000000000
000000c000b41660:  000000c0002e9b00  000000c0002ea240
000000c000b41670:  0000000000000000  0000000000000000
000000c000b41680:  0000000000000000  0000000000000000
000000c000b41690:  00007ff6cfa67000  0000000000203000
000000c000b416a0:  00000adf45d7e1f3  0000000000000000
000000c000b416b0:  0000000000000000  00007ff6cf8ef0c8
000000c000b416c0:  000000000301f780  000000c000000180
000000c000b416d0:  000000c00093f700  0000000000450b8c <runtime.makeslice+108>
000000c000b416e0:  0000000000000478  0000000001ac5100
000000c000b416f0:  0000000000000001
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:339

goroutine 18 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x301e8a0)
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1169 +0x8b
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
	/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:417 +0xd

@kkasila
Copy link
Author

kkasila commented Mar 23, 2021

Linux kernel is 4.18.0 . (#35592 had same error, but on linux 5.2 which is not the case here)

@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 24, 2021
@cagedmantis cagedmantis added this to the Backlog milestone Mar 24, 2021
@cagedmantis
Copy link
Contributor

/cc @aclements @randall77 @mknyszek @prattmic

@prattmic
Copy link
Member

@kkasila How frequently does this crash occur? Very rarely? Every run?

It sounds like you tested kubectl 1.18 on Go 1.13 and kubectl 1.19 on Go 1.15 and 1.16? Is it possible you could try kubectl 1.18 on Go 1.15, or kubectl 1.19 on Go 1.13 so we can narrow down whether the kubectl or Go version change is triggering the crash?

@networkimprov
Copy link

@gopherbot add WaitingForInfo

@gopherbot gopherbot added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 27, 2021
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Apr 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
5 participants
@cagedmantis @networkimprov @prattmic @gopherbot @kkasila