Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: random failure after upgrade to v1.16 #45537

Closed
fanqiehc opened this issue Apr 13, 2021 · 7 comments
Closed

cmd/compile: random failure after upgrade to v1.16 #45537

fanqiehc opened this issue Apr 13, 2021 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@fanqiehc
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.16.3 linux/amd64

Does this issue reproduce with the latest release?

Yes.
v1.16.3 / v1.16.2 / v1.16.0

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY="keyayun.io"
GONOSUMDB="keyayun.io"
GOOS="linux"
GOPATH="/go"
GOPRIVATE="keyayun.io"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.2"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2845262410=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Compiling a server app inside a Kubernetes Pod. The pod is created by Jenkins.

What did you expect to see?

Build without errors.

What did you see instead?

About 1/10 percent random failures with different stacktrace.

17:09:37  CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags "-X pkg/common.Version=v0.1.0-179-g58d38022 -X pkg/common.BuildDate=2021-04-10T09:09:37Z -X pkg/common.CommitHash=58d38022a33324ca2fcc306001703e529bcfba19" -o build/flow-server-linux-amd64 ./cmd/flow-server
17:09:59  # gopkg.in/jcmturner/rpc.v1/ndr
17:09:59  /go/pkg/mod/gopkg.in/jcmturner/rpc.v1@v1.1.0/ndr/pipe.go:10:9: internal compiler error: '(*Decoder).fillPipe': panic during decompose user while compiling (*Decoder).fillPipe:
17:09:59  
17:09:59  runtime error: growslice: cap out of range
17:09:59  
17:09:59  goroutine 29 [running]:
17:09:59  cmd/compile/internal/ssa.Compile.func1(0xc00061ce58, 0xc000fb3e40)
17:09:59  	/usr/local/go/src/cmd/compile/internal/ssa/compile.go:48 +0xa5
17:09:59  panic(0xc36f80, 0xdb4180)
17:09:59  	/usr/local/go/src/runtime/panic.go:965 +0x1b9
17:09:59  cmd/compile/internal/ssa.decomposeUserStructInto(0xc000fb3e40, 0xdc40b8, 0xc0005c8780, 0xc00010e180, 0x0, 0x0, 0x0, 0xc00102d500, 0x9, 0x10, ...)
17:09:59  	/usr/local/go/src/cmd/compile/internal/ssa/decompose.go:320 +0x709
17:09:59  cmd/compile/internal/ssa.decomposeUser(0xc000fb3e40)
17:09:59  	/usr/local/go/src/cmd/compile/internal/ssa/decompose.go:242 +0x47d
17:09:59  cmd/compile/internal/ssa.Compile(0xc000fb3e40)
17:09:59  	/usr/local/go/src/cmd/compile/internal/ssa/compile.go:96 +0x98d
17:09:59  cmd/compile/internal/gc.buildssa(0xc000595080, 0x0, 0x0)
17:09:59  	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:470 +0x11ba
17:09:59  cmd/compile/internal/gc.compileSSA(0xc000595080, 0x0)
17:09:59  	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:319 +0x5d
17:09:59  cmd/compile/internal/gc.compileFunctions.func2(0xc000765f80, 0xc000112710, 0x0)
17:09:59  	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:384 +0x4d
17:09:59  created by cmd/compile/internal/gc.compileFunctions
17:09:59  	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:382 +0x129
17:09:59  
17:09:59  
17:09:59  

-----------

18:52:14  # k8s.io/apimachinery/pkg/fields
18:52:14  runtime: unexpected return pc for cmd/compile/internal/ssa.Compile called from 0x2f736e696b6e656a
18:52:14  stack: frame={sp:0xc000b0fc88, fp:0xc000b0fc90} stack=[0xc000b00000,0xc000b10000)
18:52:14  000000c000b0fb88:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fb98:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fba8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fbb8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fbc8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fbd8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fbe8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fbf8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc08:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc18:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc28:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc38:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc48:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc58:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc68:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc78:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fc88: <2f736e696b6e656a >6f772f746e656761 
18:52:14  000000c000b0fc98:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fca8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fcb8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fcc8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fcd8:  0000000000000000  bfe62e42fefa39ef 
18:52:14  000000c000b0fce8:  0000000000000000  0000000000000000 
18:52:14  000000c000b0fcf8:  0000000000000000  762e3562726b6f67 
18:52:14  000000c000b0fd08:  746e65696c632f37  0000000000000000 
18:52:14  000000c000b0fd18:  0000000000000000  0000000000000020 
18:52:14  000000c000b0fd28:  000000000000002f  0000000000000030 
18:52:14  000000c000b0fd38:  0000000000527a38 <cmd/compile/internal/syntax.(*parser).appendGroup+216>  000000c00011e160 
18:52:14  000000c000b0fd48:  000000000000000b  0000000000000012 
18:52:14  000000c000b0fd58:  000000000000000e  000000c0007e8ab8 
18:52:14  000000c000b0fd68:  000000c0007e89d8  000000c0007e8b28 
18:52:14  000000c000b0fd78:  000000c00011e180  000000c000621560 
18:52:14  000000c000b0fd88:  000000c0003c8580 
18:52:14  fatal error: unknown caller pc
18:52:14  runtime stack:
18:52:14  runtime.throw(0xca876f, 0x11)
18:52:14  	/usr/local/go/src/runtime/panic.go:1117 +0x72
18:52:14  runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000083980, 0x0, 0x0, 0x7fffffff, 0xcc8d60, 0xc000591cf8, 0x0, ...)
18:52:14  	/usr/local/go/src/runtime/traceback.go:261 +0x1a56
18:52:14  runtime.copystack(0xc000083980, 0x10000)
18:52:14  	/usr/local/go/src/runtime/stack.go:908 +0x2f1
18:52:14  runtime.newstack()
18:52:14  	/usr/local/go/src/runtime/stack.go:1078 +0x297
18:52:14  runtime.morestack()
18:52:14  	/usr/local/go/src/runtime/asm_amd64.s:458 +0x8f
18:52:14  
...

-----------

16:24:51  # github.com/aws/aws-sdk-go/aws/request
16:24:51  runtime: unexpected return pc for runtime.goready called from 0x746e6567612f736e
16:24:51  stack: frame={sp:0xc0015a7e98, fp:0xc0015a7ea0} stack=[0xc0015a0000,0xc0015a8000)
16:24:51  000000c0015a7d98:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7da8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7db8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7dc8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7dd8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7de8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7df8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e08:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e18:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e28:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e38:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e48:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e58:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e68:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e78:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e88:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7e98: <746e6567612f736e >6170736b726f772f 
16:24:51  000000c0015a7ea8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7eb8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7ec8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7ed8:  0000000000000000  bfe62e42fefa39ef 
16:24:51  000000c0015a7ee8:  0000000000000000  0000000000000000 
16:24:51  000000c0015a7ef8:  0000000000000000  7377612f6f672d6b 
16:24:51  000000c0015a7f08:  747365757165722f  0000000000000000 
16:24:51  000000c0015a7f18:  0000000000000000  000000c000b77078 
16:24:51  000000c0015a7f28:  000000c0000b90f8  000000c00040d1a0 
16:24:51  000000c0015a7f38:  000000c00009ff50  0000000000000000 
16:24:51  000000c0015a7f48:  00000000004650a0 <runtime.chanrecv.func1+0>  000000c000b77020 
16:24:51  000000c0015a7f58:  000000c000c95f88  000000000040648b <runtime.chanrecv2+43> 
16:24:51  000000c0015a7f68:  000000c000b77020  000000c000c95fb0 
16:24:51  000000c0015a7f78:  000000c000936301  2f6c6165732f6563 
16:24:51  000000c0015a7f88:  656e696c6e69616d  6f6c666b726f772f 
16:24:51  000000c0015a7f98:  7265767265732f77 
16:24:51  fatal error: unknown caller pc
16:24:51  
16:24:51  runtime stack:
16:24:51  runtime.throw(0xca876f, 0x11)
16:24:51  	/usr/local/go/src/runtime/panic.go:1117 +0x72
16:24:51  runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000082600, 0x0, 0x0, 0x7fffffff, 0xcc8d60, 0xc00050fcd0, 0x0, ...)
16:24:51  	/usr/local/go/src/runtime/traceback.go:261 +0x1a56
16:24:51  runtime.copystack(0xc000082600, 0x8000)
16:24:51  	/usr/local/go/src/runtime/stack.go:908 +0x2f1
16:24:51  runtime.shrinkstack(0xc000082600)
16:24:51  	/usr/local/go/src/runtime/stack.go:1180 +0x13d
16:24:51  runtime.newstack()
16:24:51  	/usr/local/go/src/runtime/stack.go:1037 +0x30b
16:24:51  runtime.morestack()
16:24:51  	/usr/local/go/src/runtime/asm_amd64.s:458 +0x8f
...
@agnivade agnivade changed the title go build random failure after upgrade to v1.16 cmd/compile: random failure after upgrade to v1.16 Apr 13, 2021
@agnivade agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 13, 2021
@agnivade
Copy link
Contributor

/cc @josharian @mdempsky @randall77

@randall77
Copy link
Contributor

This looks like memory corruption. This can only happen if the capacity of that slice is trashed somehow.
Not sure how that would happen.

@fanqiehc
Copy link
Author

@randall77
Could that be my infrastructure's problem? My kubernetes cluster is running inside vms created by proxmox on a single host with 56 cores and 256G memory. Is there some tools I can test the environment?

@randall77
Copy link
Contributor

@fanqiehc Not sure. It's possible that some memory corruption introduced by the host is to blame. It could also be a bug in the Go runtime. I don't think there's any easy way to distinguish the two. Certainly if you have some memory stress tests you can run, that might point to an obvious cause.
Can you provide detailed instructions about how to reproduce? If we can run it ourselves, we might be able to reproduce (or will be unable to reproduce) on another machine, which would help distinguish the two cases. The fact that we haven't seen this error anywhere else yet is a small hint that your machine is bad, but certainly not determinative.

@randall77
Copy link
Contributor

If you can catch a failure while setting the environment variable GOTRACEBACK to system, we might be able to see what the bad capacity was. Not sure if that would be helpful, but maybe.

@networkimprov
Copy link

@gopherbot add WaitingForInfo

@gopherbot gopherbot added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 15, 2021
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Jun 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@networkimprov @agnivade @randall77 @gopherbot @fanqiehc