-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/compile: cmd/compile/internal/gc.buildssa panic #38463
Comments
Definitely looks like a bug in our compiler. The pointer to the function to be compiled is nil. Is this reproducible? Does it happen every time, or is it intermittent? Did this happen in earlier versions of Go? (1.4.{0,1}, and 1.13.10 would be good ones to try if you haven't yet.) I'll try to reproduce this myself. |
How do you populate |
@randall77 I have only see this a single time in our CI, and rerunning on the exact same job it did not occur. So its intermittent, I am not sure if its occurred more than once since we don't have a way to query through all of our test logs and I have only seen it once. We upgrading our CI to go 1.14.2 yesterday and previously were using 1.13.10, so its possible its related to that change - I'll update this if we see this again. The CI job is calling this make target: https://github.com/istio/istio/blob/e33092b5c59d1c13fb11d86ab6345f45194133d3/Makefile.core.mk#L403 which calls https://github.com/istio/istio/blob/e33092b5c59d1c13fb11d86ab6345f45194133d3/Makefile.core.mk#L385 which calls |
Sorry, I was asking about the source code. Presumably there are some .go files in an istioctl/cmd/istioctl/ directory somewhere. Where can I get that? If this is intermittent, it is going to be hard to debug. There aren't any obvious ways a nil pointer might make its way into a the compile queue. |
@randall77 Maybe this's a sign for #33485 |
Ah I see. That would be https://github.com/istio/istio/tree/master/istioctl/cmd/istioctl. I'll keep an eye out for any re-occurrences as well, totally understand this isn't a lot of info to go on. |
Two more examples from #41077 2020-08-27T18:33:37-ac2a1f8/linux-amd64-sid |
Based on my comments in #33485, I wonder if it's related to the call to Edit: Maybe not. There's already a check for |
@randall77 @mdempsky Checking in on this as a 1.16 release blocker. Has anybody managed to reproduce this yet? |
Nope. If I had, I would post it here. |
This happens again in https://build.golang.org/log/e062e1dff6ab680d17246672205cecd010d98e83
|
Change https://golang.org/cl/254617 mentions this issue: |
Currently, there's awkward reentrancy issue with funccompile: funccompile -> compile -> dtypesym -> geneq/genhash/genwrapper -> funccompile Though it's not a problem at this moment, some attempts by @mdempsky to move order/walk/instrument into buildssa was failed, due to SSA cache corruption. This commit fixes that reentrancy issue, by making generated functions to be pumped through the same compile workqueue that normal functions are compiled. We do this by adding them to xtop, instead of calling funccompile directly in geneq/genhash/genwrapper. In dumpdata, we look for uncompiled functions in xtop instead of compilequeue, then finish compiling them. Updates #38463 Fixes #33485 Change-Id: Ic9f0ce45b56ae2ff3862f17fd979253ddc144bb5 Reviewed-on: https://go-review.googlesource.com/c/go/+/254617 Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com> Reviewed-by: Matthew Dempsky <mdempsky@google.com> TryBot-Result: Go Bot <gobot@golang.org> Trust: Keith Randall <khr@golang.org> Trust: Cuong Manh Le <cuong.manhle.vn@gmail.com>
If this was caused by some general corruption problem in the compiler, it might be related to #41257. |
I've also triggered this with an Go 1.14.4 in CI ( $ go version
go version go1.14.4 linux/amd64
$ go build main.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x74 pc=0xce6fff]
goroutine 7 [running]:
cmd/compile/internal/gc.buildssa(0x0, 0x2, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/ssa.go:297 +0xdf
cmd/compile/internal/gc.compileSSA(0x0, 0x2)
/usr/local/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc007104240, 0xc0011ab390, 0x2)
/usr/local/go/src/cmd/compile/internal/gc/pgen.go:363 +0x49
created by cmd/compile/internal/gc.compileFunctions
/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128 |
@randall77 Should we remove the release-blocker label if this is not reproducible? |
@toothrot No, I think we've seen enough instances that this is definitely something worth figuring out. |
Can we check if this still happens in builds after CL 254617 above? |
@randall77 , would it be worth stress testing the compiler under the race detector to see if this is a race? |
I've already tried running the compiler under the race detector and didn't get anything. I didn't really stress it though, maybe worth banging on it harder. |
This hasn't happened in a while:
But it was rare enough that it's hard to say statistically that it's gone.
There were some non-determinism fixes a month ago that may have fixed this. This is so unreproducible that it may be hopeless, but we could try running the compiler under |
My earlier regexp had missed one that was indented. The chance this is still happening is getting pretty small:
|
Fading away into the distance...
|
Does greplogs/findflakes take into account the dev branches? E.g., dev.regabi and dev.typeparams have a few hundred more commits on top of master, which should provide additional testing samples. |
They do not. Actually, fetchlogs only grabs the master branch, but even if it grabbed everything, findflakes would get completely confused if there were multiple branches in its input. Now you have me thinking about how to do the time series analysis when there are multiple converging and diverging timelines. (These tools could really use a revisit! They're held together with duct tape right now.) |
I'm calling this fixed. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes, this was done on 1.14.2
What operating system and processor architecture are you using (
go env
)?go env
OutputThis was built inside a docker image: gcr.io/istio-testing/build-tools:master-2020-04-10T20-55-56.
What did you do?
Tried to build our application.
What did you expect to see?
Build succeeds
What did you see instead?
Full CI log: https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/22958/release-test_istio/11141
Rerunning the job with the same code the build succeeds: https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/22958/release-test_istio/11153
The referenced gobuild.sh script is https://github.com/istio/istio/blob/master/common/scripts/gobuild.sh.
I think the fact its in the compile code path means its a golang bug and not an issue with our code, but maybe not. Let me know if more info is needed.
The text was updated successfully, but these errors were encountered: