Definitely looks like a bug in our compiler. The pointer to the function to be compiled is nil.

Is this reproducible? Does it happen every time, or is it intermittent?

Did this happen in earlier versions of Go? (1.4.{0,1}, and 1.13.10 would be good ones to try if you haven't yet.)

I'll try to reproduce this myself.

How do you populate ./istioctl/cmd/istioctl? I don't see anything in the log that makes that directory tree.

@randall77 I have only see this a single time in our CI, and rerunning on the exact same job it did not occur. So its intermittent, I am not sure if its occurred more than once since we don't have a way to query through all of our test logs and I have only seen it once.

We upgrading our CI to go 1.14.2 yesterday and previously were using 1.13.10, so its possible its related to that change - I'll update this if we see this again.

The CI job is calling this make target: https://github.com/istio/istio/blob/e33092b5c59d1c13fb11d86ab6345f45194133d3/Makefile.core.mk#L403 which calls https://github.com/istio/istio/blob/e33092b5c59d1c13fb11d86ab6345f45194133d3/Makefile.core.mk#L385 which calls gobuild.sh ./istioctl/cmd/istioctl which calls go build ./istioctl/cmd/istioctl with some flags

Sorry, I was asking about the source code. Presumably there are some .go files in an istioctl/cmd/istioctl/ directory somewhere. Where can I get that?

If this is intermittent, it is going to be hard to debug. There aren't any obvious ways a nil pointer might make its way into a the compile queue.

@randall77 Maybe this's a sign for #33485

Ah I see. That would be https://github.com/istio/istio/tree/master/istioctl/cmd/istioctl. I'll keep an eye out for any re-occurrences as well, totally understand this isn't a lot of info to go on.

Two more examples from #41077

2020-08-27T18:33:37-ac2a1f8/linux-amd64-sid
2020-08-14T17:45:39-a20cb4c/linux-amd64-nocgo

Based on my comments in #33485, I wonder if it's related to the call to dtypesym in gc/ssa.go? In particular, could calling dtypesym be causing functions to get pushed onto compilequeue, which would be bad since that slice is being concurrently accessed by the goroutine responsible for distributing functions to the worker goroutines?

Edit: Maybe not. There's already a check for Siggen just before the call to dtypesym.

@randall77 @mdempsky Checking in on this as a 1.16 release blocker. Has anybody managed to reproduce this yet?

Nope. If I had, I would post it here.
Any ideas are welcome. This is something of a stumper.

This happens again in https://build.golang.org/log/e062e1dff6ab680d17246672205cecd010d98e83

##### ../misc/reboot
Building Go cmd/dist using /workdir/go. (devel 2c95e3a6a8377ca9c72608c25b4cf2506baf782f linux/amd64)
Building Go toolchain1 using /workdir/go.
# bootstrap/cmd/link/internal/riscv64
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0xabac1c]

goroutine 12 [running]:
cmd/compile/internal/gc.buildssa(0x0, 0x1, 0x0)
	/workdir/go/src/cmd/compile/internal/gc/ssa.go:315 +0xfc
cmd/compile/internal/gc.compileSSA(0x0, 0x1)
	/workdir/go/src/cmd/compile/internal/gc/pgen.go:317 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc000767380, 0xc0005054a0, 0x1)
	/workdir/go/src/cmd/compile/internal/gc/pgen.go:382 +0x4d
created by cmd/compile/internal/gc.compileFunctions
	/workdir/go/src/cmd/compile/internal/gc/pgen.go:380 +0x129
go tool dist: FAILED: /workdir/go/bin/go install -gcflags=-l -tags=math_big_pure_go compiler_bootstrap bootstrap/cmd/...: exit status 2
--- FAIL: TestRepeatBootstrap (17.64s)
    reboot_test.go:50: exit status 2
FAIL

Change https://golang.org/cl/254617 mentions this issue: cmd/compile: make funccompile un-reentrant

If this was caused by some general corruption problem in the compiler, it might be related to #41257.

I've also triggered this with an Go 1.14.4 in CI (go build running in Docker). I was not able to reproduce this because rerunning fixed it.

$ go version
go version go1.14.4 linux/amd64

$ go build main.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x74 pc=0xce6fff]

goroutine 7 [running]:
cmd/compile/internal/gc.buildssa(0x0, 0x2, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:297 +0xdf
cmd/compile/internal/gc.compileSSA(0x0, 0x2)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc007104240, 0xc0011ab390, 0x2)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:363 +0x49
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

@randall77 Should we remove the release-blocker label if this is not reproducible?

@toothrot No, I think we've seen enough instances that this is definitely something worth figuring out.

@toothrot No, I think we've seen enough instances that this is definitely something worth figuring out.

Can we check if this still happens in builds after CL 254617 above?

@randall77 , would it be worth stress testing the compiler under the race detector to see if this is a race?

I've already tried running the compiler under the race detector and didn't get anything. I didn't really stress it though, maybe worth banging on it harder.

This hasn't happened in a while:

greplogs -dashboard -E "^goroutine.*running.*\n.*gc\.buildssa" -l -md
2020-09-12T08:31:49-2c95e3a/linux-amd64-ssacheck
2020-08-14T17:45:39-a20cb4c/linux-amd64-nocgo
2020-05-10T14:43:46-57e32c4/openbsd-386-64
2020-03-27T21:13:06-9ceb1e5/freebsd-386-12_0
2020-02-05T22:36:18-9ee5174/freebsd-386-11_2

But it was rare enough that it's hard to say statistically that it's gone.

$ greplogs -dashboard -E "^goroutine.*running.*\n.*gc\.buildssa" -l | findflakes -paths
First observed 9ee5174 05 Feb 22:36 2020 (2659 commits ago)
Last observed  2c95e3a 12 Sep 08:31 2020 (939 commits ago)
11% chance failure is still happening
0.23% failure probability (5 of 1721 commits)
Likely culprits:
   ... (all useless) ...
No known past failures

There were some non-determinism fixes a month ago that may have fixed this.

This is so unreproducible that it may be hopeless, but we could try running the compiler under GODEBUG=gccheckmarks=1 to see if there's a GC issue.

My earlier regexp had missed one that was indented. The chance this is still happening is getting pretty small:

$ greplogs -dashboard -e "^\s*goroutine.*running.*\n.*gc\.buildssa" -l | findflakes -paths
First observed 9ee5174 05 Feb 22:36 2020 (2774 commits ago)
Last observed  2c95e3a 12 Sep 08:31 2020 (1054 commits ago)
4.6% chance failure is still happening
0.29% failure probability (6 of 1721 commits)
Likely culprits:
   ...
No known past failures

Fading away into the distance...

$ greplogs -dashboard -e "^\s*goroutine.*running.*\n.*gc\.buildssa" -l | findflakes -paths
First observed 9ee5174 05 Feb 22:36 2020 (2812 commits ago)
Last observed  2c95e3a 12 Sep 08:31 2020 (1092 commits ago)
4.2% chance failure is still happening
0.29% failure probability (6 of 1721 commits)
Likely culprits:
    ...
No known past failures

Does greplogs/findflakes take into account the dev branches? E.g., dev.regabi and dev.typeparams have a few hundred more commits on top of master, which should provide additional testing samples.

They do not. Actually, fetchlogs only grabs the master branch, but even if it grabbed everything, findflakes would get completely confused if there were multiple branches in its input. Now you have me thinking about how to do the time series analysis when there are multiple converging and diverging timelines. (These tools could really use a revisit! They're held together with duct tape right now.)

$ greplogs -dashboard -e "^\s*goroutine.*running.*\n.*gc\.buildssa" -l | findflakes -paths
First observed 9ee5174 05 Feb 22:36 2020 (2842 commits ago)
Last observed  2c95e3a 12 Sep 08:31 2020 (1122 commits ago)
3.8% chance failure is still happening
0.29% failure probability (6 of 1721 commits)

I'm calling this fixed.