Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: panic in PGO-optimized compiler after CL 539699 #64209

Closed
prattmic opened this issue Nov 16, 2023 · 4 comments
Closed

cmd/compile: panic in PGO-optimized compiler after CL 539699 #64209

prattmic opened this issue Nov 16, 2023 · 4 comments
Assignees
@prattmic
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. release-blocker
Milestone
Go1.22

Comments

@prattmic
Copy link
Member 

#!watchflakes
post <- pkg == "golang.org/x/benchmarks/sweet/cmd/sweet" && test == "TestSweetEndToEnd" && `internal compiler error`

Splitting out of #56958.

When built with the appropriate profile (I have one, but GitHub won't let me attach it), a PGO-optimized compiler panics. e.g.,

$ ../bin/go install -pgo bad.pgo cmd/compile
$ ../bin/go install -pgo bad.pgo cmd/compile
# internal/coverage/rtcov
<autogenerated>:1: internal compiler error: '.eq.internal/coverage/rtcov.CovMetaBlob': panic during prove while compiling .eq.internal/coverage/rtcov.CovMetaBlob:

runtime error: invalid memory address or nil pointer dereference

goroutine 6 [running]:
cmd/compile/internal/ssa.Compile.func1()
        ./cmd/compile/internal/ssa/compile.go:49 +0x6c
panic({0xd73cc0?, 0x1474600?})
        ./runtime/panic.go:770 +0x132
cmd/compile/internal/ssa.(*factsTable).update(0xc0020afb28, 0xc000520220, 0x0, 0xc0004f0c60, 0x8, 0x5)
        ./cmd/compile/internal/ssa/prove.go:276 +0x5cb
cmd/compile/internal/ssa.addRestrictions(0xc000520220, 0xc0020afb28, 0x8, 0x0, 0xc0004f0c60, 0x5)
        ./cmd/compile/internal/ssa/prove.go:1356 +0x6b
cmd/compile/internal/ssa.addBranchRestrictions(...)
        ./cmd/compile/internal/ssa/prove.go:1270
cmd/compile/internal/ssa.prove(0xc002086b60)
        ./cmd/compile/internal/ssa/prove.go:1154 +0x1f08
cmd/compile/internal/ssa.Compile(0xc002086b60)
        ./cmd/compile/internal/ssa/compile.go:97 +0xa44
cmd/compile/internal/ssagen.buildssa(0xc0000226c0, 0x3)
        ./cmd/compile/internal/ssagen/ssa.go:575 +0x2c16
cmd/compile/internal/ssagen.Compile(0xc0000226c0, 0x3)
        ./cmd/compile/internal/ssagen/pgen.go:216 +0x3e
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
        ./cmd/compile/internal/gc/compile.go:182 +0x34
cmd/compile/internal/gc.compileFunctions.func3.1()
        ./cmd/compile/internal/gc/compile.go:164 +0x30
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 5
        ./cmd/compile/internal/gc/compile.go:163 +0x247

<more panics>

Presumably the first build is generating bad code, which is causing crashes in the second build.
@prattmic prattmic self-assigned this Nov 16, 2023
@prattmic prattmic added this to the Go1.22 milestone Nov 16, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 16, 2023
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "golang.org/x/benchmarks/sweet/cmd/sweet" && test == "TestSweetEndToEnd" && `internal compiler error`
2023-11-08 21:52 linux-amd64-longtest benchmarks@444331c9 go@6ef98ac8 x/benchmarks/sweet/cmd/sweet.TestSweetEndToEnd (log) 
--- FAIL: TestSweetEndToEnd (245.74s)
    integration_test.go:36: phase setup @0s (duration: 1.26589676s)
    integration_test.go:36: phase sweet-get @1.26589676s (duration: 41.44978289s)
    integration_test.go:131: 	phase sweet-run-biogo-igor @42.716265893s (duration: 1m45.655662582s)
    integration_test.go:131: 	phase sweet-run-biogo-krishna @42.716041195s (duration: 1m55.783318267s)
    integration_test.go:169: output for go-build:
    integration_test.go:177: 
    integration_test.go:183: command output:
        [sweet] Work directory: /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-1
        [sweet] Benchmarks: go-build (1 runs, 1 pgo runs)
...
        <autogenerated>:1: internal compiler error: '.eq.internal/coverage/rtcov.CovMetaBlob': panic during prove while compiling .eq.internal/coverage/rtcov.CovMetaBlob:

        runtime error: invalid memory address or nil pointer dereference

        goroutine 18 [running]:
        cmd/compile/internal/ssa.Compile.func1()
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/ssa/compile.go:49 +0x6c
        panic({0xd7d8e0?, 0x1490690?})
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/panic.go:763 +0x132
        cmd/compile/internal/ssa.(*factsTable).update(0xc002b73968, 0xc000520220, 0x0, 0xc0004f0c60, 0x8, 0x5)
...
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:163 +0x247
        [sweet] error: failed to execute benchmarks, see log for details
    integration_test.go:184: exit status 1
    integration_test.go:131: 	phase sweet-run-go-build @42.716058324s (duration: 2m10.642964602s)
    integration_test.go:131: 	phase sweet-run-tile38 @42.71610869s (duration: 2m21.518052052s)
    integration_test.go:131: 	phase sweet-run-bleve-index @2m38.499437963s (duration: 39.379610011s)
    integration_test.go:131: 	phase sweet-run-markdown @3m4.234241376s (duration: 14.630075799s)
    integration_test.go:131: 	phase sweet-run-gopher-lua @2m53.359078149s (duration: 31.264638478s)
    integration_test.go:131: 	phase sweet-run-etcd @2m28.372056634s (duration: 1m36.381165807s)
    integration_test.go:36: phase sweet-run @42.71567965s (duration: 3m22.037580941s)
2023-11-08 21:52 linux-amd64-longtest-race benchmarks@444331c9 go@2f5bd4e4 x/benchmarks/sweet/cmd/sweet.TestSweetEndToEnd (log) 
--- FAIL: TestSweetEndToEnd (261.94s)
    integration_test.go:36: phase setup @0s (duration: 13.571490133s)
    integration_test.go:36: phase sweet-get @13.571490133s (duration: 44.147228088s)
    integration_test.go:131: 	phase sweet-run-biogo-igor @57.719547726s (duration: 1m41.230247251s)
    integration_test.go:131: 	phase sweet-run-biogo-krishna @57.71933239s (duration: 1m54.294601841s)
    integration_test.go:169: output for go-build:
    integration_test.go:177: 
    integration_test.go:183: command output:
        [sweet] Work directory: /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-1
        [sweet] Benchmarks: go-build (1 runs, 1 pgo runs)
...
        fatal error: fault
        [signal SIGSEGV: segmentation violation code=0x1 addr=0x180026e8690 pc=0x47b67f]

        goroutine 21 [running]:
        runtime.throw({0xdf0db7?, 0xd8d100?})
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/panic.go:1023 +0x5c fp=0xc0022295a0 sp=0xc002229570 pc=0x445fbc
        runtime.sigpanic()
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/signal_unix.go:880 +0x285 fp=0xc002229600 sp=0xc0022295a0 pc=0x45d945
        aeshashbody()
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/asm_amd64.s:1466 +0x39f fp=0xc002229608 sp=0xc002229600 pc=0x47b67f
...
        cmd/compile/internal/ssagen.Compile(0xc00306aea0, 0x1)
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:216 +0x45 fp=0xc0029f9f70 sp=0xc0029f9dd0 pc=0xab14e5
        cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:182 +0x34 fp=0xc0029f9fb0 sp=0xc0029f9f70 pc=0xcf0d94
        cmd/compile/internal/gc.compileFunctions.func3.1()
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:164 +0x30 fp=0xc0029f9fe0 sp=0xc0029f9fb0 pc=0xcf11b0
        runtime.goexit({})
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc0029f9fe8 sp=0xc0029f9fe0 pc=0x47df41
        created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 44
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:163 +0x247

— watchflakes

@gopherbot
Copy link

Change https://go.dev/cl/543115 mentions this issue: cmd/compile: disable PGO function value devirtualization by default

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "golang.org/x/benchmarks/sweet/cmd/sweet" && test == "TestSweetEndToEnd" && `internal compiler error`
2023-11-08 21:52 linux-amd64-longtest-race benchmarks@444331c9 go@d411b319 x/benchmarks/sweet/cmd/sweet.TestSweetEndToEnd (log) 
--- FAIL: TestSweetEndToEnd (200.11s)
    integration_test.go:36: phase setup @0s (duration: 9.719636039s)
    integration_test.go:36: phase sweet-get @9.719636039s (duration: 37.558436324s)
    integration_test.go:131: 	phase sweet-run-biogo-krishna @47.278894843s (duration: 1m11.087433269s)
    integration_test.go:131: 	phase sweet-run-biogo-igor @47.278482993s (duration: 1m18.730483883s)
    integration_test.go:169: output for go-build:
    integration_test.go:177: 
    integration_test.go:183: command output:
        [sweet] Work directory: /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-1
        [sweet] Benchmarks: go-build (1 runs, 1 pgo runs)
...
        <autogenerated>:1: internal compiler error: '.eq.internal/coverage/rtcov.CovMetaBlob': panic during prove while compiling .eq.internal/coverage/rtcov.CovMetaBlob:

        runtime error: invalid memory address or nil pointer dereference

        goroutine 21 [running]:
        cmd/compile/internal/ssa.Compile.func1()
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/ssa/compile.go:49 +0x6c
        panic({0xd6b9c0?, 0x1472690?})
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/panic.go:770 +0x132
        cmd/compile/internal/ssa.(*factsTable).update(0xc001f75930, 0xc0005a0220, 0x0, 0xc000570c60, 0x8, 0x5)
...
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:163 +0x247
        [sweet] error: failed to execute benchmarks, see log for details
    integration_test.go:184: exit status 1
    integration_test.go:131: 	phase sweet-run-go-build @47.278688513s (duration: 1m29.823751561s)
    integration_test.go:131: 	phase sweet-run-tile38 @47.278773013s (duration: 1m37.188254526s)
    integration_test.go:131: 	phase sweet-run-bleve-index @2m6.009115046s (duration: 27.159983066s)
    integration_test.go:131: 	phase sweet-run-gopher-lua @2m17.102547164s (duration: 20.084350647s)
    integration_test.go:131: 	phase sweet-run-markdown @2m24.467116339s (duration: 13.451645717s)
    integration_test.go:131: 	phase sweet-run-etcd @1m58.366531953s (duration: 1m21.022715924s)
    integration_test.go:36: phase sweet-run @47.278072363s (duration: 2m32.111252454s)

— watchflakes

@gopherbot
Copy link

Change https://go.dev/cl/543195 mentions this issue: cmd/compile: don't devirtualize calls to runtime.memhash_varlen

gopherbot pushed a commit that referenced this issue Nov 16, 2023
@prattmic
cmd/compile: allow disable of PGO function value devirtualization wit…
1ab9df4 
…h flag

Extend the pgodevirtualize debug flag to distinguish interface and
function devirtualization. Setting 1 keeps interface devirtualization
enabled but disables function value devirtualization.

For #64209.

Change-Id: I33aa7eb95ca0bdb215256d8c7cc8f9dac53ae30e
Reviewed-on: https://go-review.googlesource.com/c/go/+/543115
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prattmic prattmic

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. release-blocker
Projects
Test Flakes
Status: Done
Milestone
Go1.22
Development

No branches or pull requests
2 participants
@prattmic @gopherbot