Skip to content

runtime,cmd/compile: unexpected fault address in runtime.mapassign #56337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
gopherbot opened this issue Oct 19, 2022 · 16 comments
Closed

runtime,cmd/compile: unexpected fault address in runtime.mapassign #56337

gopherbot opened this issue Oct 19, 2022 · 16 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@gopherbot
Copy link
Contributor

gopherbot commented Oct 19, 2022 •
edited by bcmills
Loading 

#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`

Issue created automatically to collect these failures.

Example (log):

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x21 pc=0xb91b7a]

goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).OuterSym(...)
	C:/workdir/go/src/cmd/link/internal/loader/loader.go:1695
cmd/link/internal/ld.(*deadcodePass).flood(0xc00006aa00)
	C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:239 +0x265
cmd/link/internal/ld.deadcode(0xc0000c0000)
	C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:336 +0x89
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0xe248e3, ...}, ...})
	C:/workdir/go/src/cmd/link/internal/ld/main.go:269 +0xef2
main.main()
	C:/workdir/go/src/cmd/link/main.go:72 +0xedb

— watchflakes
@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 19, 2022
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "golang.org/x/text/internal/triegen.test" && test == ""
2022-10-14 17:33 windows-amd64-race text@1bdb400f go@bfcc3a75 x/text/internal/triegen.test [build] (log) 
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x21 pc=0xb91b7a]

goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).OuterSym(...)
	C:/workdir/go/src/cmd/link/internal/loader/loader.go:1695
cmd/link/internal/ld.(*deadcodePass).flood(0xc00006aa00)
	C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:239 +0x265
cmd/link/internal/ld.deadcode(0xc0000c0000)
	C:/workdir/go/src/cmd/link/internal/ld/deadcode.go:336 +0x89
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0xe248e3, ...}, ...})
	C:/workdir/go/src/cmd/link/internal/ld/main.go:269 +0xef2
main.main()
	C:/workdir/go/src/cmd/link/main.go:72 +0xedb

— watchflakes

@gopherbot gopherbot added this to the Unreleased milestone Oct 19, 2022
@bcmills bcmills changed the title x/text/internal/triegen.test: unrecognized failures runtime,cmd/compile: unexpected fault address in cmd/compile/internal/ssagen.InitTables Oct 20, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Oct 20, 2022
@bcmills
Copy link
Contributor

bcmills commented Oct 20, 2022

(attn @golang/runtime; CC @golang/windows)

@bcmills bcmills changed the title runtime,cmd/compile: unexpected fault address in cmd/compile/internal/ssagen.InitTables runtime,cmd/compile: unexpected fault address in runtime.mapassign Oct 20, 2022
@alexbrainman
Copy link
Member

@bcmills

There are couple of different stack traces.

I looked at the first stack trace in the log:

go build golang.org/x/text/internal/number: # golang.org/x/text/internal/number [golang.org/x/text/internal/number.test]
unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x4fe86]

goroutine 1 [running]:
runtime.throw({0x9d37da?, 0xc00011c4a8?})
	C:/workdir/go/src/runtime/panic.go:1047 +0x65 fp=0xc00011c470 sp=0xc00011c440 pc=0x79e85
runtime.sigpanic()
	C:/workdir/go/src/runtime/signal_windows.go:270 +0xd0 fp=0xc00011c4b8 sp=0xc00011c470 pc=0x8ed50
runtime.mapassign(0x95cbe0, 0xc000113e00, 0x755a2d?)
	C:/workdir/go/src/runtime/map.go:621 +0x246 fp=0xc00011c540 sp=0xc00011c4b8 pc=0x4fe86
cmd/compile/internal/ssagen.InitTables.func3({0x9dcb21, 0xb}, {0x9dbcfe, 0xb}, {0x9e9cf8, 0x17}, {0x9d2ad8, 0x5}, {0xc000126580, 0xe, ...})
	C:/workdir/go/src/cmd/compile/internal/ssagen/ssa.go:3944 +0x1fa fp=0xc00011c608 sp=0xc00011c540 pc=0x74a25a
cmd/compile/internal/ssagen.InitTables()
	C:/workdir/go/src/cmd/compile/internal/ssagen/ssa.go:4755 +0xb054 fp=0xc00011da98 sp=0xc00011c608 pc=0x72e754
cmd/compile/internal/gc.Main(0xa0cfd8)
	C:/workdir/go/src/cmd/compile/internal/gc/main.go:173 +0xaaa fp=0xc00011df20 sp=0xc00011da98 pc=0x8e672a
main.main()
	C:/workdir/go/src/cmd/compile/main.go:57 +0xdd fp=0xc00011df80 sp=0xc00011df20 pc=0x90e93d
runtime.main()
	C:/workdir/go/src/runtime/proc.go:250 +0x1fe fp=0xc00011dfe0 sp=0xc00011df80 pc=0x7c5fe
runtime.goexit()
	C:/workdir/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0xaca81

The crash happens here

go/src/runtime/map.go

Line 621 in bfcc3a7

if b.tophash[i] != top {

Which looks like uninitialised map used in

go/src/cmd/compile/internal/ssagen/ssa.go

Line 3944 in bfcc3a7

intrinsics[intrinsicKey{a, pkg, fn}] = b

I have no idea how to fix that problem.

Alex

@randall77
Copy link
Contributor

This crash happens early on during compiler startup, before it has even read any of the code to be compiled.
In fact, I think the compiler is completely deterministic up to this point, so it is hard to imagine why it would crash on these runs but not every time.
So some unlikely but possible causes: something wrong with runtime/gc, randomness in map hash triggering some rare map bug, or our favorite boogeyman, bad machine memory.

@randall77
Copy link
Contributor

Both failures are with -race, which may implicate the race detector itself or our instrumentation that calls it (e.g. if it messed up live variable stack maps that in turn caused the GC to overcollect).

@prattmic
Copy link
Member

The crashes are in the compiler and linker. Just double checking, do we build the toolchain with the race detector enabled on race builders, or just the tests?

@randall77
Copy link
Contributor

Good point, I don't think the compiler/linker is race-enabled when on a race builder. Just the resulting tests.

@mknyszek
Copy link
Contributor

In triage the consensus is to wait for more to happen. @prattmic notes that it's suspicious that 3 compiler/linker processes crashed at once. Maybe something wrong with the machine?

@mknyszek mknyszek modified the milestones: Unreleased, Go1.20 Oct 26, 2022
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2022-09-29 20:41 windows-386-2008 sync@8fcdb60f go@b7662047 x/sync/syncmap.TestMapMatchesDeepCopy (log) 
unexpected fault address 0xb4df0b0
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xb4df0b0 pc=0x40eaea]

runtime.throw({0x533c48, 0x5})
	C:/workdir/go/src/runtime/panic.go:1047 +0x4d fp=0xa8439c0 sp=0xa8439ac pc=0x43947d
runtime.sigpanic()
	C:/workdir/go/src/runtime/signal_windows.go:270 +0x105 fp=0xa8439e4 sp=0xa8439c0 pc=0x44d395
runtime.evacuated(...)
	C:/workdir/go/src/runtime/map.go:204
...
	C:/workdir/go/src/runtime/asm_386.s:541 +0x30 fp=0xa843c00 sp=0xa843bdc pc=0x467f40
reflect.Value.call({0x5198a0, 0x540e88, 0x13}, {0x5339d3, 0x4}, {0xa969390, 0x1, 0x1})
	C:/workdir/go/src/reflect/value.go:586 +0x894 fp=0xa843e9c sp=0xa843c00 pc=0x4a79a4
reflect.Value.Call({0x5198a0, 0x540e88, 0x13}, {0xa969390, 0x1, 0x1})
	C:/workdir/go/src/reflect/value.go:370 +0x7e fp=0xa843ed8 sp=0xa843e9c pc=0x4a707e
testing/quick.CheckEqual({0x5198a0, 0x540e8c}, {0x5198a0, 0x540e88}, 0x0)
	C:/workdir/go/src/testing/quick/quick.go:331 +0x314 fp=0xa843f74 sp=0xa843ed8 pc=0x505994
golang.org/x/sync/syncmap_test.TestMapMatchesDeepCopy(0xa8312c0)
	C:/workdir/gopath/src/golang.org/x/sync/syncmap/map_test.go:108 +0x47 fp=0xa843f9c sp=0xa843f74 pc=0x507b47
testing.tRunner(0xa8312c0, 0x540e80)

— watchflakes

@randall77
Copy link
Contributor

That latest failure looks unrelated, maybe?
Nothing -race about it.
There do seem to be 2 simultaneous failures in the log (like was observed above, I think), the linker crashing on a bad map operation and a golang.org/x/sync/syncmap test also crashing similarly.
This failure was also 386, not amd64.

No obvious culprit here, unfortunately.

@cherrymui
Copy link
Member

The failure looks like machine issue and there doesn't seem actionable for us for the moment. Move the backlog and see if we have failures.

@cherrymui cherrymui modified the milestones: Go1.20, Backlog Jan 13, 2023
@cherrymui cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 13, 2023
@gopherbot
Copy link
Contributor Author

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@github-project-automation github-project-automation bot moved this from Todo to Done in Go Compiler / Runtime Feb 13, 2023
@bcmills bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Feb 14, 2023
@cherrymui cherrymui mentioned this issue Feb 17, 2023
Closed
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2023-05-03 20:35 linux-ppc64-sid-power10 tools@ba892bba go@c746059a x/tools/go/analysis/passes/ifaceassert.Test (log) 
unexpected fault address 0x8000314bfe3e4a0c
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0x8000314bfe3e4a0c pc=0x181cc]

runtime.gopark(0x4d3c80?, 0xc0002e9b00?, 0x0?, 0x0?, 0x7fffae5e0108?)
	/workdir/go/src/runtime/proc.go:398 +0x114 fp=0xc000093808 sp=0xc0000937d8 pc=0x4d5e4
runtime.goparkunlock(...)
	/workdir/go/src/runtime/proc.go:404
runtime.semacquire1(0xc0002ea458, 0x0?, 0x1, 0x0, 0x0?)
	/workdir/go/src/runtime/sema.go:160 +0x244 fp=0xc000093870 sp=0xc000093808 pc=0x60c04
sync.runtime_Semacquire(0xc000083520?)
	/workdir/go/src/runtime/sema.go:62 +0x3c fp=0xc0000938b8 sp=0xc000093870 pc=0x7f43c
sync.(*WaitGroup).Wait(0xc0002ea450)
	/workdir/go/src/sync/waitgroup.go:116 +0x88 fp=0xc0000938e8 sp=0xc0000938b8 pc=0x8ae18
golang.org/x/tools/go/packages.(*loader).refine(0xc0001181c0, 0xc0000b59f0)
	/workdir/gopath/src/golang.org/x/tools/go/packages/packages.go:778 +0xd98 fp=0xc000093bf0 sp=0xc0000938e8 pc=0x26bc18
golang.org/x/tools/go/packages.Load(0xc0000aec60?, {0xc0000d6440, 0x2, 0x2})
	/workdir/gopath/src/golang.org/x/tools/go/packages/packages.go:266 +0xbc fp=0xc000093c38 sp=0xc000093bf0 pc=0x269f5c
golang.org/x/tools/go/analysis/analysistest.loadPackages(0x4ca4a0, {0xc0000e8280, 0x4e}, {0xc0000d6440, 0x2, 0x2})
	/workdir/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:332 +0x1f4 fp=0xc000093db0 sp=0xc000093c38 pc=0x276b34
golang.org/x/tools/go/analysis/analysistest.Run({0x355b88, 0xc000083380}, {0xc0000e8280, 0x4e}, 0x4ca4a0, {0xc0000d6440, 0x2, 0x2})
	/workdir/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:286 +0xc8 fp=0xc000093e90 sp=0xc000093db0 pc=0x276608
golang.org/x/tools/go/analysis/passes/ifaceassert_test.Test(0x0?)
	/workdir/gopath/src/golang.org/x/tools/go/analysis/passes/ifaceassert/ifaceassert_test.go:21 +0xe4 fp=0xc000093f30 sp=0xc000093e90 pc=0x279524
testing.tRunner(0xc000083380, 0x30a108)

— watchflakes

@gopherbot gopherbot reopened this May 4, 2023
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2023-05-08 21:24 linux-ppc64-sid-power10 oauth2@839de225 go@4406e8ef runtime [build] (log) 
unexpected fault address 0x17aef00000000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x17aef00000000 pc=0x17aef00000000]

goroutine 3031 [running]:
runtime.throw({0x96d141?, 0xc000c65e40?})
	runtime/panic.go:1077 +0x40 fp=0xc0010b4920 sp=0xc0010b48e0 pc=0x58ff0
runtime.sigpanic()
	runtime/signal_unix.go:866 +0x264 fp=0xc0010b4990 sp=0xc0010b4920 pc=0x73ae4
cmd/internal/src.XPos.IsKnown(...)
...
cmd/compile/internal/ssagen.Compile(0xc001a80b00, 0x81bc50?)
	cmd/compile/internal/ssagen/pgen.go:187 +0x48 fp=0xc0010b5f30 sp=0xc0010b5d60 pc=0x664408
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
	cmd/compile/internal/gc/compile.go:184 +0x4c fp=0xc0010b5f80 sp=0xc0010b5f30 pc=0x86a78c
cmd/compile/internal/gc.compileFunctions.func3.1()
	cmd/compile/internal/gc/compile.go:166 +0x48 fp=0xc0010b5fc0 sp=0xc0010b5f80 pc=0x86ab78
runtime.goexit()
	runtime/asm_ppc64x.s:912 +0x4 fp=0xc0010b5fc0 sp=0xc0010b5fc0 pc=0x956e4
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 177
	cmd/compile/internal/gc/compile.go:165 +0x20c
2023-05-31 19:15 linux-ppc64le-buildlet go@3cea9e68 runtime [build] (log) 
unexpected fault address 0x216e000004c18
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x3 addr=0x216e000004c18 pc=0x216e000004c18]

goroutine 2836 [running]:
runtime.throw({0x95d17f?, 0x68b614?})
	runtime/panic.go:1077 +0x40 fp=0xc000a52000 sp=0xc000a51fc0 pc=0x59260
runtime.sigpanic()
	runtime/signal_unix.go:866 +0x260 fp=0xc000a52070 sp=0xc000a52000 pc=0x73af0
cmd/compile/internal/ssagen.(*state).stmtList(...)
...
runtime.goexit()
	runtime/asm_ppc64x.s:912 +0x4 fp=0xc000a53fc0 sp=0xc000a53fc0 pc=0x95034
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 143
	cmd/compile/internal/gc/compile.go:165 +0x20c

goroutine 2822 [running]:
	goroutine running on other thread; stack unavailable
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 143
	cmd/compile/internal/gc/compile.go:165 +0x20c
go tool dist: FAILED: /workdir/go/pkg/tool/linux_ppc64le/go_bootstrap install cmd: exit status 1

— watchflakes

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Sep 18, 2023
@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 18, 2023
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
post <- `unexpected fault address 0x[0-9a-f]+` && `^\s*runtime\.mapassign`
2023-11-08 21:52 linux-amd64-longtest-race benchmarks@444331c9 go@3073f3f9 x/benchmarks/sweet/cmd/sweet.TestSweetEndToEnd (log) 
--- FAIL: TestSweetEndToEnd (266.19s)
    integration_test.go:36: phase setup @0s (duration: 13.80441237s)
    integration_test.go:36: phase sweet-get @13.80441237s (duration: 39.512148632s)
    integration_test.go:131: 	phase sweet-run-biogo-igor @53.317580879s (duration: 1m43.128894734s)
    integration_test.go:131: 	phase sweet-run-biogo-krishna @53.317240974s (duration: 1m55.915230187s)
    integration_test.go:169: output for go-build:
    integration_test.go:177: 
    integration_test.go:183: command output:
        [sweet] Work directory: /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-1
        [sweet] Benchmarks: go-build (1 runs, 1 pgo runs)
...
        <autogenerated>:1: internal compiler error: '.eq.internal/coverage/rtcov.CovMetaBlob': panic during prove while compiling .eq.internal/coverage/rtcov.CovMetaBlob:

        runtime error: invalid memory address or nil pointer dereference

        goroutine 21 [running]:
        cmd/compile/internal/ssa.Compile.func1()
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/ssa/compile.go:49 +0x6c
        panic({0xd7ac00?, 0x14896f0?})
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/panic.go:763 +0x132
        cmd/compile/internal/ssa.(*factsTable).restore(...)
...
        cmd/compile/internal/ssagen.Compile(0xc0031330e0, 0x1)
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/ssagen/pgen.go:216 +0x3e fp=0xc00256df70 sp=0xc00256dea0 pc=0xab19fe
        cmd/compile/internal/gc.compileFunctions.func5.1(0xc0003f0280?)
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:182 +0x34 fp=0xc00256dfb0 sp=0xc00256df70 pc=0xcf4054
        cmd/compile/internal/gc.compileFunctions.func3.1()
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:164 +0x30 fp=0xc00256dfe0 sp=0xc00256dfb0 pc=0xcf4470
        runtime.goexit({})
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/runtime/asm_amd64.s:1694 +0x1 fp=0xc00256dfe8 sp=0xc00256dfe0 pc=0x47dd41
        created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 33
        	../../tmp/tmp-1/go-build/go.pgo/bin/goroot/src/cmd/compile/internal/gc/compile.go:163 +0x247

— watchflakes

@gopherbot gopherbot reopened this Nov 15, 2023
@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 15, 2023
@mknyszek
Copy link
Contributor

The latest failure is almost certainly just #56958.

@golang golang locked and limited conversation to collaborators Nov 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Test Flakes
Archived in project
Milestone
Backlog
Development

No branches or pull requests
8 participants
@mknyszek @prattmic @bcmills @randall77 @gopherbot @alexbrainman @seankhliao @cherrymui