runtime: memory corruption on darwin/amd64 #60449

gopherbot · 2023-05-26T04:15:03Z

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

Issue created automatically to collect these failures.

Example (log):

— watchflakes

The text was updated successfully, but these errors were encountered:

gopherbot · 2023-05-26T04:15:06Z

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "exp" && section == "go test -short ./..." && test == ""

2023-05-22 17:56 darwin-amd64-12_0 exp@2e198f4a go@f0e356a0 (log)

— watchflakes

bcmills · 2023-06-13T21:38:40Z

Runtime heap corruption in cmd/go during go test. Looks like a runtime bug to me?

mknyszek · 2023-06-14T19:27:30Z

@prattmic notes that this is a failure on Darwin and there's a fork/exec goroutine here. Sounds like an existing issue possibly (#59995)?

mknyszek · 2023-06-14T19:27:40Z

We should probably aggregate such issues into a single issue.

bcmills · 2023-06-14T20:25:46Z

This failure mode looks substantially different from the other Darwin cluster described in #54461 (comment) (which includes #59995).

That cluster of failures is characterized by a goroutine stuck copying a pipe connected to a dead child process: either stdout, stderr, or (more rarely) the one created by syscall.forkExecPipe, but there is typically no sign of memory corruption. In contrast, the log attached to this issue shows memory corruption, and has a syscall.Wait4 frame on the stack that suggests that the child process is probably still alive.

So I think this issue is probably independent of #59995.

mknyszek · 2023-06-14T21:23:10Z

Agreed. I changed my mind after posting the last message and looking at the other failures again, but then I had a meeting. :P

mknyszek · 2023-06-16T18:31:47Z

I was going back through issues assigned to me (oops, I have a backlog) and I noticed that #58715 is also memory corruption, but none of those failures include forkExec. Just mentioning it here since it might be related.

bcmills · 2023-06-20T13:48:35Z

#55167 (comment) looks like plausibly the same kind of corruption; updated the watchflakes pattern to catch it more specifically.

gopherbot · 2023-06-20T13:48:36Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && `fatal error: found pointer to free object`

2023-06-16 20:47 darwin-amd64-10_15 go@0183c1aa cmd/go.TestScript (log)

vcs-test.golang.org rerouted to http://127.0.0.1:54179
https://vcs-test.golang.org rerouted to https://127.0.0.1:54180
go test proxy running at GOPROXY=http://127.0.0.1:54181/mod
runtime: marked free object in span 0x1e6f868, elemsize=48 freeindex=106 (bad use of unsafe.Pointer? try -d=checkptr)
0xc000042000 alloc marked  
0xc000042030 alloc marked  
0xc000042060 alloc marked  
0xc000042090 alloc marked  
0xc0000420c0 alloc marked  
0xc0000420f0 alloc marked  
...
fatal error: found pointer to free object

runtime stack:
runtime.throw({0x17b70aa?, 0xc000043d10?})
	/tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x70000d701c50 sp=0x70000d701c20 pc=0x103b47c
runtime.(*mspan).reportZombies(0x1e6f868)
	/tmp/buildlet/go/src/runtime/mgcsweep.go:861 +0x2e5 fp=0x70000d701cd0 sp=0x70000d701c50 pc=0x102a745
runtime.(*sweepLocked).sweep(0x2030002?, 0x0)
	/tmp/buildlet/go/src/runtime/mgcsweep.go:644 +0xa88 fp=0x70000d701dd8 sp=0x70000d701cd0 pc=0x1029f68
runtime.(*mcentral).uncacheSpan(0x70000d701e50?, 0x1031946?)
...
	/tmp/buildlet/go/src/os/file_unix.go:315 +0x75 fp=0xc000856d10 sp=0xc000856cb8 pc=0x10b9815
os.(*File).Close(...)
	/tmp/buildlet/go/src/os/file_posix.go:23
os.WriteFile({0xc000b882a0?, 0x64?}, {0xc00080de81, 0xe, 0xf}, 0x18e2578?)
	/tmp/buildlet/go/src/os/file.go:766 +0x7d fp=0xc000856d58 sp=0xc000856d10 pc=0x10b819d
cmd/go/internal/script.(*State).ExtractFiles(0xc000e7ad00, 0xc000538a80?)
	/tmp/buildlet/go/src/cmd/go/internal/script/state.go:171 +0x1c9 fp=0xc000856e20 sp=0xc000856d58 pc=0x16421c9
cmd/go_test.TestScript.func3(0xc0005d7ba0)
	/tmp/buildlet/go/src/cmd/go/script_test.go:128 +0x2a5 fp=0xc000856f70 sp=0xc000856e20 pc=0x166ff25
testing.tRunner(0xc0005d7ba0, 0xc0004daeb0)

— watchflakes

gopherbot · 2023-07-25T03:01:28Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap`)

2023-07-24 17:08 darwin-amd64-12_0 tools@67ba5997 go@df0a1297 x/tools/go/packages.TestAll (log)

runtime: pointer 0xc01477ee10 to unused region of span span.base()=0xc01477e000 span.limit=0xc01477ffe0 span.state=1
runtime: found in object at *(0xc0394e8000+0x17d8)
object=0xc0394e8000 s.base()=0xc039408000 s.limit=0xc03a370000 s.spanclass=0 s.elemsize=16154624 s.state=mSpanInUse
 *(object+0) = 0x4
 *(object+8) = 0x1350008
 *(object+16) = 0x14d37c0
 *(object+24) = 0x1351c90
 *(object+32) = 0xc037d90e38
 *(object+40) = 0x5
 *(object+48) = 0x13500f8
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x12f6b3f?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x700005b78d90 sp=0x700005b78d60 pc=0x10360dc
runtime.badPointer(0x5f407c8, 0x700005b78e10?, 0xc0394e8000, 0xc00791dfe0?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:321 +0x165 fp=0x700005b78de8 sp=0x700005b78d90 pc=0x1013885
runtime.findObject(0xc02fca9280?, 0x10563de?, 0x106c4a0?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:364 +0xa6 fp=0x700005b78e20 sp=0x700005b78de8 pc=0x1013a06
runtime.scanobject(0xc0394e8000, 0xc000030140)
...
	/tmp/buildlet/go/src/runtime/chan.go:442 +0x12 fp=0xc00010ac98 sp=0xc00010ac70 pc=0x1005f12
testing.(*T).Run(0xc00cc761a0, {0x12e60ab?, 0xc000171e20?}, 0xc00d581278)
	/tmp/buildlet/go/src/testing/testing.go:1649 +0x3c8 fp=0xc00010ad58 sp=0xc00010ac98 pc=0x10eec48
golang.org/x/tools/go/packages_test.testLoadDifferentPatterns(0xc00cc761a0, {0x1350d40, 0x150c880})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/overlay_test.go:997 +0x517 fp=0xc00010aed8 sp=0xc00010ad58 pc=0x1257a97
golang.org/x/tools/go/packages_test.testAllOrModulesParallel.func1(0xc00cc761a0, {0x1350d40, 0x150c880})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages_test.go:81 +0x138 fp=0xc00010af30 sp=0xc00010aed8 pc=0x1258f78
golang.org/x/tools/go/packages/packagestest.TestAll.func1(0xc00cc761a0)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packagestest/export.go:162 +0x51 fp=0xc00010af70 sp=0xc00010af30 pc=0x1247c91
testing.tRunner(0xc00cc761a0, 0xc00e8fd5c0)

— watchflakes

gopherbot · 2023-07-25T15:01:41Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty`)

2023-06-23 21:36 darwin-amd64-10_15 telemetry@5a4cd7a1 go@bca81759 (log)

fatal error: workbuf is not empty

runtime stack:
runtime.throw({0x16519f6?, 0x700003f65e08?})
	/tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x700003f65de0 sp=0x700003f65db0 pc=0x103449d
runtime.(*workbuf).checkempty(0x4582128?)
	/tmp/buildlet/go/src/runtime/mgcwork.go:344 +0x36 fp=0x700003f65e00 sp=0x700003f65de0 pc=0x1025a16
runtime.getempty()
	/tmp/buildlet/go/src/runtime/mgcwork.go:357 +0x1e8 fp=0x700003f65e60 sp=0x700003f65e00 pc=0x1025c28
runtime.handoff(0xc001c81000)
	/tmp/buildlet/go/src/runtime/mgcwork.go:439 +0x25 fp=0x700003f65e90 sp=0x700003f65e60 pc=0x1025e25
runtime.(*gcWork).balance(0xc000035238)
	/tmp/buildlet/go/src/runtime/mgcwork.go:297 +0x45 fp=0x700003f65ea8 sp=0x700003f65e90 pc=0x1025905
runtime.gcDrain(0xc000035238, 0x3)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1085 +0x125 fp=0x700003f65f08 sp=0x700003f65ea8 pc=0x101e7e5
runtime.gcBgMarkWorker.func2()
	/tmp/buildlet/go/src/runtime/mgc.go:1348 +0xad fp=0x700003f65f58 sp=0x700003f65f08 pc=0x101af6d
runtime.systemstack()
	/tmp/buildlet/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x700003f65f60 sp=0x700003f65f58 pc=0x10655c9

— watchflakes

gopherbot · 2023-07-28T06:39:31Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty`)

2023-07-27 16:59 darwin-amd64-12_0 tools@bacac149 go@77863e42 x/tools/gopls/test.TestStringChanges (log)

runtime: marked free object in span 0x6bf8728, elemsize=16 freeindex=10 (bad use of unsafe.Pointer? try -d=checkptr)
0xc00027c000 alloc marked  
0xc00027c010 alloc marked  
0xc00027c020 alloc marked  
0xc00027c030 alloc marked  
0xc00027c040 alloc marked  
0xc00027c050 alloc marked  
0xc00027c060 alloc marked  
0xc00027c070 alloc marked  
0xc00027c080 alloc marked  
...
fatal error: found pointer to free object

runtime stack:
runtime.throw({0x1248123?, 0xc00027d4d0?})
	/tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x700010f8b8d8 sp=0x700010f8b8a8 pc=0x103ac7c
runtime.(*mspan).reportZombies(0x6bf8728)
	/tmp/buildlet/go/src/runtime/mgcsweep.go:861 +0x2e5 fp=0x700010f8b958 sp=0x700010f8b8d8 pc=0x102a025
runtime.(*sweepLocked).sweep(0x700010f8ba90?, 0x0)
	/tmp/buildlet/go/src/runtime/mgcsweep.go:644 +0xa88 fp=0x700010f8ba60 sp=0x700010f8b958 pc=0x1029848
runtime.(*mcentral).uncacheSpan(0xc000042900?, 0x700010f8bab8?)
...
	/tmp/buildlet/gopath/pkg/mod/github.com/google/go-cmp@v0.5.9/cmp/report.go:47 +0x89 fp=0xc0002a3ca8 sp=0xc0002a3c38 pc=0x11b38a9
github.com/google/go-cmp/cmp.Diff({0x121a3c0, 0xc00018ea80}, {0x121a3c0, 0xc00018ee00}, {0xc0001c2fc0?, 0x1252abc?, 0xc000736000?})
	/tmp/buildlet/gopath/pkg/mod/github.com/google/go-cmp@v0.5.9/cmp/compare.go:130 +0x16f fp=0xc0002a3d08 sp=0xc0002a3ca8 pc=0x11aa0ef
golang.org/x/tools/gopls/test_test.tryChange(0x133f, 0x1348, {0x1240e31, 0x2})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:134 +0x366 fp=0xc0002a3e38 sp=0xc0002a3d08 pc=0x11dd026
golang.org/x/tools/gopls/test_test.allDeltas(0xc0000f0340, {0xc0000185a0, 0x12, 0x100000001?}, {0xc0002a3f40, 0x2, 0xfc90b1b56b?})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:94 +0x136 fp=0xc0002a3f00 sp=0xc0002a3e38 pc=0x11dcb16
golang.org/x/tools/gopls/test_test.TestStringChanges(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:67 +0x11c fp=0xc0002a3f70 sp=0xc0002a3f00 pc=0x11dc73c
testing.tRunner(0xc0000f0340, 0x1253838)

— watchflakes

mknyszek · 2023-08-23T19:40:19Z

These represent more general runtime memory corruption, but I suspect this is all part of the same issue.

2023-08-21 23:20 darwin-amd64-11_0 go@e3c80f57 cmd/go.TestScript (log)
2023-08-22 08:38 darwin-amd64-longtest go@05f95115 runtime.TestEINTR (log)

gopherbot · 2023-08-23T19:54:23Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty` || `schedule: holding locks` || `fatal error: bad summary data`)

2023-08-21 23:20 darwin-amd64-11_0 go@e3c80f57 cmd/go.TestScript (log)

vcs-test.golang.org rerouted to http://127.0.0.1:53045
https://vcs-test.golang.org rerouted to https://127.0.0.1:53046
go test proxy running at GOPROXY=http://127.0.0.1:53047/mod
--- FAIL: TestScript (1.07s)
    --- FAIL: TestScript/mod_list_test (1.11s)
        script_test.go:132: 2023-08-21T23:59:53Z
        script_test.go:134: $WORK=/tmp/buildlet/tmp/cmd-go-test-340886821/tmpdir633785442/mod_list_test250661805
        script_test.go:156: 
            > env GO111MODULE=on
            # go list -compiled -test must handle test-only packages
...
            fatal error: bad summary data

            runtime stack:
            runtime.throw({0x880bda5?, 0x7000100efed0?})
            	/tmp/buildlet/go/src/runtime/panic.go:1018 +0x5c fp=0x7000100efe08 sp=0x7000100efdd8 pc=0x81c693c
            runtime.(*pageAlloc).alloc(0x8e82ca8, 0x4)
            	/tmp/buildlet/go/src/runtime/mpagealloc.go:896 +0x26a fp=0x7000100efe50 sp=0x7000100efe08 pc=0x81bbe0a
            runtime.(*mheap).allocSpan(0x8e82ca0, 0x4, 0x0, 0x8a?)
            	/tmp/buildlet/go/src/runtime/mheap.go:1247 +0x192 fp=0x7000100efef0 sp=0x7000100efe50 pc=0x81b8052
            runtime.(*mheap).alloc.func1()
...
            cmd/go/internal/work.ActorFunc.Act(0x8a73460?, 0x8e99720?, {0x8a73460?, 0x8e99720?}, 0x0?)
            	/tmp/buildlet/go/src/cmd/go/internal/work/action.go:78 +0x2d fp=0xc00009fd70 sp=0xc00009fd40 pc=0x870282d
            cmd/go/internal/work.(*Builder).Do.func3({0x8a73460, 0x8e99720}, 0xc000479080)
            	/tmp/buildlet/go/src/cmd/go/internal/work/exec.go:151 +0x80c fp=0xc00009ff20 sp=0xc00009fd70 pc=0x871106c
            cmd/go/internal/work.(*Builder).Do.func4()
            	/tmp/buildlet/go/src/cmd/go/internal/work/exec.go:219 +0xb9 fp=0xc00009ffe0 sp=0xc00009ff20 pc=0x8710699
            runtime.goexit()
            	/tmp/buildlet/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00009ffe8 sp=0xc00009ffe0 pc=0x81fd701
            created by cmd/go/internal/work.(*Builder).Do in goroutine 1
            	/tmp/buildlet/go/src/cmd/go/internal/work/exec.go:205 +0x3fe

2023-08-22 08:38 darwin-amd64-longtest go@05f95115 runtime.TestEINTR (log)

fatal error: schedule: holding locks

runtime stack:
runtime.throw({0x632a2a8?, 0x66cf960?})
	/tmp/buildlet/go/src/runtime/panic.go:1018 +0x5c fp=0x7ff7baf93388 sp=0x7ff7baf93358 pc=0x5faa69c
runtime.schedule()
	/tmp/buildlet/go/src/runtime/proc.go:3577 +0x276 fp=0x7ff7baf933c0 sp=0x7ff7baf93388 pc=0x5fb50d6
runtime.park_m(0xc0004cf520?)
	/tmp/buildlet/go/src/runtime/proc.go:3765 +0x12d fp=0x7ff7baf93408 sp=0x7ff7baf933c0 pc=0x5fb546d
runtime.mcall()
...
	/tmp/buildlet/go/src/os/exec/exec.go:590 +0x2d fp=0xc00018cd70 sp=0xc00018cd50 pc=0x60b0eed
os/exec.(*Cmd).CombinedOutput(0xc000610000)
	/tmp/buildlet/go/src/os/exec/exec.go:977 +0x94 fp=0xc00018cd98 sp=0xc00018cd70 pc=0x60b2b34
runtime_test.runBuiltTestProg(0xc000824d00, {0xc00002c500, 0x34}, {0x631da22, 0x5}, {0x0, 0x0, 0x0?})
	/tmp/buildlet/go/src/runtime/crash_test.go:76 +0x327 fp=0xc00018ce80 sp=0xc00018cd98 pc=0x6265ae7
runtime_test.runTestProg(0xc000824d00, {0x6320719, 0xb}, {0x631da22, 0x5}, {0x0, 0x0, 0x0})
	/tmp/buildlet/go/src/runtime/crash_test.go:59 +0x19e fp=0xc00018cef0 sp=0xc00018ce80 pc=0x626575e
runtime_test.TestEINTR(0xc000824d00)
	/tmp/buildlet/go/src/runtime/crash_cgo_test.go:736 +0x5c fp=0xc00018cf70 sp=0xc00018cef0 pc=0x62647dc
testing.tRunner(0xc000824d00, 0x6482498)

— watchflakes

mknyszek · 2023-08-30T19:10:17Z

#62165 may be related.

mknyszek · 2023-08-30T19:10:40Z

Actually, it almost certainly is the same memory corruption. Folding it in.

gopherbot · 2023-11-15T17:30:47Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))

2023-11-10 20:32 darwin-amd64-13 exp@9a3e6036 go@a0df2388 x/exp/cmd/gorelease.TestRelease.testRelease (log)

runtime: pointer 0xc0024e40c0 to unused region of span span.base()=0xc00372c000 span.limit=0xc00372df80 span.state=1
runtime: found in object at *(0xc002a00000+0x181c0)
object=0xc002a00000 s.base()=0xc002a00000 s.limit=0xc002a6ff90 s.spanclass=0 s.elemsize=458752 s.state=mSpanInUse
 *(object+0) = 0x785243bce938bd
 *(object+8) = 0xc0016e3420
 *(object+16) = 0xc00103d460
 *(object+24) = 0xc0011157e0
 *(object+32) = 0xc0018870e0
 *(object+40) = 0xc000cd8720
 *(object+48) = 0xc00115e5e0
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0xe9ecc10?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:1016 +0x5c fp=0x70000470cd88 sp=0x70000470cd58 pc=0xe76a91c
runtime.badPointer(0x55cd33c8, 0xc0024e40c0, 0xc002a00000, 0x181c0)
	/tmp/buildlet/go/src/runtime/mbitmap.go:286 +0x165 fp=0x70000470cdd8 sp=0x70000470cd88 pc=0xe748305
runtime.findObject(0xc001b1ef60?, 0xc001ed3968?, 0x1?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:329 +0xa6 fp=0x70000470ce10 sp=0x70000470cdd8 pc=0xe748486
runtime.scanobject(0xc000031240?, 0xc000031240)
...
	/tmp/buildlet/gopath/pkg/mod/golang.org/x/tools@v0.15.0/go/packages/packages.go:262 +0x5c fp=0xc0001c4c30 sp=0xc0001c4bb8 pc=0xe9b90dc
golang.org/x/exp/cmd/gorelease.loadPackages({0xeac8d18?, 0xc003542480?}, {0x0?, 0xc002894150?}, {0xc002f4ff20, 0x53}, {0xc0019deb70?, 0xc0023e9996?}, {0xc00272b000, 0x3f, ...}, ...)
	/tmp/buildlet/gopath/src/golang.org/x/exp/cmd/gorelease/gorelease.go:1334 +0x125 fp=0xc0001c4db0 sp=0xc0001c4c30 pc=0xe9caa45
golang.org/x/exp/cmd/gorelease.loadDownloadedModule({_, _}, {_, _}, {_, _}, {_, _})
	/tmp/buildlet/gopath/src/golang.org/x/exp/cmd/gorelease/gorelease.go:535 +0x797 fp=0xc0001c4f08 sp=0xc0001c4db0 pc=0xe9c3a37
golang.org/x/exp/cmd/gorelease.runRelease({0xeac8d18, 0xc003542480}, {0xeac5de8, 0xc003542660}, {0xc00241e140, 0x1b}, {0xc001771960?, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/exp/cmd/gorelease/gorelease.go:246 +0xd1c fp=0xc0001c5e08 sp=0xc0001c4f08 pc=0xe9c117c
golang.org/x/exp/cmd/gorelease.TestRelease.testRelease.func1(0xc00038f860)
	/tmp/buildlet/gopath/src/golang.org/x/exp/cmd/gorelease/gorelease_test.go:402 +0x7de fp=0xc0001c5f70 sp=0xc0001c5e08 pc=0xe9d247e
testing.tRunner(0xc00038f860, 0xc00037c960)

— watchflakes

gopherbot · 2023-11-29T14:33:50Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))

2023-11-29 14:02 darwin-amd64-13 tools@c8019b94 go@d77307f8 (log)

fatal error: workbuf is not empty
fatal error: workbuf is not empty

runtime stack:
runtime.throw({0x1651e3f?, 0x0?})
	/tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x7000065f6d20 sp=0x7000065f6cf0 pc=0x103449d
runtime.(*workbuf).checkempty(0x0?)
	/tmp/buildlet/go/src/runtime/mgcwork.go:344 +0x36 fp=0x7000065f6d40 sp=0x7000065f6d20 pc=0x1025a16
runtime.getempty()
	/tmp/buildlet/go/src/runtime/mgcwork.go:357 +0x1e8 fp=0x7000065f6da0 sp=0x7000065f6d40 pc=0x1025c28
...
runtime.greyobject(0xc000b9eaa0, 0x0?, 0x0?, 0x0?, 0x700006a91e58?, 0x1025dc5?)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1506 +0x190 fp=0x700006a91e18 sp=0x700006a91dc8 pc=0x101f5f0
runtime.scanobject(0xc000040638?, 0xc000040638)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1336 +0x187 fp=0x700006a91ea8 sp=0x700006a91e18 pc=0x101f047
runtime.gcDrain(0xc000040638, 0xb)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1103 +0x1d4 fp=0x700006a91f08 sp=0x700006a91ea8 pc=0x101e894
runtime.gcBgMarkWorker.func2()
	/tmp/buildlet/go/src/runtime/mgc.go:1365 +0x8f fp=0x700006a91f58 sp=0x700006a91f08 pc=0x101af4f
runtime.systemstack()
	/tmp/buildlet/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x700006a91f60 sp=0x700006a91f58 pc=0x10655c9

— watchflakes

gopherbot · 2023-11-29T19:25:33Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2023-11-29 15:35 darwin-amd64-11_0 tools@147b88d0 go@6111acc8 x/tools/go/packages (log)

SIGSEGV: segmentation violation
PC=0x8140827 m=12 sigcode=1 addr=0x0

goroutine 0 gp=0xc00edb4680 m=12 mp=0xc00137d008 [idle]:
runtime.spanOfUnchecked(...)
	/tmp/buildlet/go/src/runtime/mheap.go:709
runtime.scanobject(0xc000031268?, 0xc000031268?)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1407 +0x47 fp=0x700010391ea0 sp=0x700010391e10 pc=0x8140827
runtime.gcDrain(0xc000031268, 0x2)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1242 +0x1f4 fp=0x700010391f08 sp=0x700010391ea0 pc=0x81401f4
...
r11    0x4c
r12    0x700010391e90
r13    0x8190fe1
r14    0xc00edb4680
r15    0x1
rip    0x8140827
rflags 0x10246
cs     0x2b
fs     0x0
gs     0x0

— watchflakes

gopherbot · 2023-12-18T20:30:36Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2023-12-07 19:37 darwin-amd64-11_0 pkgsite@00698da1 go@9b4b3e5a (log)

runtime: pointer 0xc000ced4b9 to unused region of span span.base()=0xc000cea000 span.limit=0xc000cf9f00 span.state=1
runtime: found in object at *(0xc001265d60+0x0)
object=0xc001265d60 s.base()=0xc001264000 s.limit=0xc001265fe0 s.spanclass=14 s.elemsize=80 s.state=mSpanInUse
 *(object+0) = 0xc000ced4b9 <==
 *(object+8) = 0x6
 *(object+16) = 0xc0004047e8
 *(object+24) = 0x8
 *(object+32) = 0xc00133e0a0
 *(object+40) = 0x1d
 *(object+48) = 0xc0000280af
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0xf8793f0?, 0x6?})
	runtime/panic.go:1023 +0x5c fp=0x700008ed5d88 sp=0x700008ed5d58 pc=0xf2f73dc
runtime.badPointer(0x16cb2980, 0xc000ced4b9, 0xc001265d60, 0x0)
	runtime/mbitmap.go:286 +0x165 fp=0x700008ed5dd8 sp=0x700008ed5d88 pc=0xf2d4e05
runtime.findObject(0xc00002a080?, 0xc001265d60?, 0xf2e8545?)
	runtime/mbitmap.go:329 +0xa6 fp=0x700008ed5e10 sp=0x700008ed5dd8 pc=0xf2d4f86
runtime.scanobject(0xc00003f768?, 0xc00003f768)
...
os/exec.(*Cmd).writerDescriptor.func1()
	os/exec/exec.go:577 +0x34 fp=0xc001a85f60 sp=0xc001a85f00 pc=0xf3e04f4
os/exec.(*Cmd).Start.func2(0xc0005382c0?)
	os/exec/exec.go:724 +0x2c fp=0xc001a85fc8 sp=0xc001a85f60 pc=0xf3e118c
os/exec.(*Cmd).Start.gowrap1()
	os/exec/exec.go:736 +0x24 fp=0xc001a85fe0 sp=0xc001a85fc8 pc=0xf3e1124
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc001a85fe8 sp=0xc001a85fe0 pc=0xf331c21
created by os/exec.(*Cmd).Start in goroutine 7451
	os/exec/exec.go:723 +0x9ab

— watchflakes

gopherbot · 2023-12-19T04:31:12Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2023-12-18 20:56 darwin-amd64-13 tools@ee47fd47 go@c564d4ae (log)

— watchflakes

gopherbot · 2024-01-09T23:31:41Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2024-01-08 18:46 darwin-amd64-12_0 build@21e06540 go@59ecebf4 (log)

— watchflakes

bcmills · 2024-01-10T17:21:04Z

I wonder if this might be due to an uninitialized pointer field in a syscall result. That could explain the bad pointers, at least, but I guess it doesn't explain the other failure modes.

bcmills · 2024-01-11T16:13:28Z

darwin/amd64 is supposed to be a first class port.

@golang/runtime, should this be marked as a release-blocker for Go 1.22?

bcmills · 2024-01-11T16:35:52Z

Possibly related?

gopherbot · 2024-01-11T17:40:09Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2023-12-11 23:10 darwin-amd64-nocgo tools@f40889dc go@f4ff0e25 x/tools/go/analysis/passes/deepequalerrors (log)

SIGSEGV: segmentation violation
PC=0xe7c69a5 m=7 sigcode=1 addr=0x0

goroutine 0 gp=0xc0002081c0 m=7 mp=0xc000204008 [idle]:
runtime.(*mheap).allocNeedsZero(0xe7a96f6?, 0x800?, 0xc004000000?)
	/tmp/buildlet/go/src/runtime/mheap.go:1024 +0x65 fp=0x70000176ee38 sp=0x70000176ee18 pc=0xe7c69a5
runtime.(*mheap).initSpan(0xeca5740, 0x1586cdc8, 0x0, 0xc, 0xc004010000, 0x1)
	/tmp/buildlet/go/src/runtime/mheap.go:1381 +0x51 fp=0x70000176ee70 sp=0x70000176ee38 pc=0xe7c71b1
runtime.(*mheap).allocSpan(0xeca5740, 0x1, 0x0, 0xc)
	/tmp/buildlet/go/src/runtime/mheap.go:1340 +0x43d fp=0x70000176ef10 sp=0x70000176ee70 pc=0xe7c6fbd
...
r11    0x0
r12    0x80c00407ffff
r13    0xec7bbe0
r14    0xc0002081c0
r15    0x3
rip    0xe7c69a5
rflags 0x10246
cs     0x2b
fs     0x0
gs     0x0

— watchflakes

mknyszek · 2024-01-11T20:49:25Z

darwin/amd64 is supposed to be a first class port.

@golang/runtime, should this be marked as a release-blocker for Go 1.22?

I may be misunderstanding the policy, but I think technically it's not a release-blocker since we released Go 1.21 with the same issue, so it's not new to this release. (The fact that we released Go 1.21 without figuring out what it is is on it's own something that we should maybe revisit, but I think that's where we are today.) I don't want to be the sole judge on this so I'll add it to the next weekly meeting and/or revisit this in triage.

@cherrymui spent a good bit of time investigating this and had a lot of difficulty reproducing. (Mentioning that here because I don't think that was recorded in this issue.) The crash rate is also fairly low, so it's difficult to continue to prioritize this against other things that need fixing, especially since the amount of time this bug could take to resolve is unknown and likely high. Again, not trying to make a decision, just trying to rationalize how we got here.

The other aspect to this is that it doesn't seem to have impacted downstream users much, if at all. It's possible that users are just ignoring it, but IIRC other Darwin issues that appeared on the builders have been reported in the past, such as the CPU profile related one.

This makes me wonder if maybe it's our postsubmit Mac machines specifically that are the problem. It also occurs to me that I haven't seen the same kinds of failures on the LUCI infrastructure. Those builders have been up long enough that I would've expected to see at least one such failure by now. They have a few failures from other unique sources, but I haven't seen anything like this yet.

I will say that it's possible the failure got missed on the LUCI builders because watchflakes doesn't run against the LUCI infrastructure yet. At the very least I can do an ad-hoc-but-systematic analysis of all our LUCI darwin builds and see if anything crops up. If not, then that's an interesting data point.

bcmills · 2024-01-17T17:52:09Z

Previously, the darwin failures in #49692 were resolved by some combination of OS updates and perhaps better ventilation of the builder machines.

Obviously we don't have much control over the ventilation of Amazon's EC2 Mac Minis, though. It's unfortunate that so few Macs support ECC memory. 😒

gopherbot · 2024-01-19T17:51:19Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2024-01-18 23:21 darwin-amd64-13 build@911ff433 go@2540b143 (log)

— watchflakes

cherrymui · 2024-01-19T20:45:06Z

On the main repo, main branch, watchflakes doesn't find any failure matching this for most recent builds in 45 days. I could run it for longer time period and see.

Update: also no match on LUCI builders in a longer period of time.

gopherbot · 2024-01-24T23:13:26Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2024-01-22 20:52 darwin-amd64-11_0 tools@e2ca5942 go@a95136a8 x/tools/go/analysis/passes/stdmethods.TestAnalyzeEncodingXML (log)

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1026791]

runtime stack:
runtime.throw({0x12e4d8e?, 0x14b5700?})
	/tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x7ffeefbff6c8 sp=0x7ffeefbff698 pc=0x103535d
runtime.sigpanic()
	/tmp/buildlet/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0x7ffeefbff728 sp=0x7ffeefbff6c8 pc=0x104c409
runtime.(*mheap).allocNeedsZero(0xc00c100000?, 0x7c?, 0x180?)
	/tmp/buildlet/go/src/runtime/mheap.go:1021 +0x71 fp=0x7ffeefbff748 sp=0x7ffeefbff728 pc=0x1026791
...
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:758 +0xd77 fp=0xc00650dc48 sp=0xc00650d9d8 pc=0x1258317
golang.org/x/tools/go/packages.Load(0xc0003c12c0?, {0xc007dd4a70, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:249 +0x2c7 fp=0xc00650dcc0 sp=0xc00650dc48 pc=0x12566e7
golang.org/x/tools/go/analysis/analysistest.loadPackages(0x14b03e0, {0x0, 0x0}, {0xc007dd4a70, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:375 +0x29a fp=0xc00650de50 sp=0xc00650dcc0 pc=0x126361a
golang.org/x/tools/go/analysis/analysistest.Run({0x13375a0, 0xc007abb860}, {0x0, 0x0}, 0x14b03e0, {0xc007dd4a70, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:322 +0xbf fp=0xc00650df18 sp=0xc00650de50 pc=0x1262fbf
golang.org/x/tools/go/analysis/passes/stdmethods_test.TestAnalyzeEncodingXML(0x100523d?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/passes/stdmethods/stdmethods_test.go:20 +0x5f fp=0xc00650df70 sp=0xc00650df18 pc=0x126673f
testing.tRunner(0xc007abb860, 0x12ec930)

— watchflakes

gopherbot · 2024-02-07T04:22:21Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2024-02-06 21:16 darwin-amd64-12_0 tools@0d875898 go@f2920803 x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckStd (log)

serve.go:441: debug server listening at http://localhost:49266
serve.go:441: debug server listening at http://localhost:49267
#### Start Gopls Test Logs for "TestRunGovulncheckStd/default"
[Trace - 20:03:18.284 PM] Sending request 'initialize - (1)'.
Params: {"processId":0,"rootUri":"","capabilities":{"workspace":{"workspaceEdit":{"resourceOperations":["rename"]},"didChangeConfiguration":{},"didChangeWatchedFiles":{"dynamicRegistration":true},"configuration":true},"textDocument":{"completion":{"completionItem":{"snippetSupport":true,"tagSupport":{"valueSet":[1]}}},"documentSymbol":{"hierarchicalDocumentSymbolSupport":true},"codeAction":{"codeActionLiteralSupport":{"codeActionKind":{"valueSet":null}}},"publishDiagnostics":{},"semanticTokens":{"requests":{"full":true},"tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","macro","keyword","modifier","comment","string","number","regexp","operator"],"tokenModifiers":["declaration","definition","readonly","static","deprecated","abstract","async","modification","documentation","defaultLibrary"],"formats":null}},"window":{"workDoneProgress":true}},"initializationOptions":{"codelenses":{"run_govulncheck":true},"completionBudget":"0s","diagnosticsDelay":"10ms","env":{"GO111MODULE":"","GOMODCACHE":"","GOPACKAGESDRIVER":"off","GOPATH":"/tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/gopath","GOPROXY":"file:///tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/proxy","GOSUMDB":"off","GOVULNDB":"file:///tmp/buildlet/tmp/vulndb-test2317938991/ID","_GOPLS_TEST_BINARY_RUN_AS_GOPLS":"true","_GOPLS_TEST_VULNCHECK_GOVERSION":"go1.18"},"verboseWorkDoneProgress":true},"trace":"messages","workspaceFolders":[{"uri":"file:///tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/work","name":"work"}]}


[Trace - 20:03:18.284 PM] Received response 'initialize - (1)' in 0ms.
Result: {"capabilities":{"textDocumentSync":{"openClose":true,"change":2,"save":{}},"completionProvider":{"triggerCharacters":["."]},"hoverProvider":true,"signatureHelpProvider":{"triggerCharacters":["(",","]},"definitionProvider":true,"typeDefinitionProvider":true,"implementationProvider":true,"referencesProvider":true,"documentHighlightProvider":true,"documentSymbolProvider":true,"codeActionProvider":true,"codeLensProvider":{},"documentLinkProvider":{},"workspaceSymbolProvider":true,"documentFormattingProvider":true,"renameProvider":true,"foldingRangeProvider":true,"selectionRangeProvider":true,"executeCommandProvider":{"commands":["gopls.add_dependency","gopls.add_import","gopls.add_telemetry_counters","gopls.apply_fix","gopls.change_signature","gopls.check_upgrades","gopls.diagnose_files","gopls.edit_go_directive","gopls.fetch_vulncheck_result","gopls.gc_details","gopls.generate","gopls.go_get_package","gopls.list_imports","gopls.list_known_packages","gopls.maybe_prompt_for_telemetry","gopls.mem_stats","gopls.regenerate_cgo","gopls.remove_dependency","gopls.reset_go_mod_diagnostics","gopls.run_go_work_command","gopls.run_govulncheck","gopls.run_tests","gopls.start_debugging","gopls.start_profile","gopls.stop_profile","gopls.test","gopls.tidy","gopls.toggle_gc_details","gopls.update_go_sum","gopls.upgrade_dependency","gopls.vendor","gopls.views","gopls.workspace_stats"]},"callHierarchyProvider":true,"semanticTokensProvider":{"legend":{"tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","macro","keyword","modifier","comment","string","number","regexp","operator"],"tokenModifiers":["declaration","definition","readonly","static","deprecated","abstract","async","modification","documentation","defaultLibrary"]},"range":true,"full":true},"inlayHintProvider":{},"workspace":{"workspaceFolders":{"supported":true,"changeNotifications":"workspace/didChangeWorkspaceFolders"}}},"serverInfo":{"name":"gopls","version":"{\"GoVersion\":\"go1.21.7\",\"Path\":\"\",\"Main\":{\"Path\":\"\",\"Version\":\"\",\"Sum\":\"\",\"Replace\":null},\"Deps\":null,\"Settings\":null,\"Version\":\"(unknown)\"}"}}

...
--- FAIL: TestRunGovulncheckStd (3.56s)
    --- FAIL: TestRunGovulncheckStd/default (3.53s)
        vuln_test.go:212: waiting on:
            Unmeetable: once "completed work for token 680114519661641662" is met, must have:
            received window/showMessage containing "Found GOSTDLIB"
            no diagnostics for file "go.mod"

            err:condition has final verdict Unmeetable

            state:
...
            #### diagnostics:
            	go.mod (version 1):

            #### outstanding work:
            #### completed work:
            	govulncheck: 1
            	diagnosing initial workspace load: 1
            	Setting up workspace: 1
            	Checking telemetry prompt: 1
            	diagnosing opened files: 1

— watchflakes

gopherbot · 2024-02-13T22:33:24Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2024-02-13 14:38 darwin-amd64-11_0 tools@1b39a8b6 go@1b541502 x/tools/go/packages (log)

runtime: pointer 0xc004212000 to unallocated span span.base()=0xc004212000 span.limit=0xc004213fe0 span.state=0
runtime: found in object at *(0xc015497560+0x58)
object=0xc015497560 s.base()=0xc015496000 s.limit=0xc015497fe0 s.spanclass=16 s.elemsize=96 s.state=mSpanInUse
 *(object+0) = 0xc01540d500
 *(object+8) = 0x0
 *(object+16) = 0x0
 *(object+24) = 0x0
 *(object+32) = 0x0
 *(object+40) = 0x0
 *(object+48) = 0x0
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x177fe8d?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:1021 +0x5c fp=0x70000644cd88 sp=0x70000644cd58 pc=0x152179c
runtime.badPointer(0x8f10308, 0xc004212000, 0xc015497560, 0x58)
	/tmp/buildlet/go/src/runtime/mbitmap.go:286 +0x165 fp=0x70000644cdd8 sp=0x70000644cd88 pc=0x14fdde5
runtime.findObject(0xc02575eb40?, 0xc015497560?, 0x1511665?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:329 +0xa6 fp=0x70000644ce10 sp=0x70000644cdd8 pc=0x14fdf66
runtime.scanobject(0xc000031268?, 0xc000031268)
...
golang.org/x/tools/go/packages.(*loader).loadRecursive(0xc022737ce0?, 0xc000a217d0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:822 +0xf fp=0xc000a217a0 sp=0xc000a21768 pc=0x17330cf
golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.1(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:829 +0x26 fp=0xc000a217c8 sp=0xc000a217a0 pc=0x1733386
golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.gowrap1()
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:831 +0x24 fp=0xc000a217e0 sp=0xc000a217c8 pc=0x1733324
runtime.goexit({})
	/tmp/buildlet/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000a217e8 sp=0xc000a217e0 pc=0x155b1c1
created by golang.org/x/tools/go/packages.(*loader).loadRecursive.func1 in goroutine 7427
	/tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:828 +0x94

— watchflakes

gopherbot · 2024-03-27T21:22:50Z

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
	log ~ `fatal error: found pointer to free object` ||
	log ~ `fatal error: found bad pointer in Go heap` ||
	log ~ `fatal error: workbuf is not empty` ||
	log ~ `fatal error: workbuf is empty` ||
	log ~ `schedule: holding locks` ||
	log ~ `fatal error: bad summary data` ||
	log ~ `fatal: morestack on gsignal` ||
	log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
	(log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
	(log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

2024-03-27 17:55 darwin-amd64-nocgo go@979b34b3 cmd/compile/internal/ssa [build] (log)

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0xc022ed3]

goroutine 34 gp=0xc000104c40 m=0 mp=0xc5076a0 [running]:
panic({0xc2ff3e0?, 0xc4f4360?})
	runtime/panic.go:778 +0x168 fp=0xc000029570 sp=0xc0000294c0 pc=0xbfcbe08
runtime.panicmem(...)
	runtime/panic.go:261
runtime.sigpanic()
	runtime/signal_unix.go:881 +0x378 fp=0xc0000295d0 sp=0xc000029570 pc=0xbfe4ad8
...
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func2(0xc4fef80)
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:333 +0x19e fp=0xc002a85fa0 sp=0xc002a85ea0 pc=0xc2065de
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3.1(0x5454545454545454?)
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:404 +0x23 fp=0xc002a85fc8 sp=0xc002a85fa0 pc=0xc206403
cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3.gowrap1()
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:406 +0x24 fp=0xc002a85fe0 sp=0xc002a85fc8 pc=0xc2063a4
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc002a85fe8 sp=0xc002a85fe0 pc=0xc003dc1
created by cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3 in goroutine 1
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:403 +0x47

— watchflakes

gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 26, 2023

bcmills added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jun 13, 2023

mknyszek self-assigned this Jun 14, 2023

mknyszek added this to the Go1.21 milestone Jun 14, 2023

bcmills changed the title ~~go test -short ./...: unrecognized failures~~ runtime: found pointer to free object on darwin/amd64 Jun 20, 2023

bcmills mentioned this issue Jun 20, 2023

runtime: corruption in tests #55167

Open

bcmills changed the title ~~runtime: found pointer to free object on darwin/amd64~~ runtime: pointer corruption on darwin/amd64 Jun 20, 2023

bcmills added the OS-Darwin label Jun 20, 2023

bcmills mentioned this issue Jun 29, 2023

net: slice bounds out of range #61060

Closed

bcmills mentioned this issue Jul 26, 2023

cmd/compile: panic in cmd/compile/internal/types.(*Type).Size via cmd/compile/internal/ssa.(*slotCanonicalizer).lookup #61570

Open

gopherbot modified the milestones: Go1.21, Go1.22 Aug 8, 2023

mknyszek changed the title ~~runtime: pointer corruption on darwin/amd64~~ runtime: memory corruption on darwin/amd64 Aug 23, 2023

mknyszek mentioned this issue Aug 30, 2023

runtime: SIGSEGV in sweepone on darwin/amd64 #62165

Closed

bcmills mentioned this issue Nov 29, 2023

x/tools/go/packages: unrecognized failures #64450

Closed

bcmills mentioned this issue Jan 11, 2024

runtime: SIGSEGV in allocNeedsZero on darwin #58715

Closed

mknyszek removed their assignment Jan 11, 2024

mknyszek assigned mknyszek and cherrymui Jan 17, 2024

gopherbot modified the milestones: Go1.22, Go1.23 Feb 6, 2024

This was referenced Feb 16, 2024

x/tools/go/analysis/unitchecker: test fail with "illegal byte sequence" (EILSEQ) #65550

Open

build: make.bash fail on LUCI darwin builders due to unexpected stale targets #65484

Open

This was referenced Feb 29, 2024

all: failures on darwin/amd64 with device error #66016

Open

net/http: TestIdentityTransferEncoding failures #66010

Open

mknyszek mentioned this issue Mar 27, 2024

cmd/compile/internal/ssa: unrecognized failures #66565

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: memory corruption on darwin/amd64 #60449

runtime: memory corruption on darwin/amd64 #60449

gopherbot commented May 26, 2023 •

edited by mknyszek

gopherbot commented May 26, 2023

bcmills commented Jun 13, 2023

mknyszek commented Jun 14, 2023

mknyszek commented Jun 14, 2023

bcmills commented Jun 14, 2023 •

edited

mknyszek commented Jun 14, 2023

mknyszek commented Jun 16, 2023

bcmills commented Jun 20, 2023 •

edited

gopherbot commented Jun 20, 2023

gopherbot commented Jul 25, 2023

gopherbot commented Jul 25, 2023

gopherbot commented Jul 28, 2023

mknyszek commented Aug 23, 2023

gopherbot commented Aug 23, 2023

mknyszek commented Aug 30, 2023

mknyszek commented Aug 30, 2023

gopherbot commented Nov 15, 2023

gopherbot commented Nov 29, 2023

gopherbot commented Nov 29, 2023

gopherbot commented Dec 18, 2023

gopherbot commented Dec 19, 2023

gopherbot commented Jan 9, 2024

bcmills commented Jan 10, 2024

bcmills commented Jan 11, 2024

bcmills commented Jan 11, 2024

gopherbot commented Jan 11, 2024

mknyszek commented Jan 11, 2024

bcmills commented Jan 17, 2024

gopherbot commented Jan 19, 2024

cherrymui commented Jan 19, 2024 •

edited

gopherbot commented Jan 24, 2024

gopherbot commented Feb 7, 2024

gopherbot commented Feb 13, 2024

gopherbot commented Mar 27, 2024

runtime: memory corruption on darwin/amd64 #60449

runtime: memory corruption on darwin/amd64 #60449

Comments

gopherbot commented May 26, 2023 • edited by mknyszek

gopherbot commented May 26, 2023

bcmills commented Jun 13, 2023

mknyszek commented Jun 14, 2023

mknyszek commented Jun 14, 2023

bcmills commented Jun 14, 2023 • edited

mknyszek commented Jun 14, 2023

mknyszek commented Jun 16, 2023

bcmills commented Jun 20, 2023 • edited

gopherbot commented Jun 20, 2023

gopherbot commented Jul 25, 2023

gopherbot commented Jul 25, 2023

gopherbot commented Jul 28, 2023

mknyszek commented Aug 23, 2023

gopherbot commented Aug 23, 2023

mknyszek commented Aug 30, 2023

mknyszek commented Aug 30, 2023

gopherbot commented Nov 15, 2023

gopherbot commented Nov 29, 2023

gopherbot commented Nov 29, 2023

gopherbot commented Dec 18, 2023

gopherbot commented Dec 19, 2023

gopherbot commented Jan 9, 2024

bcmills commented Jan 10, 2024

bcmills commented Jan 11, 2024

bcmills commented Jan 11, 2024

gopherbot commented Jan 11, 2024

mknyszek commented Jan 11, 2024

bcmills commented Jan 17, 2024

gopherbot commented Jan 19, 2024

cherrymui commented Jan 19, 2024 • edited

gopherbot commented Jan 24, 2024

gopherbot commented Feb 7, 2024

gopherbot commented Feb 13, 2024

gopherbot commented Mar 27, 2024

gopherbot commented May 26, 2023 •

edited by mknyszek

bcmills commented Jun 14, 2023 •

edited

bcmills commented Jun 20, 2023 •

edited

cherrymui commented Jan 19, 2024 •

edited