I suspect your guess is right (or certainly close). If the span gets completely freed before the test expects it to, I could easily see it getting this output instead of the expected output.

It's not obvious to me how we get into this situation, though. The test alternates between allocating kept-alive and freed objects, so you'd think the spans wouldn't be released. It could be that a poorly timed preemption moves it to another P or it spills just onto another span, causing a freed object to be allocated alone on some span.

This all suggests this is just a bug in the test (so I agree with okay-after-beta1). It's possible zombie detection could be improved to handle this case, but probably we should just fix the test for now.

@aclements Yes, you are right, that's a cross-span problem.

Look into the error message, the "object=0xc00018c100" which is "zombies" slice, the incorrect element is “ *(object+0) = 0xc00007e000” which corresponding span can no longer be found, and from the two error logs, you can see a certain rule, that is, the error pointer is always at the first.

So it is speculated that there is a cross-span situation in the process of adding objects to the "free" ("zombies") slice during the loop of for i := 0; i < count; i++ { obj := make([]byte, size) ...}.

Imagine the following scenario:

1. The is a span with spanclass=36, and which freeindex point to the last slot. let's call this SpanA
2. During the loop of "for i := 0; i < count; i++ { obj := make([]byte, size) ...}", here i == 1, take up object from the last slot of SpanA (Different P and span with i==0 condition, G's scheduling has occurred)
3. During the loop of "for i := 0; i < count; i++ { obj := make([]byte, size) ...}", when i > 1, take up object from the new span.
4. When it reaches the First GC, If all of the objects in SpanA happen to be released, so the entire SpanA is recycled.
5. At this time, the second GC will trigger the badPointer error.

In order to verify the speculation and increase the probability of recurrence, I made the following adjustments to the code, that is, alloc some objects with the same size before and after that loop, in order to construct a cross-Span scene,
it can reproduce the same phenomenon in one minute on my computer:

And i argee with just fix the test for now. GODEBUG=invalidptr=0 or limit the GOMAXPROCS to 1 both can fix this, i think GODEBUG=invalidptr=0 is better and have tested on my locally for a long time it works well.

diff --git a/src/runtime/testdata/testprog/gc.go b/src/runtime/testdata/testprog/gc.go
index 7d371a6a89..5825fb0cb2 100644
--- a/src/runtime/testdata/testprog/gc.go
+++ b/src/runtime/testdata/testprog/gc.go
@@ -6,6 +6,7 @@ package main

 import (
        "fmt"
+       "math/rand"
        "os"
        "runtime"
        "runtime/debug"
@@ -279,6 +280,15 @@ func GCZombie() {
        keep := make([]*byte, 0, (count+1)/2)
        free := make([]uintptr, 0, (count+1)/2)
        zombies := make([]*byte, 0, len(free))
+
+       var dummyCount = rand.Intn(32*10) + 1
+       dummyFree := make([]uintptr, 0, dummyCount)
+       for i := 0; i < dummyCount / 2; i++ {
+               obj := make([]byte, size)
+               p := &obj[0]
+               dummyFree = append(dummyFree, uintptr(unsafe.Pointer(p)))
+       }
+
        for i := 0; i < count; i++ {
                obj := make([]byte, size)
                p := &obj[0]
@@ -289,6 +299,12 @@ func GCZombie() {
                }
        }

+       for i := dummyCount / 2; i < dummyCount; i++ {
+               obj := make([]byte, size)
+               p := &obj[0]
+               dummyFree = append(dummyFree, uintptr(unsafe.Pointer(p)))
+       }
+
        // Free the unreferenced objects.
        runtime.GC()

cd ./src/runtime/testdata/testprog/
go build, and run the fullowing script:

#/bin/bash
for ((i=1; i<=100000; i++))
do
    ERRSTRS=`./testprog GCZombie 2>&1`
        ERRSTR=`echo $ERRSTRS | grep "found pointer to free object"`
        if [ -z "$ERRSTR" ]
        then
           echo $ERRSTRS
           exit -1
        fi
done

han@DESKTOP-2MU542E:/mnt/d/gobuild/golang/go1.18/src/runtime/testdata/testprog$ ./rundptr1.bash
dummyFree 81 of 162 dummyFree 162 of 162 runtime: pointer 0xc0000a6000 to unused region of span span.base()=0xc0000aa000 span.limit=0xc0000ac000 span.state=1 runtime: found in object at *(0xc000184100+0x0) object=0xc000184100 s.base()=0xc000184000 s.limit=0xc000186000 s.spanclass=36 s.elemsize=256 s.state=mSpanInUse *(object+0) = 0xc0000a6000 <== *(object+8) = 0xc0001860c0 *(object+16) = 0xc000186240 *(object+24) = 0xc0001863c0 *(object+32) = 0xc000186540 *(object+40) = 0xc0001866c0 *(object+48) = 0xc000186840 *(object+56) = 0xc0001869c0 *(object+64) = 0xc000186b40 *(object+72) = 0xc000186cc0 *(object+80) = 0xc000186e40 *(object+88) = 0xc000186fc0 *(object+96) = 0xc000187140 *(object+104) = 0xc0001872c0 *(object+112) = 0xc000187440 *(object+120) = 0xc0001875c0 *(object+128) = 0xc000187740 *(object+136) = 0xc0001878c0 *(object+144) = 0xc000187a40 *(object+152) = 0xc000187bc0 *(object+160) = 0xc000187d40 *(object+168) = 0x0 *(object+176) = 0x0 *(object+184) = 0x0 *(object+192) = 0x0 *(object+200) = 0x0 *(object+208) = 0x0 *(object+216) = 0x0 *(object+224) = 0x0 *(object+232) = 0x0 *(object+240) = 0x0 *(object+248) = 0x0 fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0x4e5f1b?, 0x6?}) /mnt/d/gobuild/golang/go1.18/src/runtime/panic.go:992 +0x71 fp=0xc00005bdf8 sp=0xc00005bdc8 pc=0x4326f1 runtime.badPointer(0x7fd36ca14998, 0xc000082820?, 0xc000184100, 0xc00005bec0?) /mnt/d/gobuild/golang/go1.18/src/runtime/mbitmap.go:368 +0x150 fp=0xc00005be48 sp=0xc00005bdf8 pc=0x412310 runtime.findObject(0xc00004a900?, 0xc00005bee8?, 0x40acc9?) /mnt/d/gobuild/golang/go1.18/src/runtime/mbitmap.go:410 +0xa6 fp=0xc00005be80 sp=0xc00005be48 pc=0x4124a6 runtime.scanobject(0xc000027238?, 0xc000027238) /mnt/d/gobuild/golang/go1.18/src/runtime/mgcmark.go:1318 +0x190 fp=0xc00005bf18 sp=0xc00005be80 pc=0x41d210 runtime.gcDrain(0xc000027238, 0x2) /mnt/d/gobuild/golang/go1.18/src/runtime/mgcmark.go:1083 +0x1d4 fp=0xc00005bf78 sp=0xc00005bf18 pc=0x41ca74 runtime.gcBgMarkWorker.func2() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1282 +0x154 fp=0xc00005bfc8 sp=0xc00005bf78 pc=0x419c54 runtime.systemstack() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:469 +0x49 fp=0xc00005bfd0 sp=0xc00005bfc8 pc=0x45cfe9 goroutine 36 [GC worker (idle)]: runtime.systemstack_switch() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:436 fp=0xc00009bf58 sp=0xc00009bf50 pc=0x45cf80 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1254 +0x1b1 fp=0xc00009bfe0 sp=0xc00009bf58 pc=0x419871 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00009bfe8 sp=0xc00009bfe0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 1 [wait for GC cycle]: runtime.gopark(0x100?, 0x597300?, 0xa0?, 0x80?, 0xc000112a30?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000112a50 sp=0xc000112a30 pc=0x4351f6 runtime.goparkunlock(...) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:372 runtime.gcWaitOnMark(0x3) /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:493 +0x5b fp=0xc000112a80 sp=0xc000112a50 pc=0x41795b runtime.GC() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:436 +0x4e fp=0xc000112ab8 sp=0xc000112a80 pc=0x41780e main.GCZombie() /mnt/d/gobuild/golang/go1.18/src/runtime/testdata/testprog/gc.go:321 +0x5ea fp=0xc000112ee8 sp=0xc000112ab8 pc=0x4b632a main.main() /mnt/d/gobuild/golang/go1.18/src/runtime/testdata/testprog/main.go:34 +0x14d fp=0xc000112f80 sp=0xc000112ee8 pc=0x4b762d runtime.main() /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:255 +0x227 fp=0xc000112fe0 sp=0xc000112f80 pc=0x434e27 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000112fe8 sp=0xc000112fe0 pc=0x45f061 goroutine 2 [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000046fb0 sp=0xc000046f90 pc=0x4351f6 runtime.goparkunlock(...) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:372 runtime.forcegchelper() /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:306 +0xad fp=0xc000046fe0 sp=0xc000046fb0 pc=0x43508d runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000046fe8 sp=0xc000046fe0 pc=0x45f061 created by runtime.init.6 /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:294 +0x25 goroutine 3 [GC sweep wait]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000047790 sp=0xc000047770 pc=0x4351f6 runtime.goparkunlock(...) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:372 runtime.bgsweep(0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc0000477c8 sp=0xc000047790 pc=0x421997 runtime.gcenable.func1() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:177 +0x26 fp=0xc0000477e0 sp=0xc0000477c8 pc=0x4176c6 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000477e8 sp=0xc0000477e0 pc=0x45f061 created by runtime.gcenable /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:177 +0x6b goroutine 4 [GC scavenge wait]: runtime.gopark(0x10eb4b1b94ec4?, 0x10000?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000047f30 sp=0xc000047f10 pc=0x4351f6 runtime.goparkunlock(...) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:372 runtime.bgscavenge(0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/mgcscavenge.go:357 +0x2a8 fp=0xc000047fc8 sp=0xc000047f30 pc=0x41f7e8 runtime.gcenable.func2() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:178 +0x26 fp=0xc000047fe0 sp=0xc000047fc8 pc=0x417666 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000047fe8 sp=0xc000047fe0 pc=0x45f061 created by runtime.gcenable /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:178 +0xaa goroutine 17 [finalizer wait]: runtime.gopark(0x0?, 0xc000046670?, 0x70?, 0x67?, 0x441db1?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000046630 sp=0xc000046610 pc=0x4351f6 runtime.goparkunlock(...) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:372 runtime.runfinq() /mnt/d/gobuild/golang/go1.18/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000467e0 sp=0xc000046630 pc=0x416753 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000467e8 sp=0xc0000467e0 pc=0x45f061 created by runtime.createfing /mnt/d/gobuild/golang/go1.18/src/runtime/mfinal.go:157 +0x45 goroutine 18 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000042758 sp=0xc000042738 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc0000427e0 sp=0xc000042758 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000427e8 sp=0xc0000427e0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 33 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc00009a758 sp=0xc00009a738 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc00009a7e0 sp=0xc00009a758 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00009a7e8 sp=0xc00009a7e0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 49 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000096758 sp=0xc000096738 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc0000967e0 sp=0xc000096758 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000967e8 sp=0xc0000967e0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 19 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000042f58 sp=0xc000042f38 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc000042fe0 sp=0xc000042f58 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 34 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc00009af58 sp=0xc00009af38 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc00009afe0 sp=0xc00009af58 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00009afe8 sp=0xc00009afe0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 50 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000096f58 sp=0xc000096f38 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc000096fe0 sp=0xc000096f58 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000096fe8 sp=0xc000096fe0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 35 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc00009b758 sp=0xc00009b738 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc00009b7e0 sp=0xc00009b758 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00009b7e8 sp=0xc00009b7e0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 20 [GC worker (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000043758 sp=0xc000043738 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc0000437e0 sp=0xc000043758 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000437e8 sp=0xc0000437e0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 51 [GC worker (idle)]: runtime.gopark(0x10eb4b1bfa120?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000097758 sp=0xc000097738 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc0000977e0 sp=0xc000097758 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000977e8 sp=0xc0000977e0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 21 [GC worker (idle)]: runtime.gopark(0x10eb4b1bfa24c?, 0x0?, 0x0?, 0x0?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000043f58 sp=0xc000043f38 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc000043fe0 sp=0xc000043f58 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000043fe8 sp=0xc000043fe0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25 goroutine 52 [GC worker (idle)]: runtime.gopark(0x5c6220?, 0x1?, 0xf8?, 0xb?, 0x0?) /mnt/d/gobuild/golang/go1.18/src/runtime/proc.go:366 +0xd6 fp=0xc000097f58 sp=0xc000097f38 pc=0x4351f6 runtime.gcBgMarkWorker() /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1198 +0xe5 fp=0xc000097fe0 sp=0xc000097f58 pc=0x4197a5 runtime.goexit() /mnt/d/gobuild/golang/go1.18/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000097fe8 sp=0xc000097fe0 pc=0x45f061 created by runtime.gcBgMarkStartWorkers /mnt/d/gobuild/golang/go1.18/src/runtime/mgc.go:1122 +0x25

Change https://golang.org/cl/367044 mentions this issue: runtime: add invalidptr=0 for TestGcZombieReporting