Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime/pprof: found bad pointer in Go heap during TestGoroutineProfileConcurrency with TestMorestack #54605

Closed
bcmills opened this issue Aug 22, 2022 · 4 comments
Closed

runtime/pprof: found bad pointer in Go heap during TestGoroutineProfileConcurrency with TestMorestack #54605

bcmills opened this issue Aug 22, 2022 · 4 comments
Assignees
@mknyszek
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge
Milestone
Go1.20

Comments

@bcmills
Copy link
Contributor

bcmills commented Aug 22, 2022 

runtime: pointer 0x616e6966 to unallocated span span.base()=0x8f78000 span.limit=0x8f79ff8 span.state=0
runtime: found in object at *(0x8d3a420+0x0)
object=0x8d3a420 s.base()=0x8d3a000 s.limit=0x8d3c000 s.spanclass=5 s.elemsize=16 s.state=mSpanInUse
 *(object+0) = 0x616e6966 <==
 *(object+4) = 0x657a696c
 *(object+8) = 0x8114ed7
 *(object+12) = 0x0
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x81d8da0, 0x3e})
	/workdir/go/src/runtime/panic.go:1047 +0x4d fp=0x8c53e2c sp=0x8c53e18 pc=0x807fbdd
runtime.badPointer(0xef804360, 0x616e6966, 0x8d3a420, 0x0)
	/workdir/go/src/runtime/mbitmap.go:316 +0xa2 fp=0x8c53e4c sp=0x8c53e2c pc=0x8058f12
runtime.findObject(0x616e6966, 0x8d3a420, 0x0)
	/workdir/go/src/runtime/mbitmap.go:359 +0x9e fp=0x8c53e64 sp=0x8c53e4c pc=0x805916e
runtime.scanobject(0x8d3a420, 0x8c2895c)
	/workdir/go/src/runtime/mgcmark.go:1341 +0x183 fp=0x8c53eb4 sp=0x8c53e64 pc=0x8065613
runtime.markrootSpans(0x8c2895c, 0x0)
	/workdir/go/src/runtime/mgcmark.go:390 +0x18e fp=0x8c53ef0 sp=0x8c53eb4 pc=0x80630fe
runtime.markroot(0x8c2895c, 0x4, 0x1)
	/workdir/go/src/runtime/mgcmark.go:192 +0xe9 fp=0x8c53f4c sp=0x8c53ef0 pc=0x80628c9
runtime.gcDrain(0x8c2895c, 0x3)
	/workdir/go/src/runtime/mgcmark.go:1069 +0x4ed fp=0x8c53fb0 sp=0x8c53f4c pc=0x8064ffd
runtime.gcBgMarkWorker.func2()
	/workdir/go/src/runtime/mgc.go:1307 +0xaa fp=0x8c53fdc sp=0x8c53fb0 pc=0x806060a
runtime.systemstack()
	/workdir/go/src/runtime/asm_386.s:370 +0x41 fp=0x8c53fe0 sp=0x8c53fdc pc=0x80af4e1

…

goroutine 72 [wait for GC cycle]:
runtime.gopark(0x81dccd8, 0x82ea18c, 0x17, 0x14, 0x1)
	/workdir/go/src/runtime/proc.go:363 +0x108 fp=0x8ca0f10 sp=0x8ca0efc pc=0x80829e8
runtime.goparkunlock(...)
	/workdir/go/src/runtime/proc.go:369
runtime.gcWaitOnMark(0x37)
	/workdir/go/src/runtime/mgc.go:505 +0x5f fp=0x8ca0f24 sp=0x8ca0f10 pc=0x805dbdf
runtime.GC()
	/workdir/go/src/runtime/mgc.go:448 +0x6e fp=0x8ca0f44 sp=0x8ca0f24 pc=0x805da4e
runtime/pprof.TestGoroutineProfileConcurrency.func5(0x8c830e0)
	/workdir/go/src/runtime/pprof/pprof_test.go:1465 +0x19e fp=0x8ca0f9c sp=0x8ca0f44 pc=0x818a15e
testing.tRunner(0x8c830e0, 0x8c100f0)
	/workdir/go/src/testing/testing.go:1446 +0x113 fp=0x8ca0fe4 sp=0x8ca0f9c pc=0x8114fb3
testing.(*T).Run.func1()
	/workdir/go/src/testing/testing.go:1493 +0x2e fp=0x8ca0ff0 sp=0x8ca0fe4 pc=0x8115dae
runtime.goexit()
	/workdir/go/src/runtime/asm_386.s:1326 +0x1 fp=0x8ca0ff4 sp=0x8ca0ff0 pc=0x80b0871
created by testing.(*T).Run
	/workdir/go/src/testing/testing.go:1493 +0x374

greplogs -l -e runtime\\.throw\(\?:.\*\\n\)\*FAIL\\s+runtime/pprof\\s --since=2022-07-01
2022-08-08T17:36:35-ebf182c/linux-386-longtest

attn @golang/runtime
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Aug 22, 2022
@mknyszek
Copy link
Contributor

I think it's likely this might be the same bug fixed by https://go.dev/425194, because the span in question being scanned is noscan. It would be even more evidence if this test was using cgo. Putting in WaitingForInfo in case it shows up again.

@mknyszek
Copy link
Contributor

Actually, if there's no cgo then this is new, so I guess we should check that.

@mknyszek mknyszek added this to the Go1.20 milestone Aug 24, 2022
@mknyszek mknyszek self-assigned this Aug 24, 2022
@mknyszek
Copy link
Contributor

Also, although the failure is from August 8th, we did confirm in triage that this in the window where @randall77 first landed the bitmap changes, which would cause this issue. That was then reverted and landed again because of a different issue, but that issue could be related. I'll check that as well.

@mknyszek
Copy link
Contributor

https://go.dev/cl/422435 was landed in between the revert and the re-land, which specifically mentions noscan objects being scanned. I think this is exactly that, and this issue should be closed. Keith also landed a more general fix for this kind of situation recently, so it should be less likely to happen anyway, but we'll reopen this issue if we see it again.

@golang golang locked and limited conversation to collaborators Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mknyszek mknyszek

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge
Projects
None yet
Milestone
Go1.20
Development

No branches or pull requests
3 participants
@mknyszek @bcmills @gopherbot