Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: bad pointer 0xc84809c000 unexpected span #13289

Closed
rhysh opened this issue Nov 17, 2015 · 4 comments
Closed

runtime: bad pointer 0xc84809c000 unexpected span #13289

rhysh opened this issue Nov 17, 2015 · 4 comments
Labels
FrozenDueToAge
Milestone
Go1.6

Comments

@rhysh
Copy link
Contributor

rhysh commented Nov 17, 2015

I encountered this crash on linux/amd64 with go1.5.1 compiled with GOEXPERIMENT=framepointer. There is no use of cgo or unsafe. I re-ran my program with the race detector, which found nothing.

The program is running on Linux 3.13 on a machine with 60GB memory and 36 cores.

It looks like this crash is different than #13176 - the bad pointer isn't nearly full of zeros, it's outside the selected span, and the span state is set to mSpanInUse.

runtime:objectstart Span weird: p=0xc84809c000 k=0x642404e s.start=0xc8d25fe000 s.limit=0xc8d2600000 s.state=0
fatal error: objectstart: bad pointer in unexpected span

goroutine 69 [running]:
runtime.throw(0xb77420, 0x2b)
    /usr/local/go/src/runtime/panic.go:527 +0x9b fp=0xc82054c5f8 sp=0xc82054c5d8
runtime.heapBitsForObject(0xc84809c000, 0x0, 0x0, 0xc800000000, 0x7fac7bd79c70)
    /usr/local/go/src/runtime/mbitmap.go:217 +0x29b fp=0xc82054c638 sp=0xc82054c5f8
runtime.scanobject(0xc90cb64000, 0xc820027c20)
    /usr/local/go/src/runtime/mgcmark.go:878 +0x253 fp=0xc82054c710 sp=0xc82054c638
runtime.gcDrain(0xc820027c20, 0x7d0)
    /usr/local/go/src/runtime/mgcmark.go:689 +0x103 fp=0xc82054c748 sp=0xc82054c710
runtime.gcBgMarkWorker(0xc820026a00)
    /usr/local/go/src/runtime/mgc.go:1320 +0x297 fp=0xc82054c7b8 sp=0xc82054c748
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82054c7c0 sp=0xc82054c7b8
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1239 +0x9f

goroutine 1 [semacquire, 3278 minutes]:
sync.runtime_Semacquire(0xc8203fa55c)
    /usr/local/go/src/runtime/sema.go:43 +0x30
sync.(*WaitGroup).Wait(0xc8203fa550)
    /usr/local/go/src/sync/waitgroup.go:126 +0xc3
main.main()
    /redacted.go:211 +0xfb0

CC @RLH @aclements
@ianlancetaylor ianlancetaylor added this to the Go1.6 milestone Nov 17, 2015
@rhysh
Copy link
Contributor Author

rhysh commented Nov 17, 2015

I just got another one of these - from the same program, still on go1.5.1 with GOEXPERIMENT=framepointer.

Again the span is marked as mSpanInUse, and the pointer is "weird" because it's outside of the span's address range.

There's no use of package unsafe in our first-party code, but there is use of unsafe in the program via github.com/golang/protobuf/proto (which I trust to be correct, and does not have any GitHub issues related to crashes like this).

runtime:objectstart Span weird: p=0xc843bb6000 k=0x6421ddb s.start=0xc826b24000 s.limit=0xc826b26000 s.state=0
fatal error: objectstart: bad pointer in unexpected span

goroutine 40 [running]:
runtime.throw(0xb82480, 0x2b)
    /usr/local/go/src/runtime/panic.go:527 +0x9b fp=0xc8201cbdf0 sp=0xc8201cbdd0
runtime.heapBitsForObject(0xc843bb6000, 0x0, 0x0, 0xc800000000, 0x7fc7f89f72c0)
    /usr/local/go/src/runtime/mbitmap.go:217 +0x29b fp=0xc8201cbe30 sp=0xc8201cbdf0
runtime.scanobject(0xc841a24400, 0xc82002e720)
    /usr/local/go/src/runtime/mgcmark.go:878 +0x253 fp=0xc8201cbf08 sp=0xc8201cbe30
runtime.gcDrainUntilPreempt(0xc82002e720, 0x7d0)
    /usr/local/go/src/runtime/mgcmark.go:741 +0x162 fp=0xc8201cbf48 sp=0xc8201cbf08
runtime.gcBgMarkWorker(0xc82002d500)
    /usr/local/go/src/runtime/mgc.go:1329 +0x48a fp=0xc8201cbfb8 sp=0xc8201cbf48
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201cbfc0 sp=0xc8201cbfb8
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1239 +0x9f

goroutine 1 [semacquire, 1175 minutes]:
sync.runtime_Semacquire(0xc82040b1cc)
    /usr/local/go/src/runtime/sema.go:43 +0x30
sync.(*WaitGroup).Wait(0xc82040b1c0)
    /usr/local/go/src/sync/waitgroup.go:126 +0xc3
main.main()
    /redacted.go:211 +0xfb0

@rsc
Copy link
Contributor

rsc commented Nov 18, 2015

If you can rebuild from an alternate version of Go, please try release-branch.go1.5 (currently a10b270). I'm nearly certain this is the atomic pointer write problem, which is fixed on the release branch and will be part of Go1.5.2

@rhysh
Copy link
Contributor Author

rhysh commented Nov 19, 2015

I'm running my app now with a10b270, I'll update if I see the crash again. Thank you.

@rhysh
Copy link
Contributor Author

rhysh commented Nov 23, 2015

I haven't seen this crash since upgrading to a10b270 four days ago. Thanks @rsc.

@rhysh rhysh closed this as completed Nov 23, 2015
@thisisaaronland thisisaaronland mentioned this issue Jan 8, 2016
Closed
@golang golang locked and limited conversation to collaborators Nov 27, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.6
Development

No branches or pull requests
4 participants
@rsc @rhysh @ianlancetaylor @gopherbot