Skip to content

runtime: bad pointer 0xc900000000 in unexpected span (dup #13160?) #13176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fluhus opened this issue Nov 6, 2015 · 13 comments
Closed

runtime: bad pointer 0xc900000000 in unexpected span (dup #13160?) #13176

fluhus opened this issue Nov 6, 2015 · 13 comments
Labels
FrozenDueToAge
Milestone
Go1.5.2

Comments

@fluhus
Copy link

fluhus commented Nov 6, 2015

I am getting this mysterious error message after my program runs for 2-3 hours (full stack trace below). The program runs on tons of data as a part of my academic research, so unfortunately I can't provide the data for you to reproduce it.

The code is here:
https://github.com/fluhus/prices/tree/master/src/items2
(sorry for scrappiness; It's in the middle of a rework)

More info

Let me know if you need additional information.

Thanks a lot!

runtime:objectstart Span weird: p=0xc900000000 k=0x6480000 s.start=0xc8fffe8000 s.limit=0xc900001fe0 s.state=2
fatal error: objectstart: bad pointer in unexpected span

runtime stack:
runtime.throw(0x65c4c0, 0x2b)
        /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.heapBitsForObject(0xc900000000, 0x0, 0x0, 0xc800000000, 0x7fad6c9123e0)
        /usr/local/go/src/runtime/mbitmap.go:217 +0x287
runtime.scanobject(0xc8ccd33b00, 0xc820036720)
        /usr/local/go/src/runtime/mgcmark.go:878 +0x239
runtime.gcDrainN(0xc820036720, 0x485e5)
        /usr/local/go/src/runtime/mgcmark.go:779 +0x8e
runtime.gcAssistAlloc.func1()
        /usr/local/go/src/runtime/mgcmark.go:232 +0x172
runtime.systemstack(0xc820022a00)
        /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
        /usr/local/go/src/runtime/proc1.go:674
@ianlancetaylor ianlancetaylor added this to the Go1.5.2 milestone Nov 6, 2015
@ianlancetaylor
Copy link
Member

CC @RLH @aclements

See also #12157.

@fluhus
Copy link
Author

fluhus commented Nov 7, 2015

I just tried running it using go 1.4.3 and it worked perfectly on the same machine.

@otoolep
Copy link

otoolep commented Nov 11, 2015

This issue has forced us to roll back building InfluxDB 0.9.5 with Go 1.5.1. We've reverted to Go 1.4.2 for the 0.9.5 release.

We are quite disappointed the 0.9.5 release will not be able to use the improved GC. Hopefully we'll have more luck with 0.9.6.

@RLH
Copy link
Contributor

RLH commented Nov 12, 2015

Any clues at how to reproduce this would be appreciated. You have already
indicated that there is no use of CGO/C and that the race detector reveals
no problem. This is good.

Is there any use of the unsafe package in the code?

If not then the next step is to work on reproducing the problem.

On Wed, Nov 11, 2015 at 6:53 PM, otoolep notifications@github.com wrote:

This issue has forced us to roll back building InfluxDB 0.9.5 with Go
1.5.1. We've reverted to Go 1.4.2 for this 0.9.5 release.

We are quite disappointed the 0.9.5 release will not be able to use the
improved GC. Hopefully we'll have more luck with 0.9.6.

—
Reply to this email directly or view it on GitHub
#13176 (comment).

@otoolep
Copy link

otoolep commented Nov 12, 2015

We don't have specific repro steps unfortunately. The problems were independently reported by two users in the InfluxDB community, who had their systems panic on them during normal usage, with this error.

@aclements
Copy link
Member

Given the number of zeroes in the bad pointer p=0xc900000000, I strongly suspect this is the same as issue #13160. @fluhus, is it possible for you to try your program with current Go master, or with 4b7d5f0 cherry-picked on to the release-branch.go1.5 branch?

Also, if you need a workaround, set the GODEBUG=invalidptr=0 environment variable. (But obviously we'd like to fix this issue rather than work around it. :)

@aclements aclements changed the title Go 1.5.1: fatal error: objectstart: bad pointer in unexpected span runtime: fatal error: objectstart: bad pointer in unexpected span Nov 12, 2015
@fluhus
Copy link
Author

fluhus commented Nov 12, 2015

@RLH No use of unsafe in my code. I am trying to find the exact input that would make it crash (though this might be gigs of data).

@aclements Will try and update on results. Thanks!

@aclements
Copy link
Member

@fluhus, it's most likely not a particular input that makes it crash. The underlying bug is a concurrency issue between the garbage collector and copying/zeroing structures that have a pointer field at the end, so you just have to be unlucky. Thanks for working on reproducing it!

@RLH
Copy link
Contributor

RLH commented Nov 12, 2015

I would try Austin's suggestions before working on a repo....

On Thu, Nov 12, 2015 at 1:14 PM, Amit Lavon notifications@github.com
wrote:

@RLH https://github.com/RLH No use of unsafe in my code. I am trying to
find the exact input that would make it crash (though this might be gigs of
data).

@aclements https://github.com/aclements Will try and update on results.
Thanks!

—
Reply to this email directly or view it on GitHub
#13176 (comment).

@davecheney
Copy link
Contributor

Would reducing GOGC below 100 increase the chance of reproducing the issue ?

On Fri, Nov 13, 2015 at 5:31 AM, Austin Clements notifications@github.com
wrote:

@fluhus https://github.com/fluhus, it's most likely not a particular
input that makes it crash. The underlying bug is a concurrency issue
between the garbage collector and copying/zeroing structures that have a
pointer field at the end, so you just have to be unlucky. Thanks for
working on reproducing it!

—
Reply to this email directly or view it on GitHub
#13176 (comment).

@aclements
Copy link
Member

Would reducing GOGC below 100 increase the chance of reproducing the issue ?

Hmm. Probably, though I don't have an intuition for how much. Of course, if you set it too low it may perturb things enough to make the failure less likely.

@rsc
Copy link
Contributor

rsc commented Nov 13, 2015

I'm almost certain this is a duplicate of #13160, for the reasons Austin gives (low zeros in pointer).

@rsc rsc changed the title runtime: fatal error: objectstart: bad pointer in unexpected span runtime: bad pointer 0xc900000000 in unexpected span Nov 17, 2015
@rsc rsc changed the title runtime: bad pointer 0xc900000000 in unexpected span runtime: bad pointer 0xc900000000 in unexpected span (dup #13160?) Nov 17, 2015
@rhysh rhysh mentioned this issue Nov 17, 2015
Closed
@rsc
Copy link
Contributor

rsc commented Nov 18, 2015

Closing as duplicate of #13160.

@rsc rsc closed this as completed Nov 18, 2015
@golang golang locked and limited conversation to collaborators Nov 17, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.5.2
Development

No branches or pull requests
8 participants
@davecheney @rsc @otoolep @RLH @aclements @ianlancetaylor @fluhus @gopherbot