Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: MSpanList_Remove #14990

Closed
googollee opened this issue Mar 28, 2016 · 12 comments
Closed

runtime: fatal error: MSpanList_Remove #14990

googollee opened this issue Mar 28, 2016 · 12 comments
Milestone

Comments

@googollee
Copy link

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    go version go1.6 linux/amd64
  2. What operating system and processor architecture are you using (go env)?
    Linux xxxx 3.19.0-25-generic unknown pc's #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
  3. What did you do?
    If possible, provide a recipe for reproducing the error.
    A complete runnable program is good.
    A link on play.golang.org is best.

Our program is a long TCP connection server, which receiving data from a TCP and forwarding it to other. When it upgrade to 1.6, it run a long time (around 4 days) then panic with same error. Log attached below:

failed MSpanList_Remove 0x7f27485cf7f0 0x7f27481d3aa8 0xb6f170 0xb6f180
fatal error: MSpanList_Remove

runtime stack:
runtime.throw(0x947fd0, 0x10)
        /opt/go1.6/src/runtime/panic.go:530 +0x90
runtime.(*mSpanList).remove(0xb6f180, 0x7f27485cf7f0)
        /opt/go1.6/src/runtime/mheap.go:911 +0x1ad
runtime.(*mcentral).freeSpan(0xb6f160, 0x7f27485cf7f0, 0x16, 0xc8cbdc4080, 0xc8cbdc5b80, 0xc847e62100, 0x100000001)
        /opt/go1.6/src/runtime/mcentral.go:178 +0x131
runtime.(*mspan).sweep(0x7f27485cf7f0, 0x3efd50003ef00, 0xc800021b01)
        /opt/go1.6/src/runtime/mgcsweep.go:319 +0x613
runtime.sweepone(0x4)
        /opt/go1.6/src/runtime/mgcsweep.go:112 +0x23e
runtime.gosweepone.func1()
        /opt/go1.6/src/runtime/mgcsweep.go:124 +0x21
runtime.systemstack(0x7f272ab15db8)
        /opt/go1.6/src/runtime/asm_amd64.s:307 +0xab
runtime.gosweepone(0xb6eb08)
        /opt/go1.6/src/runtime/mgcsweep.go:125 +0x3d
runtime.deductSweepCredit(0x8000, 0x0)
        /opt/go1.6/src/runtime/mgcsweep.go:384 +0xc6
runtime.(*mcentral).cacheSpan(0xb70a50, 0x7f27703fd000)
        /opt/go1.6/src/runtime/mcentral.go:36 +0x56
runtime.(*mcache).refill(0x7f278da73000, 0xc900000042, 0xc867dc1380)
        /opt/go1.6/src/runtime/mcache.go:119 +0xcc
runtime.mallocgc.func2()
        /opt/go1.6/src/runtime/malloc.go:642 +0x2b
runtime.systemstack(0xc82002b500)
        /opt/go1.6/src/runtime/asm_amd64.s:291 +0x79
runtime.mstart()
        /opt/go1.6/src/runtime/proc.go:1048
  1. What did you expect to see?
    No crash
  2. What did you see instead?
    Crash
@googollee
Copy link
Author

There is a old issue with same panic backtrace: #14831

@davecheney
Copy link
Contributor

@googollee #14831 was due to a bug in some code called from cgo, does your code use cgo ?

Is your code available ? Can you provide a sample which reproduces the problem ?

@bits01
Copy link

bits01 commented Mar 28, 2016

@davecheney to be correct, #14831 did not involve cgo at all, it was just a syscall (ioctl) and there was a bug in a proprietary kernel driver that in some edge case went past the provided buffer by 4 bytes, resulting in memory corruption.
@googollee if you are dealing with memory corruptions, they can happen with syscalls, or use of the unsafe package (pointers) if there's a bug in the pointer math or type conversions via unsafe.Pointer/uintptr

@googollee
Copy link
Author

There is no cgo or syscall or unsafe in my code. I will try to make a sample to reproduce it, but the code is really complex and mess ,and panic after a long run. I'm not sure which part cause this issue now.

@davecheney
Copy link
Contributor

The first thing I think your should do is build a race enabled version of your application, go build -race and deploy that.

On 28 Mar 2016, at 19:41, Googol Lee notifications@github.com wrote:

There is no cgo or syscall or unsafe in my code. I will try to make a sample to reproduce it, but the code is really complex and mess ,and panic after a long run. I'm not sure which part cause this issue now.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

@googollee
Copy link
Author

OK. I will try.

@ianlancetaylor ianlancetaylor added this to the Go1.7 milestone Mar 28, 2016
@ianlancetaylor
Copy link
Contributor

CC @aclements @RLH

@aclements
Copy link
Member

failed MSpanList_Remove 0x7f27485cf7f0 0x7f27481d3aa8 0xb6f170 0xb6f180

Looks like the span was on nonempty, but was being removed from empty because its freelist was nil. Unfortunately, I don't think there's enough information in the traceback to glean which of these conditions was wrong.

@bits01
Copy link

bits01 commented Mar 28, 2016

@googollee it may be worth looking at the dependencies/3rd party libs you may be using and they may be using in turn. Your code may not be doing anything with the unsafe package/syscall/cgo directly but some dependency may.

@googollee
Copy link
Author

I try go build -race and hope to get some log to check now.

@bits01 I have checked all 3rd party libs and all of them are not using cgo/unsafe/sysctl.

@googollee
Copy link
Author

I find some race code and try to fix them. After that I will check if the program will panic again.

Thanks.

@bradfitz
Copy link
Contributor

I'm going to close this until there's something actionable for us to do. Feel free to reopen if you have more details to suggest it's a bug in Go.

@golang golang locked and limited conversation to collaborators Mar 30, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants