runtime: crash with "leftover defer" #11023
Labels
Milestone
Comments
|
CL https://golang.org/cl/10713 mentions this issue. |
|
CL https://golang.org/cl/10791 mentions this issue. |
methane
pushed a commit
to methane/go
that referenced
this issue
Jun 6, 2015
Issues golang#10240, golang#10541, golang#10941, golang#11023, golang#11027 and possibly others are indicating memory corruption in the runtime. One of the easiest places to both get corruption and detect it is in the allocator's free lists since they appear throughout memory and follow strict invariants. This commit adds a check when sweeping a span that its free list is sane and, if not, it prints the corrupted free list and panics. Hopefully this will help us collect more information on these failures. Change-Id: I6d417bcaeedf654943a5e068bd76b58bb02d4a64
aclements
added a commit
that referenced
this issue
Jun 7, 2015
Stack barriers assume that writes through pointers to frames above the current frame will get write barriers, and hence these frames do not need to be re-scanned to pick up these changes. For normal writes, this is true. However, there are places in the runtime that use typedmemmove to potentially write through pointers to higher frames (such as mapassign1). Currently, typedmemmove does not execute write barriers if the destination is on the stack. If there's a stack barrier between the current frame and the frame being modified with typedmemmove, and the stack barrier is not otherwise hit, it's possible that the garbage collector will never see the updated pointer and incorrectly reclaim the object. Fix this by making heapBitsBulkBarrier (which lies behind typedmemmove and its variants) detect when the destination is in the stack and unwind stack barriers up to the point, forcing mark termination to later rescan the effected frame and collect these pointers. Fixes #11084. Might be related to #10240, #10541, #10941, #11023, #11027 and possibly others. Change-Id: I323d6cd0f1d29fa01f8fc946f4b90e04ef210efd Reviewed-on: https://go-review.googlesource.com/10791 Reviewed-by: Russ Cox <rsc@golang.org>
aclements
added a commit
that referenced
this issue
Jun 16, 2015
Issues #10240, #10541, #10941, #11023, #11027 and possibly others are indicating memory corruption in the runtime. One of the easiest places to both get corruption and detect it is in the allocator's free lists since they appear throughout memory and follow strict invariants. This commit adds a check when sweeping a span that its free list is sane and, if not, it prints the corrupted free list and panics. Hopefully this will help us collect more information on these failures. Change-Id: I6d417bcaeedf654943a5e068bd76b58bb02d4a64 Reviewed-on: https://go-review.googlesource.com/10713 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Russ Cox <rsc@golang.org> Run-TryBot: Austin Clements <austin@google.com>
|
Hi @rhysh. We've fixed several memory corruption and lost write barrier issues in the runtime over the past few weeks. Please try to reproduce the problem with current master and reopen this issue if it's still happening. Thanks! |
|
Hi @aclements - I ran my app with 8fa1a69 (from 17 Jun 2015) for a few days and did not observe any crashes. It looks like this is resolved. Thanks! |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I have a process that receives data over a few hundred concurrent TCP connections and writes them to files. It's been crashing on recent versions of tip (it was stable on 1.4.1) with messages like "runtime: g422: leftover defer sp=0xdeaddeaddeaddead pc=0x48d60e" and "runtime: g883: leftover defer sp=0x6576452e67736d62 pc=0x69203a746553746e". This may be related to #9610 and #10941, but I've seen it (once) with sp != 0xdeaddeaddeaddead .
The stack trace with non-0xdead address is a bit more interesting - it looks like two separate threads are dumping stacks.
The text was updated successfully, but these errors were encountered: