runtime: sweep increased allocation count crash on arm64 #36101
Comments
ALTree
changed the title
ALTree
added
the
NeedsInvestigation
|
We can reproduce crash using golang master branch: stderr print like below: 326 if nalloc > s.allocCount {
|
|
Do you have any patches on the master branch you are using? Tell us about your program. The most common cause of this kind of error is a pointer being incorrectly converted to uintptr, or C code storing a Go pointer into Go memory. Those are both forbidden. So: do you use unsafe? Do you use cgo? Is there a way that we can reproduce the problem ourselves? |
no.
the program use unsafe. if we set GODEBUG=gcstoptheworld=2, we can not reproduce the crash.
reproduce steps:
I'm trying to find a simple way to reproduce it. |
|
You could try |
|
using GODEBUG=gccheckmark=1 Unfortunately, we did not get the stderr/stdout info. we'll update later when get it. |
|
update with stderr info dump 0x4000218ac0 unsafe pointer mainly used by golang.org/x/sys, I'm trying to update sys package and try again. |
|
We create a program which can reproduce on go1.13.3.(we'll test later on master) we try to create many(more than 10) go-issue-36101 processs, then wait hours get message in nohup.out.
|
|
yes, we can reproduce on go master branch.
|
|
@ianlancetaylor would you please try https://github.com/jing-rui/go-issue-36101 to reproduce. We can reproduce at 2 different type of physical arm machine, set GODEBUG=stoptheworld=1 or GODEBUG=stoptheworld=2 still panic. |
|
We compared using jsoniter to replace json lib, the program using jsoniter can run days without crash. So I think there must be something wrong in encoding/json. The reproduce program is updated at https://github.com/jing-rui/go-issue-36101. |
|
Thanks @jing-rui . I'm able to reproduce this with https://github.com/jing-rui/go-issue-36101 on the Linux/ARM64 builder. Running with @jing-rui , could you test CL http://golang.org/cl/212626 ? Thanks. |
|
Change https://golang.org/cl/212626 mentions this issue: |
|
The test run 2 days, it works. Thanks @cherrymui . |
|
@gopherbot, please open a backport to Go 1.13 |
|
Backport issue(s) opened: #36361 (for 1.13). Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases. |
|
@gopherbot, please open a backport to Go 1.12 |
|
Change https://golang.org/cl/213418 mentions this issue: |
|
Change https://golang.org/cl/213683 mentions this issue: |
|
Change https://golang.org/cl/213684 mentions this issue: |
…lly on ARM64 If a pointer write is not atomic, if the GC is running concurrently, it may observe a partially updated pointer, which may point to unallocated or already dead memory. Most pointer writes, like the store instructions generated by the compiler, are already atomic. But we still need to be careful in places like memmove. In memmove, we don't know which bits are pointers (or too expensive to query), so we ensure that all aligned pointer-sized units are written atomically. Fixes #36367. Updates #36101. Change-Id: I1b3ca24c6b1ac8a8aaf9ee470115e9a89ec1b00b Reviewed-on: https://go-review.googlesource.com/c/go/+/212626 Reviewed-by: Austin Clements <austin@google.com> (cherry picked from commit ffbc027) Reviewed-on: https://go-review.googlesource.com/c/go/+/213684 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
…lly on ARM64 If a pointer write is not atomic, if the GC is running concurrently, it may observe a partially updated pointer, which may point to unallocated or already dead memory. Most pointer writes, like the store instructions generated by the compiler, are already atomic. But we still need to be careful in places like memmove. In memmove, we don't know which bits are pointers (or too expensive to query), so we ensure that all aligned pointer-sized units are written atomically. Fixes #36361. Updates #36101. Change-Id: I1b3ca24c6b1ac8a8aaf9ee470115e9a89ec1b00b Reviewed-on: https://go-review.googlesource.com/c/go/+/212626 Reviewed-by: Austin Clements <austin@google.com> (cherry picked from commit ffbc027) Reviewed-on: https://go-review.googlesource.com/c/go/+/213683 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
Unlike C's memmove, Go's memmove must be careful to do indivisible writes of pointer values because it may be racing with the garbage collector reading the heap. We've had various bugs related to this over the years (#36101, #13160, #12552). Indeed, memmove is a great target for optimization and it's easy to forget the special requirements of Go's memmove. The CL documents these (currently unwritten!) requirements. We're also adding a test that should hopefully keep everyone honest going forward, though it's hard to be sure we're hitting all cases of memmove. Change-Id: I2f59f8d8d6fb42d2f10006b55d605b5efd8ddc24 Reviewed-on: https://go-review.googlesource.com/c/go/+/213418 Reviewed-by: Cherry Zhang <cherryyz@google.com>
What version of Go are you using (
go version)?we applied some patch to go1.13.3:
tested using go1.11.2 and go1.13.3, crash occurs on both golang version.
Does this issue reproduce with the latest release?
not tested, the go1.13.5 shall reproducible. I'll try master branch soon.
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
we get 2 core files core_containerd-shim_31862_1576112309 core_containerd-shim_53653_1576133943
What did you expect to see?
containerd-shim not crash and no core files generated.
What did you see instead?
containerd-shim crash on arm64.
The text was updated successfully, but these errors were encountered: