runtime: r2 corrupted by mid-stack inlining on ppc64le #30283
Comments
|
/cc @laboger |
|
Does this happen with 1.11.5? If not, this is a regression that needs attention soon. |
@mvdan tests are passing without any issues in 1.11.5 version, test results are attached here - https://gist.github.com/mkumatag/24f2068a803f36758f938da206312d9f |
|
Thank you for the report @mkumatag and @mvdan for the initial response. @mkumatag, initially I couldn't reproduce your produce using the Docker container steps as some steps seem to be missing. However, after a plain: go get github.com/containerd/containerd && \
cd $GOPATH/src/github.com/containerd/containerd/metadata && \
go test -buildmode=pie -vI cannot get a crash either on the Docker container nor on my Macbook pro, where both are |
This is happening in the ppc64le architecture, lemme modify the description |
mkumatag
changed the title
|
Just from looking at the location where this was failing (see moby/moby#38404 (comment)), these changes could be possible suspects: 05166bf and 020a18c /cc @martisch |
|
I have found that it starts to fail with commit 69c2c56. @randall77 The SEGV occurs because r2 has been corrupted during a call to github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).inlineable from github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill. Before this change, the code at the end of github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).inlineable looks like this: After this change, the same code looks like this. It is incorrectly loading the value of r2 from 24(r1). That location has never been initialized with the value of r2 so some random value is being loaded and then used by the code when this function returns. |
|
FYI... I was able to reproduce the problem without building it in a container. |
ianlancetaylor
added
NeedsInvestigation
ianlancetaylor
changed the title
|
This same error happens in pkg math test Nextafter32 when build with -buildmode=pie go test -buildmode=pie -test.run=Nextafter32 goroutine 5 [running]: goroutine 1 [chan receive]: |
|
I am a Fedora maintainer for Go and during our latest mass rebuild with 1.12 rc1, we encountered several packages segfaulting on ppc64le during tests. For example: Do you think it could be related to this bug? |
|
Yes, anything built for PIC (i.e., using -buildmode=pie or -shared) could hit this bug. The commit that I noted above is adding the load of r2 from a location that might not have been initialized, and if that is used to generate an address then a SEGV can definitely occur. It doesn't look like anything has been happening here, I have a fix for it but not sure what the process is at this point. Submit into the go1.12 branch only or upstream too? |
|
Change https://golang.org/cl/163337 mentions this issue: |
|
@mkumatag Can you test this out to verify it fixes your issue? I verified it on the cases I was aware of. |
|
The headline is misleading, the problem isn't with the inlining but with the insertion of NOPs to fix stacktraces of inlined functions. |
|
@laboger Submit your fix to master, and when it is approved and submitted we can cherry pick it to the 1.12 branch. |
|
Not that it matters, but the NOPs are a result of adding support for mid-stack inlining, so it seems to me that the subject line isn't completely wrong. But of course feel free to edit it. |
@laboger I ran the containerd tests(tests, integration) with the submitted patch. I don't see any issues. test logs: https://gist.github.com/mkumatag/dd0cafcf968700119da401e6a652683f |
|
Change https://golang.org/cl/163717 mentions this issue: |
…c64le for mid-stack inlining tracebacks A recent change to fix stacktraces for inlined functions introduced a regression on ppc64le when compiling position independent code. That happened because ginsnop2 was called for the purpose of inserting a NOP to identify the location of the inlined function, when ginsnop should have been used. ginsnop2 is intended to be used before deferreturn to ensure r2 is properly restored when compiling position independent code. In some cases the location where r2 is loaded from might not be initialized. If that happens and r2 is used to generate an address, the result is likely a SEGV. This fixes that problem. Fixes #30283 Change-Id: If70ef27fc65ef31969712422306ac3a57adbd5b6 Reviewed-on: https://go-review.googlesource.com/c/163337 Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com> Reviewed-by: Keith Randall <khr@golang.org> Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> (cherry picked from commit 2d34740) Reviewed-on: https://go-review.googlesource.com/c/163717 Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes, reproduced with 1.12rc1
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
Steps to reproduce this issues:
What did you expect to see?
No panic
What did you see instead?
Seen panic with
fatal error: faulterrorThe text was updated successfully, but these errors were encountered: