gollvm: runtime and runtime/pprof packages test failed #37257
Comments
erifan
changed the title
|
Error info of runtime package test: Invalid user of intrinsic instruction! |
|
The error log of the test of package runtime/pprof if very long, part of the log: --- FAIL: TestCPUProfile (7.53s) |
toothrot
added
the
NeedsInvestigation
|
/cc @thanm |
|
I'll take a look |
|
I spent some time looking at the test failure of the runtime package. It contains several problems. One is that Memmove intrinsic functions cannot be called indirectly. The second is a segmentation fault. This segfault happens during invoking morestack function. Since there is very little stack trace information, I don't know how to debug it further. Any suggestions? |
|
By the way, the segmentation fault only happens on Linux Amd64 platform when building in release mode. This is my cmake command: gofrontend version: tip |
|
I have been very busy with other work, sorry for not responding. Not sure how much time I can devote to this, things are a bit oversubscribed. The memmove problem is due to recent changes in the runtime test (d6bf2d7 in the main Go repo) and in the LLVM IR verifier. The verifier now issues an error if you try to take the address of an intrinsic function, which is now happening in one of the runtime tests. Problem can be reproduced with this code: I think the way forward here is to catch this case in the bridge and emit a stub or wrapper function, but I haven't written the code for this yet. If you comment out that test and/or run with a release build (no verifier assertions), the SEGV looks like it is happening in runtime_test.TestSemaHandoff. From my gdb session: So far I don't see anything too suspicious -- at this point in the test it is basically just starting a lot of goroutines. I linked the test with a better copy of libgcc.a (for debugging symbols), here's what it reported: This seems to point to __morestack_current_segment being corrupted somehow, which I don't have a good theory about. |
|
Yes, I have debugged here too. I just don't know why __morestack_current_segment->next is an illegal address. I commented out several tests which can trigger SIGSEGV, including TestCallersNilPointerPanic, TestCallersDeferNilFuncPanic and TestCallersDivZeroPanic in caller_test.go, and TestPanicInlined in crash_test.go, but the segment fault still occurs. In addition, I used valgrind to check the memory leak situation. The error message is very long, mostly like this: Under normal circumstances, valgrind check will also report many errors, but the error type is not the case: I have some time to look at this issue recently, but I don't know which direction to go deeper. |
|
I suspect that valgrind is not going to be a lot of help; it is not really set up for Go binaries. In particular it seems to be confused by the TLS references that the morestack code is making. Regarding commenting out tests, you can also run just a single testpoint with the ./a.out -test.short -test.v -test.run=TestSemaHandoff In terms of directions to go, maybe running under the debugger and using hardware watchpoints? It would also be useful to see why the same test isn't failing with gccgo (e.g. what's different between gccgo and gollvm). |
|
Just a quick thought: Could you try disabling getg inlining? (I lost access to my Linux machine so it's not easy for me to test myself.) The getg inlining is one place where gollvm and gccgo differ, and it delicately depends on the detail of LLVM optimizer. If this is broken, a bad G pointer could lead to a bad stack context, which could cause a fault in morestack. |
|
Hi Cherry，disabling inline or building in debug mode can fix this issue.
|
Hi, this is not correct. Sorry for the false information. I retested turning off inline, and the problem persists. Ian perfected this test in the Go project not long ago (https://go-review.googlesource.com/c/go/+/219277), and I tried his patch in gollvm. This problem still exists. |
|
Turning off split-stack fixes the failure, So I think it should be related to stack splitting, and the location where the error happens also reveals this. I found the following text in the source code of morestack (generic-morestack.c) I don't know if it is related to this problem. |
|
@erifan That comment is unlikely to indicate a problem. That comment is mainly aimed at C code that uses |
|
Change https://golang.org/cl/228239 mentions this issue: |
|
Change https://golang.org/cl/229337 mentions this issue: |
|
Change https://golang.org/cl/230218 mentions this issue: |
|
With this patch https://go-review.googlesource.com/c/gofrontend/+/229337/, I found that the pprof package still has a random segfault on arm64. This segfault may occur when running these tests: The frames below the frame |
|
I fixed a few issues but there are still three more difficult ones. |
|
I'm willing to tackle the first issue (taking address of memmove) at some point. Sorry for the delays in responding on this, there have been a lot of other competing projects. |
|
CC @bsd-hacker |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
$ ninja check-gollvm
What did you expect to see?
All test pass
What did you see instead?
Tests of package runtime and runtime/pprof failed
@thanm @ianlancetaylor @cherrymui
The text was updated successfully, but these errors were encountered: