runtime: fail with: Fatal glibc error: rseq registration failed
#51315
Comments
mengzhuo
changed the title
Fatal glibc error: rseq registration failedFatal glibc error: rseq registration failed
|
Could you provide |
|
I can't reproduce with your steps: |
mengzhuo
added
the
NeedsInvestigation
|
@mengzhuo yes, I have plenty of coredumps! Some more info about my glibc: Some extra info about the binary crashing: Here is the output of And the file: filebeat.dump.gz |
|
The code that is failing appears to have been added to glibc very recently, in December, 2021. The first version it appears in is glibc 2.35. I think the patch series starts here: https://sourceware.org/pipermail/libc-alpha/2021-December/133656.html . I don't know why it would affect Go programs in this way. CC @fweimer for thoughts. @belimawr Do all Go programs fail? Do all Go programs that use cgo fail? Or is there something particular to the program that you are reporting? Thanks. |
|
@ianlancetaylor So far only this program is affected, it's a huge project so it's hard to pinpoint something, but there are a few high-level things I know happen when I experience the crash:
So far I haven't had much opportunity to experiment with other applications, but the go tool/compile is working fine, also my unit tests run without any problems. I opened up this issue because the only change to create this problem was a system update (no code change) and both the error and stack trace show low level stuff leading me to believe could be related to how the runtime interacts with the OS/Kernel. I'll try to gather more information about how cgo is used on this application. |
|
I've been playing with delve trying to find some pattern for those crashes, and I managed to get the program crashing on the same line multiple times. Interestingly those crashes are on a channel read, I also got a coredump right before the crash. Here is the coredump: I built the binary from this commit on elastic/beats, to build the binary:
A few interesting things:
The stack trace and where the program crashed: The list of goroutines |
|
I'm not sure the goroutine stack traces are all that helpful. The error is almost certainly coming from the libc function It is also possible that the C code called by your program is calling If you can capture a backtrace from the code in |
|
https://www.phoronix.com/scan.php?page=news_item&px=Glibc-RSEQ-Still-2020
cc @compudj per author of this patch |
|
It is likely that beats installs a seccomp handler that blocks the |
Hi, I can confirm it crashes journalbeat and journalbeat-oss instances too |
This sounds correct to me. I see a policy here which does not include |
|
Thank you so much for the help folks ❤️ ! @fweimer-rh, @prattmic that was the issue. I added I'm closing this issue as it not a runtime bug. Sorry for the trouble. |
|
For what its worth, difficult to debug cases like this are why when using a seccomp policy I prefer to:
|
Thank you so much for those tips ❤️ ! I'll try to implement them so we won't be bitten by it again. |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
I built filebeat and run it, then I get this message:
I looked at the core dump with delve and that's what I got:
The issue started when I updated my system (Arch Linux) and as part of the update glibc was updated to 2.35.
Somehow it seems related to starting threads, so I have not managed to reproduce on a simple Go program.
I'm not sure if this is an issue with glibc or the way the Go runtime interacts with it. The rest of my system is stable, so I'm inclined to believe it's Go related. I have not changed my application at all, just the system update. I also had the same behaviour with an older Go version (1.16 I believe). I also have some colleges reporting similar problems when running Go programs.
Disabling CGO when compiling makes the issue go away.
I see the same issues on an Arch Linux VM running glibc 2.35 and the latest Go compiler
Everything works on a Arch Linux with glibc 2.33.
Some more information about my environment:
Linux distro: Arch Linux
Full steps to reproduce
Clone, compile and run filebeat:
The
config.ymlfile is this:What did you expect to see?
The program to run normally.
What did you see instead?
The program crashing and a core dump generated. Most of the times it breaks, but some times (very few times) it just runs without any issue.
The text was updated successfully, but these errors were encountered: