Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: ebpf uretprobe support #22008

Open
shashankmjain opened this issue Sep 25, 2017 · 12 comments
Open

runtime: ebpf uretprobe support #22008

shashankmjain opened this issue Sep 25, 2017 · 12 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted OS-Linux
Milestone

Comments

@shashankmjain
Copy link

We had issue with using uprobes with ebpf due to dynamic nature of golang stacks. Is there a proposal/way to address that?

@ianlancetaylor
Copy link
Contributor

What is ebpf? What are uprobes?

The Go compiler is committed to having stacks that grow, and therefore move, as needed. That is a key part of supporting programs that use a very large number of goroutines.

@shashankmjain
Copy link
Author

ebpf is the linux tracing mechanism and uprobes are dynamic probes for user space programs. The ebpf program attaches to a user probe and gets executed when that function is invoked. Here it seems the golang stack is resized and the return of the ebpf program gets a segmentation fault as it points to a invalid address

@cznic
Copy link
Contributor

cznic commented Sep 25, 2017

Sounds like ebpf/uprobes supports only processes having a pure C ABI with thread-static stacks .

@ianlancetaylor
Copy link
Contributor

I don't see how we could support a dynamic probe executed by the kernel that refers to a local address on the stack.

@ianlancetaylor ianlancetaylor changed the title ebpf uprobe support runtime: ebpf uprobe support Sep 25, 2017
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Sep 25, 2017
@brendangregg
Copy link

Anyway for go to pause the moving of stacks, eg, by sending it a signal? That way the behavior could be disabled temporarily while the kernel did user-level instrumentation.

@ianlancetaylor
Copy link
Contributor

The SIGSTOP signal would work.

There is no way to disable moving stacks other than stopping the program, though. There can't be: program execution could require additional stack space at any time.

@gianlucaborello
Copy link

FYI I posted a tentative solution that I'm experimenting with to work around this problem, and would appreciate comments and skepticisms: iovisor/bcc#1320 (comment)

Thanks

@Lekensteyn
Copy link
Contributor

Lekensteyn commented Aug 20, 2019

The title and description is misleading. Uprobes do in fact work with Go programs. What does not work are uretprobes.

A uprobe is associated with a userspace program binary and offset. When a probe is added, Linux will load that program, save the old instruction at the absolute file offset and patch it with a trap instruction. When the probe is hit, the eBPF program will be executed in kernel space. This works fine with normal uprobes.

A uretprobe patches a function in a similar way at entry, but it will modify the return address on the stack to a trampoline function. Once hit, the EBPF program is executed and the instruction pointer is modified to the original return address again. If the stack changes, this will likely cause corruption and crashes.

uretprobes should not be used with Go programs.

@seankhliao seankhliao changed the title runtime: ebpf uprobe support runtime: ebpf uretprobe support Jan 24, 2021
@lollipopman
Copy link

Does the possibility of adding uretprobe support to Go improve at all with the new register based calling convention coming with Go 1.17?

@psanford
Copy link

Does the possibility of adding uretprobe support to Go improve at all with the new register based calling convention coming with Go 1.17?

As mentioned above, the issue with uretprobes is that they modify values on the stack with the assumption that the application's runtime doesn't ever move the stack segments. That assumption is incorrect for Go applications. Go 1.17 still has stacks that grow and move, so uretprobes are still unsafe to use.

@lollipopman
Copy link

@psanford thanks for clarification!

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
@BrytonLee
Copy link

The title and description is misleading. Uprobes do in fact work with Go programs. What does not work are uretprobes.

A uprobe is associated with a userspace program binary and offset. When a probe is added, Linux will load that program, save the old instruction at the absolute file offset and patch it with a trap instruction. When the probe is hit, the eBPF program will be executed in kernel space. This works fine with normal uprobes.

A uretprobe patches a function in a similar way at entry, but it will modify the return address on the stack to a trampoline function. Once hit, the EBPF program is executed and the instruction pointer is modified to the original return address again. If the stack changes, this will likely cause corruption and crashes.

Could you give more details about how the corruption/crashes happen after changing stack?

Let's say we have two stacks: s1, and s2. And s1 has been installed an uretprobe (a trampoline) before an activity frame. After changing to s2, would it possible get a wrong return address from s2?

I meant nobody changed anything on s2. Would its behavior just look like disable an uretprobe which is installed on s1?

uretprobes should not be used with Go programs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted OS-Linux
Projects
None yet
Development

No branches or pull requests

10 participants