-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime/pprof: missing profile info for CGo code when compiled with CC="zig cc" #59466
Comments
Attaching both binaries for quick reference (this is x86_64 ELF, mind you) |
CC @golang/runtime |
It looks like the profile you got from zig is running |
The second profile looks like it has symbolization wrong. e.g., this doesn't make any sense:
Are you sure that the binary passed to pprof matches the binary used for the profile? In fact, you don't need to pass the binary to pprof at all, as the profile contains the symbol information. Do you get the same results if you drop the binary from the pprof cli? |
I may have messed something up. A fresh honest copy-paste now, single command, no previous state: zig
clang-15
I get an interactive pprof shell:
|
In the new one the call stack also doesn't make much sense. Maybe the symbolization is completely broken. And you probably run Do you know how zig's linker is implemented? Does it wrap a C linker, or it has its own implementation? Is there anything special? (I sent CL https://golang.org/cl/482975 to fix the C compiler warning.) |
On linux it wraps clang for compiling and ld.lld for linking, adds a few flags. One can see the exact lld invocations with cc @andrewrk
Thanks! |
Hmmm, the symbolization definitely is wrong. I ran the zig binary you pasted above. I got a profile, which contains a sample like
But that PC is actually corresponding to
which looks correct. So it seems the symbolization collected at run time by the profiler is wrong. I wonder if you use zig to link a cgo program that just panics, does it print the right panic stack trace? The zig linking is definitely somewhat unusual. For a quick look, it puts the |
Well, this is interesting: with the profile I have, I got the similar incorrect symbolization
Rename the binary, so the pprof command cannot find it from the memory mapping, then we get the correct symbolization! (It cannot symbolize C functions because it cannot find the binary's symbol table, but at least the Go functions are correct.)
So the mapping and the binary actually confused the pprof command. (The symbol information collected at run time is actually fine.) Looking at the binary, this is the
Note that the 4th segment,
It has alignment 0x1000, so the address and offset should be aligned to 0x1000. But it doesn't. This leads to this interesting memory mapping at run time: (from GDB)
The second one is the text segment. The start address of the segment in the binary is 0x337220, but the mapping starts at 0x337000 due to misalignment. I think this confused the pprof command, causing a shift of 0x220 in address. In fact, if we shift the PCs by 0x220, we get the wrong symbolization:
The second one is the first one +0x220. The profile incorrectly shows |
Change https://go.dev/cl/483035 mentions this issue: |
Wow, thanks for the investigation! I was able to replicate the zig's issue with clang+lld too:
I can confirm your fix fixed the original issue, with both Now looking at the different linker:
This is what lld does. Here is a quick example: lld
gnu ld
As you can observe above, I will create two follow-up issues soon:
|
TLDR: when profiling CGo programs compiled with
zig cc
, we are missing the profiles of pure C functions as observed withgo tool pprof
.I work with the C++ toolchain at Uber and we recently switched to using
zig cc
to compile our CGo programs. I have not yet heard complains internally ofmissing CGo profiles, but I suspect it's a matter of time. :)
Also, this seems to be the last suite of tests that fail when running
CC="zig cc" ./all.bash
on Linux amd64.Go version: master (8d68b38)
Zig version: 0.10.1 and master (ziglang/zig@8853005)
Steps to reproduce
Compile
testprogcgo
withzig cc
andclang-15
:Run both resulting binaries:
Compare the profiles:
Note that
cpuHog
andcpuHog2
are not present in the profile of the program that was compiled withzig cc
. This is what the unit testTestCgoPprof
asserts.Background / steps taken
I have observed and compared the resulting compiler and linker flags from both builds did not observe anything standing out.
I would appreciate some help debugging this. I am somewhat comfortable looking at disassembly and the low-level compiler flags, but, in this case, I don't know where to look.
The text was updated successfully, but these errors were encountered: