-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/pprof: Local symbolization failed for http.test: open /tmp/go-build348974326/b001/http.test: no such file or directory #25743
Comments
I don't think this is a new error but became more noticeable because of the new terminal support (with color). I happened to ask the same question in google/pprof#386 today. The error was from https://github.com/google/pprof/blob/master/internal/symbolizer/symbolizer.go#L343 Should we set these values based on the collected samples? (if all pcs from the mapping were successfully symbolized, set those fields) |
Setting those fields may prevent locations in the C++ code from being symbolized, there is discussion on b/36645883 which I believe relevant to this. |
Sorry, I missed that you already mentioned what I said. Setting the flags only if all pcs are symbolized sounds reasonable. |
Change https://golang.org/cl/118275 mentions this issue: |
I sent cl/118275 for review. I am not sure whether it's for 1.11. |
The pprof tool utilizes attributes of mapping entries such as HasFunctions to determine whether the profile includes necessary symbol information. If none of the attributes is set, pprof tool tries to read the corresponding binary to use for local symbolization. If the binary doesn't exist, it prints out error messages. Go runtime generated profiles without any of the attributes set so the pprof tool always printed out the error messages. The error messages became more obvious with the new terminal support that uses red color for error messages. Go runtime can symbolize all Go symbols and generate self-contained profile for pure Go program. Thus, there is no reason for the pprof tool to look for the copy of the binary. So, this CL sets one of the attributes (HasFunctions) true if all PCs in samples look fully symbolized. For non-pure Go program, however, it's possible that symbolization of non-Go PCs is incomplete. In this case, we need to leave the attributes all false so pprof can attempt to symbolize using the local copy of the binary if available. It's hard to determine whether a mapping includes non-Go code. Instead, this CL checks PCs from collected samples. If unsuccessful symbolization is observed, it skips setting the HasFunctions attribute. Fixes golang#25743 Change-Id: I5108be45bbc37ab486d145fa03e7ce37d88fad50 Reviewed-on: https://go-review.googlesource.com/118275 Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Change https://golang.org/cl/123779 mentions this issue: |
Profile's Mapping field is currently populated by reading /proc/self/maps. On systems where /proc/self/maps is not available, the profile generated by Go's runtime will not have any Mapping entry. Pprof command then adds a fake entry and links all Location entries in the profile with the fake entry to be used during symbolization. https://github.com/google/pprof/blob/a8644067d5a3c9a6386e7c88fa4a3d9d37877ca3/internal/driver/fetch.go#L437 The fake entry is not enough to suppress the error or warning messages pprof command produces. We need to tell pprof that Location entries are symbolized already by Go runtime and pprof does not have to attempt to perform further symbolization. In #25743, we made Go runtime mark Mapping entries with HasFunctions=true when all Location entries from the Mapping entries are successfully symbolized. This change makes the Go runtime add a fake mapping entry, otherwise the pprof command tool would add, and set the HasFunctions=true following the same logic taken when the real mapping information is available. Updates #19790. Fixes #26255. Tested pprof doesn't report the error message any more for pure Go program. Change-Id: Ib12b62e15073f5d6c80967e44b3e8709277c11bd Reviewed-on: https://go-review.googlesource.com/123779 Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Note: If anybody is getting this error and wants to suppress it, you can add |
@jsha That is only when the remote server (the profile origin) or any symbolization service the pprof can utilize is available. |
Thanks @hyangah! Maybe what I'm actually looking for for a quick fix is I'm seeing the same error described here, in Do you think that qualifies enough as "still reproduces" that I should file a separate bug? Or is that working as intended? |
I think that's working as intended. On the other hand, I wonder how -symbolize=none makes pprof symbolize non-Go symbols. |
I was just profiling something like normal and pprof gave me some angry red text about something not working, but it seems to work. I'm not sure what's broken. @hyangah?
The text was updated successfully, but these errors were encountered: