-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: runtime/pprof: option to profile without symbolization #22294
Comments
Change https://golang.org/cl/70870 mentions this issue: |
@hyangah https://golang.org/cl/70870 does not yet add WriteToWithParam, is that to be added in that CL? |
@hyangah I can see now the CL says there will be a separate CL. |
I would rather just always symbolize and not worry about this. If there are efficiency bugs let's fix those separately. |
@rsc The efficiency issue we are concerning about is more in the foreign language side called by cgoSymbolizer. It will be much easier and more accessible for users if an option to delegate symbolization to pprof is given than requiring them to optimize their C++ or other foreign language libraries sufficiently so not to interfere with Go's profiling code. When the server is already over-loaded, I'd rather want to minimize work on the server side, and reduce the chance of failed profile collection due to extra operations that can be done offline. |
Change https://golang.org/cl/76211 mentions this issue: |
Per discussion on CL, we'd like to fix any performance problems before resorting to another mode and recreating the old "you have a meaningless profile because it has no symbol info and your binary is gone" failure mode. |
The pprof tool performs symbolization immediately after collecting the profile from a running server and tries its best to produce a profile output with symbol info in it. So, the output is always carrying the symbol lookup if the server's symbolz (debug/pprof/symbol) responds to the symbol lookup requests. I still feel that we need to distinguish the case where we get the profile using the 'go test' or through the current runtime/pprof API, from the case where we get the profile from the live server. Unfortunately, I just noticed that the symbolz or debug/pprof/symbol handler is implemented with runtime.FuncForPC which is incomplete compared to runtime.CallersFrames (missing inline function info, and also not retrieving non-go symbols). |
@hyangah, have you been able to optimized the C++ symbol lookup instead? Can we close this issue? |
We propose to extend runtime/pprof API to allow users to choose whether to
include symbol info in the profile output.
Background
Since Go1.9, profile outputs include symbol information
(https://golang.org/doc/go1.9#go-tool-pprof) so they can be interpreted
without the binary. Symbolization is done by the profiled binary and we
observed in some cases, such symbolization work in the profiled target
binary is not desirable.
For example,
debugging CPU overload issue.
symbolization service, etc) and users may favor to use them instead.
is sometimes not easy to control.
The pprof tool still supports retrieving symbol information from external sources
when it finds the fetched profile lacks symbol information.
API changes
runtime/pprof: In the scope of this proposal we care only about controlling
symbolization, but one can imagine that we may want to support more customization
in profiling (for example, filtering, choosing different output formats, debug levels,
changing CPU profiling rate, etc). In order to support the future extension easily, we
propose to add profile parameter types.
Then, StartCPUProfile and Profile.WriteTo are variations that take those new parameter
types.
net/http/pprof:
Profile handlers can take a new 'nosymbol' form value. If it is set to non-zero integer
value, profile output will not include symbol information.
There will be no change affecting existing API users.
Implementation
Implementation is trivial - we skip symbolization that had been done while formatting
the output. The pprof tool can perform symbolization correctly as long as the Location
entries of the output proto contain correct Address and Mapping information.
One caveat is that currently expansion of inlined frames are also handled by
runtime.CallersFrames used for symbolization. There is no way to perform expansion
without symbolization. Once the correct DWARF debug information for inlined frames
work (in progress targetting for 1.10) is done, the DWARF reader can expand them
using the PCs.
The text was updated successfully, but these errors were encountered: