Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: runtime/pprof: option to profile without symbolization #22294

Closed
hyangah opened this issue Oct 16, 2017 · 9 comments
Closed

proposal: runtime/pprof: option to profile without symbolization #22294

hyangah opened this issue Oct 16, 2017 · 9 comments

Comments

@hyangah
Copy link
Contributor

hyangah commented Oct 16, 2017

We propose to extend runtime/pprof API to allow users to choose whether to
include symbol info in the profile output.

Background

Since Go1.9, profile outputs include symbol information
(https://golang.org/doc/go1.9#go-tool-pprof) so they can be interpreted
without the binary. Symbolization is done by the profiled binary and we
observed in some cases, such symbolization work in the profiled target
binary is not desirable.

For example,

  • we may want to avoid adding extra load to a busy server when
    debugging CPU overload issue.
  • symbol information may be available in different sources (binary, external
    symbolization service, etc) and users may favor to use them instead.
  • currently runtime delegates symbolization of non-Go symbols to cgoSymbolizer and its efficiency
    is sometimes not easy to control.

The pprof tool still supports retrieving symbol information from external sources
when it finds the fetched profile lacks symbol information.

API changes

runtime/pprof: In the scope of this proposal we care only about controlling
symbolization, but one can imagine that we may want to support more customization
in profiling (for example, filtering, choosing different output formats, debug levels,
changing CPU profiling rate, etc). In order to support the future extension easily, we
propose to add profile parameter types.

Then, StartCPUProfile and Profile.WriteTo are variations that take those new parameter
types.

package pprof 
// CPUProfileParam is a set of parameters to configure CPU profiling.
type CPUProfileParam struct {
  ProfileParam
  // Place for future CPU profile-specific parameters
}

// ProfileParam is a set of parameters to configure profile output generation.
type ProfileParam struct {
  NoSymbol bool
}

// StartCPUProfileWithParam is like StartCPUProfile but it allow to configure
// profiling and its output generation.
func StartCPUProfileWithParam(w io.Writer, param CPUProfileParam) error

// WriteToWithParam is like WriteTo, but it allows to configure profile output generation.
func (* Profile) WriteToWithParam(w io.Writer, param ProfileParam) error

net/http/pprof:

Profile handlers can take a new 'nosymbol' form value. If it is set to non-zero integer
value, profile output will not include symbol information.

There will be no change affecting existing API users.

Implementation

Implementation is trivial - we skip symbolization that had been done while formatting
the output. The pprof tool can perform symbolization correctly as long as the Location
entries of the output proto contain correct Address and Mapping information.

One caveat is that currently expansion of inlined frames are also handled by
runtime.CallersFrames used for symbolization. There is no way to perform expansion
without symbolization. Once the correct DWARF debug information for inlined frames
work (in progress targetting for 1.10) is done, the DWARF reader can expand them
using the PCs.

@gopherbot
Copy link

Change https://golang.org/cl/70870 mentions this issue: runtime/pprof: allow cpu profiling without symbolization

@ianlancetaylor ianlancetaylor changed the title runtime/pprof: option to profile without symbolization proposal: runtime/pprof: option to profile without symbolization Oct 16, 2017
@gopherbot gopherbot added this to the Proposal milestone Oct 16, 2017
@aalexand
Copy link
Contributor

@hyangah https://golang.org/cl/70870 does not yet add WriteToWithParam, is that to be added in that CL?

@aalexand
Copy link
Contributor

@hyangah I can see now the CL says there will be a separate CL.

@rsc
Copy link
Contributor

rsc commented Oct 31, 2017

I would rather just always symbolize and not worry about this. If there are efficiency bugs let's fix those separately.

@hyangah
Copy link
Contributor Author

hyangah commented Oct 31, 2017

@rsc The efficiency issue we are concerning about is more in the foreign language side called by cgoSymbolizer. It will be much easier and more accessible for users if an option to delegate symbolization to pprof is given than requiring them to optimize their C++ or other foreign language libraries sufficiently so not to interfere with Go's profiling code.

When the server is already over-loaded, I'd rather want to minimize work on the server side, and reduce the chance of failed profile collection due to extra operations that can be done offline.

@gopherbot
Copy link

Change https://golang.org/cl/76211 mentions this issue: runtime: new Mode field in cgo symbolizer function arg

@rsc
Copy link
Contributor

rsc commented Nov 6, 2017

Per discussion on CL, we'd like to fix any performance problems before resorting to another mode and recreating the old "you have a meaningless profile because it has no symbol info and your binary is gone" failure mode.

@hyangah
Copy link
Contributor Author

hyangah commented Nov 6, 2017

The pprof tool performs symbolization immediately after collecting the profile from a running server and tries its best to produce a profile output with symbol info in it. So, the output is always carrying the symbol lookup if the server's symbolz (debug/pprof/symbol) responds to the symbol lookup requests.

I still feel that we need to distinguish the case where we get the profile using the 'go test' or through the current runtime/pprof API, from the case where we get the profile from the live server.

Unfortunately, I just noticed that the symbolz or debug/pprof/symbol handler is implemented with runtime.FuncForPC which is incomplete compared to runtime.CallersFrames (missing inline function info, and also not retrieving non-go symbols).

@rsc
Copy link
Contributor

rsc commented Nov 20, 2017

@hyangah, have you been able to optimized the C++ symbol lookup instead? Can we close this issue?

@hyangah hyangah closed this as completed Nov 20, 2017
@golang golang locked and limited conversation to collaborators Nov 20, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants