runtime/pprof: generate profiles in gzipped proto format #16093

matloob · 2016-06-17T15:23:28Z

[edit (3 Oct 2016) we now propose to just return the profiles in the serialized proto format. The proposal previously also gzipped the proto]

I propose to add a function
func runtime/pprof.StartCPUProfileProto(w io.Writer) error
and support in the runtime, for generating profiles in serialized profile protocol buffers, rather than in the legacy pprof format.

The profile proto is more extensible than the legacy pprof format and is pprof's preferred profile format. The extensibility will be useful to support generating profiles with labels. pprof supports rendering labeled profiles, but labels can only appear in the proto profile, not in the legacy format.

The profile proto format is already understood by pprof itself and pprof already includes an encoder for producing serialized profile protos without requiring a dependency on the proto package or library.

Here's the pprof proto description: https://github.com/google/pprof/blob/master/proto/profile.proto
and here's the encoder: https://github.com/google/pprof/blob/master/profile/proto.go
For more about labels, take a look at the https://github.com/google/pprof/blob/master/doc/developer/profile.proto.md#labels

The text was updated successfully, but these errors were encountered:

hyangah · 2016-06-17T16:51:33Z

@rsc @rauls5382 @aclements

rauls5382 · 2016-06-17T17:18:54Z

I agree the runtime should generate profile.proto directly. I would further propose to deprecate the legacy pprof formats.

For pprof to retrieve profiles in profile.proto format, the net/http/pprof package would also have to be changed to serve the profiles in that format.

aclements · 2016-06-17T18:17:20Z

I definitely agree that the runtime should generate profiles in the protocol buffer format. However, I don't think you need to add a new API: just replace the legacy format with the protobuf format. My understanding is that that was the idea from the start with the protobuf format. I believe @rsc even prototyped part of the runtime support just to make sure we weren't backing ourselves into a corner.

matloob · 2016-06-17T20:09:20Z

Oh, that's even better!

DarKol13 · 2016-07-04T09:56:23Z

I'm interested in the switch to the new format too and could participate in it, I don't want to duplicate efforts, so, could you tell, is any job done?

matloob · 2016-08-11T15:12:23Z

@DarKol13 I haven't started any work yet. Feel free to contribute. Is there any part in particular you're interested in?

DarKol13 · 2016-08-19T11:26:10Z

@matloob How can I contact you to discuss it in real-time?

matloob · 2016-08-19T11:38:57Z

Send me an email off thread to (github username) @golang.org and we can
coordinate there.
On Fri, Aug 19, 2016 at 7:27 AM DarKol13 notifications@github.com wrote:

@matloob https://github.com/matloob How can I contact you to discuss it
in real-time?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#16093 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/APtQJc-AhgOkLrygbNSPrU3pkzHtngLuks5qhZMRgaJpZM4I4dy7
.

rhysh · 2016-08-26T23:36:43Z

What are your plans for how the information included in the profiles will change?

Will they include full symbol info (function, file, line), or none at all?
Will they set the build_id field .. and will that be the contents of .note.gnu.build-id if present, or the Go toolchain's build id?
CPU profiles on linux now include the contents of /proc/self/maps. Will this continue, and extend to non-CPU profiles as well?
Where will labels be used?

matloob · 2016-09-01T13:55:58Z

@rhysh We will not drop any of the information currently included in the profile.

We're going to start off using some code we've open-sourced from Google that converts from the legacy format to the proto format. You can take a look at in patch set 1 of golang.org/cl/28349. See package internal/protopprof.

Profiles will include symbol data.
I don't think we've been setting that field in the newly open sourced code, but I don't see why not.
It will continue. I don't see why it shouldn't extend to non-CPU profiles, but maybe someone else can comment on that.
We're going to have to add some mechanism to the runtime for providing label annotations, but I don't know how that will work. I'm working on a proposal, but it's very much open for now. Once the facility is there users will be able to attach additional annotations to profile samples. The go pprof command already knows how to display them. When you get a graphical profile, little "3-d" boxes will appear coming out with each label. Here's a picture I grabbed: https://goo.gl/photos/5WWKPCpXFsyWDGRN7

robpike · 2016-09-01T21:19:48Z

You're going to put gzip very low in the dependency chain, which doesn't strike me as a good idea. Can you do this without gzip?

ianlancetaylor · 2016-09-01T21:48:38Z

Or perhaps just the gzip part could most to net/http/pprof where it is most useful.

matloob · 2016-09-01T21:49:52Z

Yes, we can! pprof understands the ungzipped profiles.

I just talked to @rauls5382 who had the same idea as @ianlancetaylor: runtime/pprof returns uncompressed protos, and the http handler gzips it.

adg · 2016-10-03T22:43:35Z

I'd like to see how much code this involves. The proto seems non-trivial, and so the encoder is likely non-trivial too, but I'm prepared to be pleasantly surprised.

robpike · 2016-10-04T00:08:22Z

It would be nice to split this into two separate proposals: 1) switching to protobuf format. 2) gzipping the result.

matloob · 2016-10-04T00:25:49Z

@adg Most of the code for working with the proto is already in the go tree. For example the core of the encoder code is here: https://golang.org/src/cmd/internal/pprof/profile/proto.go

@robpike Renaming this proposal now. It'll now just be (1).

gopherbot · 2016-10-07T16:10:36Z

CL https://golang.org/cl/30556 mentions this issue.

rsc · 2016-10-18T16:00:04Z

On-disk pprof profiles are defined to be gzipped, never ungzipped. We were very clear about this when we wrote the spec for them. If pprof accepts ungzipped, that's arguably a bug in pprof. Go should only ever generate the gzipped form.

That said, I don't think putting a gzip dependency in runtime/pprof is too big a problem: runtime/pprof (the package you import to actually get the profiles out of the runtime) is not a low-level package. I would be concerned if runtime itself grew a gzip dependency, but runtime/pprof is OK.

rsc · 2016-10-18T16:00:20Z

Sounds like the proposal was accepted earlier in the thread; just adding the label to reflect that.

gopherbot · 2016-10-28T19:15:27Z

CL https://golang.org/cl/32257 mentions this issue.

Original Change by Daria Kolistratova <daria.kolistratova@intel.com> Added functions with suffix proto and stuff from pprof tool to translate to protobuf. Done as the profile proto is more extensible than the legacy pprof format and is pprof's preferred profile format. Large part was taken from https://github.com/google/pprof tool. Tested by hand and compared the result with translated by pprof tool, profiles are identical. Fixes #16093 Change-Id: I2751345b09a66ee2b6aa64be76cba4cd1c326aa6 Reviewed-on: https://go-review.googlesource.com/32257 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alan Donovan <adonovan@google.com>

gopherbot · 2016-10-30T14:05:23Z

CL https://golang.org/cl/32353 mentions this issue.

gopherbot · 2016-11-04T17:00:55Z

CL https://golang.org/cl/32811 mentions this issue.

This change adds code, originally written by Russ Cox <rsc@golang.org> and open-sourced by Google, that converts from the "legacy" binary pprof profile format to a struct representation of the new protocol buffer pprof profile format. This code reads the entire binary format for conversion to the protobuf format. In a future change, we will update the code to incrementally read and convert segments of the binary format, so that the entire profile does not need to be stored in memory. This change also contains contributions by Daria Kolistratova <daria.kolistratova@intel.com> from the rolled-back change golang.org/cl/30556 adapting the code to be used by the package runtime/pprof. This code also appeared in the change golang.org/cl/32257, which was based on Daria Kolistratova's change, but was also rolled back. Updates #16093 Change-Id: I5c768b1134bc15408d80a3ccc7ed867db9a1c63d Reviewed-on: https://go-review.googlesource.com/32811 Run-TryBot: Michael Matloob <matloob@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Russ Cox <rsc@golang.org>

gopherbot · 2016-11-10T18:40:23Z

CL https://golang.org/cl/33071 mentions this issue.

gopherbot · 2016-11-21T17:20:25Z

CL https://golang.org/cl/33422 mentions this issue.

When debug is 0, emit the compressed proto format. The debug>0 format stays the same. Updates #16093 Change-Id: I45aa1874a22d34cf44dd4aa78bbff9302381cb34 Reviewed-on: https://go-review.googlesource.com/33422 Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>

matloob added the Proposal label Jun 17, 2016

matloob changed the title ~~proposal: runtime: Also generate profiles in gzipped proto format~~ proposal: runtime: Generate profiles in gzipped proto format Jun 17, 2016

quentinmit added this to the Go1.8 milestone Jul 29, 2016

aclements mentioned this issue Aug 30, 2016

runtime: additional memory profiler report modes or similar #13463

Open

matloob changed the title ~~proposal: runtime: Generate profiles in gzipped proto format~~ proposal: runtime: Generate profiles in proto format Oct 4, 2016

rsc changed the title ~~proposal: runtime: Generate profiles in proto format~~ runtime/pprof: generate profiles in gzipped proto format Oct 18, 2016

rsc added the Proposal-Accepted label Oct 18, 2016

gopherbot closed this as completed in 7d14401 Oct 28, 2016

matloob self-assigned this Nov 3, 2016

matloob reopened this Nov 3, 2016

gopherbot closed this as completed in 76f12cd Nov 10, 2016

rsc removed the Proposal label Nov 22, 2016

rsc mentioned this issue Nov 22, 2016

runtime/pprof: generate binary profile formats by default #17764

Closed

golang locked and limited conversation to collaborators Nov 22, 2017

gopherbot added the FrozenDueToAge label Nov 22, 2017

rsc unassigned matloob Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime/pprof: generate profiles in gzipped proto format #16093

runtime/pprof: generate profiles in gzipped proto format #16093

matloob commented Jun 17, 2016 •

edited

hyangah commented Jun 17, 2016

rauls5382 commented Jun 17, 2016

aclements commented Jun 17, 2016

matloob commented Jun 17, 2016

DarKol13 commented Jul 4, 2016

matloob commented Aug 11, 2016

DarKol13 commented Aug 19, 2016

matloob commented Aug 19, 2016

rhysh commented Aug 26, 2016

matloob commented Sep 1, 2016

robpike commented Sep 1, 2016

ianlancetaylor commented Sep 1, 2016

matloob commented Sep 1, 2016

adg commented Oct 3, 2016

robpike commented Oct 4, 2016

matloob commented Oct 4, 2016

gopherbot commented Oct 7, 2016

rsc commented Oct 18, 2016

rsc commented Oct 18, 2016

gopherbot commented Oct 28, 2016

gopherbot commented Oct 30, 2016

gopherbot commented Nov 4, 2016

gopherbot commented Nov 10, 2016

gopherbot commented Nov 21, 2016

runtime/pprof: generate profiles in gzipped proto format #16093

runtime/pprof: generate profiles in gzipped proto format #16093

Comments

matloob commented Jun 17, 2016 • edited

hyangah commented Jun 17, 2016

rauls5382 commented Jun 17, 2016

aclements commented Jun 17, 2016

matloob commented Jun 17, 2016

DarKol13 commented Jul 4, 2016

matloob commented Aug 11, 2016

DarKol13 commented Aug 19, 2016

matloob commented Aug 19, 2016

rhysh commented Aug 26, 2016

matloob commented Sep 1, 2016

robpike commented Sep 1, 2016

ianlancetaylor commented Sep 1, 2016

matloob commented Sep 1, 2016

adg commented Oct 3, 2016

robpike commented Oct 4, 2016

matloob commented Oct 4, 2016

gopherbot commented Oct 7, 2016

rsc commented Oct 18, 2016

rsc commented Oct 18, 2016

gopherbot commented Oct 28, 2016

gopherbot commented Oct 30, 2016

gopherbot commented Nov 4, 2016

gopherbot commented Nov 10, 2016

gopherbot commented Nov 21, 2016

matloob commented Jun 17, 2016 •

edited