The samples are there, but the inuse_space default for the memprofile is... unfortunate. It can be changed using the sample_index command.

$ go tool pprof fmt.test mem.profile
File: fmt.test
Type: inuse_space
Time: Mar 19, 2018 at 9:51am (CET)
Entering interactive mode (type "help" for commands, "o" for options)

(pprof) top5       
Showing nodes accounting for 0, 0% of 0 total
      flat  flat%   sum%        cum   cum%

(pprof) sample_index = alloc_space

(pprof) top5
Showing nodes accounting for 11051.10MB, 94.73% of 11666.12MB total
Dropped 22 nodes (cum <= 58.33MB)
Showing top 5 nodes out of 33
      flat  flat%   sum%        cum   cum%
 9026.02MB 77.37% 77.37%  9838.06MB 84.33%  fmt.Sprintf
  676.52MB  5.80% 83.17%  2286.14MB 19.60%  fmt_test.BenchmarkSprintfHexBytes.func1
  655.52MB  5.62% 88.79%   655.52MB  5.62%  fmt_test.BenchmarkFprintfBytes
  359.03MB  3.08% 91.87%   359.03MB  3.08%  reflect.mapiterinit
  334.01MB  2.86% 94.73%  1009.05MB  8.65%  fmt_test.BenchmarkSprintfBytes.func1

I tend to do go tool pprof -alloc_space (but apparently those flags are legacy?)

@hyangah

/cc @adg

Related: google/pprof#182

Could we make pprof require a flag for memory profiles indicating which kind is wanted instead of defaulting to inuse_space (which is almost never what I want)? The default seems to be a source of significant, persistent confusion.

For live server profiling inuse_space is more meaningful than the alloc_space.
Moreover, memory profiling libraries for other languages (e.g. C++) often choose to report only inuse_* equivalent metrics.

For live server profiling inuse_space is more meaningful than the alloc_space.

@hyangah this doesn't match my experience. I run memory profiles on live servers with alloc_space more often than with inuse_space. The most common reason that I run a memory profile is as part of performance optimizations and reducing allocations is a really common optimization. inuse_space is useful for debugging leaks (and, rarely, surprisingly-large-but-not-growing heaps), which I do less often.

This might be different from C++ where it's easier to write memory leaks.

@cespare not only memory leak. inuse_space is one of the first places to look into when optimizing the memory usage in stable state or comparing multiple different instances with different configuration. alloc_space is of course useful for allocation optimization.

For benchmark or test or short running programs, this doesn't make much - why do we care about the inuse_space from the last GC after the end of the program, right? @rauls5382 and @aalexand, is there any way to tell pprof to choose alloc_space over inuse_space for such profile data? For example, what do you think about recording preferred value sample index in the profile, or just dropping inuse_space samples from benchmark or test generated profile out?

I'd settle for a print statement conditioned by no output that suggests trying another index and a way to find out what they are.

@hyangah default_sample_type field in the profile can be used to specify the metric to display by default. See https://github.com/google/pprof/blob/e8d30eee164b66e3e349506f2bd220395cd2c9d1/proto/profile.proto#L82.

To echo the other discussions here, I think inuse_space is more useful for long-running servers than alloc_space, so if the default metric is going to change for anything other than the local "go tool" runs, please let users know in advance. I'd opt against that since alloc_space is currently cumulative since program start which makes it difficult to aggregate across different replicas of a server or over time unless some custom normalization is applied.

Change https://golang.org/cl/101677 mentions this issue: runtime/pprof: comment on sample types in heap profile

Thanks @aalexand
I looked into a way to set the default_sample_type differently for memory profiles produced during test but it is trickier than I hoped. What I can think of are either 1) change runtime/pprof.WriteHeapProfile to accept an option, or 2) use a magical environment variable so if it's set mark alloc_space as the default_sample_type. None of them seems like a minor change to me.

My PR to add a print statement in case of zero sample was merged to upstream. I will try to update the vendored version of pprof as well.

The change 101677 is to additionally embed comments in the generated profile output - they explain what those different measurement types are.

Change https://golang.org/cl/101736 mentions this issue: cmd/vendor/.../pprof: refresh from upstream@a74ae6f

One thing we could do is introduce a new named profile that means "heap allocations for all time" and make it the same as the regular heap profile but with a different default. Then -memprofile can write that new profile out instead of the old one.

Go runtime/pprof.WriteMemProfile is equivalent to runtime/pprof.Lookup("heap").WriteTo, so we'd just have a new name "allocs" or something like that, which is almost exactly the same output, with a different default. Then you can ask for it from running servers too.

/cc @hyangah see what you think of previous comment

SGTM

Change https://golang.org/cl/102696 mentions this issue: runtime/pprof: introduce "allocs" profile