Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proposal: runtime/pprof: unused/duplicate/opportunity heap profile #66744

Open
dvyukov opened this issue Apr 9, 2024 · 4 comments
Open

proposal: runtime/pprof: unused/duplicate/opportunity heap profile #66744

dvyukov opened this issue Apr 9, 2024 · 4 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Proposal
Milestone
Proposal

Comments

@dvyukov
Copy link
Member

dvyukov commented Apr 9, 2024

Proposal Details

There are some common "mistakes" (or missed opportunities) that lead to increased heap size:

  • reading in a large chunk of memory and then referencing just some bits of it via []byte/string (subject to detaching)
  • producing lots of duplicate strings after reading in some data (subject to interning)
  • alignment padding in structs (subject to field reordering)
  • maybe we can find more opportunities

Potentially runtime could do an expensive analysis (for simplicity can start with STW) and dump a profile with these missed opportunities. Namely, run heap marking on intra-object level (mark/scan only reachable parts of objects); run full heap duplication analysis (hash by object size/type/contents) and then cross this with heap profile data and dump intersection.

This may provide a very cost-efficient way to optimize heap size. E.g. this stack allocated N GB that are 99% unreachable, or this stack allocated M GB of duplicate strings, etc.
@gopherbot gopherbot added this to the Proposal milestone Apr 9, 2024
@dvyukov
Copy link
Member Author

dvyukov commented Apr 9, 2024

cc @golang/runtime @prattmic @felixge sorry if this was proposed/discussed already, I did not follow development lately

@ianlancetaylor ianlancetaylor added the compiler/runtime Issues related to the Go compiler and/or runtime. label Apr 9, 2024
@nsrip-dd
Copy link
Contributor

This sounds like a good use case for gocore (see #57447).

@dvyukov
Copy link
Member Author

dvyukov commented Apr 10, 2024

Agree, I guess it will make things easier to implement.

@rhysh
Copy link
Contributor

rhysh commented Apr 10, 2024

I'd be uneasy about exposing something so expensive to collect (including a STW that's proportional to something other than the number of Ps) as a runtime/pprof.Profile.

Plus 1 on offline analysis via gocore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Proposal
Projects
Go Compiler / Runtime
Status: No status
Proposals
Status: Incoming
Milestone
Proposal
Development

No branches or pull requests
5 participants
@rhysh @dvyukov @ianlancetaylor @gopherbot @nsrip-dd