Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/go/gcexportdata: reads entire object file, not just the needed type information #48954

Closed
Tracked by #1100
dominikh opened this issue Oct 13, 2021 · 1 comment
Closed
Tracked by #1100

x/tools/go/gcexportdata: reads entire object file, not just the needed type information #48954

dominikh opened this issue Oct 13, 2021 · 1 comment
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
Unreleased

Comments

@dominikh
Copy link
Member

When using gcimportdata.NewReader and gcimportdata.Read, the entire object file will be read into memory, not just the portion containing type information. For packages that have large variable initializers, or that use embed, not doing this could save almost 100% of memory usage and I/O.

https://play.golang.org/p/kAXdktA3WTD contains an example that loads the archive for https://play.golang.org/p/MvxM7Jlxjhk – a package that embeds a 100 MB file. It demonstrates how much data gets read unnecessarily.

This should be fairly straightforward to fix. The necessary length information exists, gcimporter.FindExportData just doesn't expose it. If it did, we could use an io.LimitedReader.

A quickly hacked together patch drops the read amount from 100 MB to 100 bytes.

I'd be happy to send a change, but I'm not sure how the development between x/tools's gcimporter and the Go compiler's gcimporter get synchronized.

/cc @mdempsky because I'm curious how this interacts with his lazy loading work.
@dominikh dominikh mentioned this issue Oct 13, 2021
Closed
2 tasks
@seankhliao seankhliao changed the title golang.org/x/tools/go/gcexportdata: reads entire object file, not just the needed type information x/tools/go/gcexportdata: reads entire object file, not just the needed type information Oct 14, 2021
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Oct 14, 2021
@gopherbot gopherbot added this to the Unreleased milestone Oct 14, 2021
@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 15, 2021
@gopherbot
Copy link

Change https://golang.org/cl/357209 mentions this issue: go/gcexportdata: limit reader to the export data section of the archive

@golang golang locked and limited conversation to collaborators Oct 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@toothrot @dominikh @gopherbot