Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: store package metadata in build cache #31417

Closed
jayconrod opened this issue Apr 11, 2019 · 1 comment
Closed

cmd/go: store package metadata in build cache #31417

jayconrod opened this issue Apr 11, 2019 · 1 comment
Assignees
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Milestone

Comments

@jayconrod
Copy link
Contributor

The Go command currently spends a lot of time gathering package metadata using go/build.Import and cmd/go/internal/imports.ScanDir. Both functions read and parse .go files in directories in order to gather imports and evaluate build constraints. This is expensive both in I/O and CPU time. Source files rarely change, especially in the module cache.

We should investigate saving the package metadata we gather in the build cache. The cache key for a package would be derived from readdir on a package directory and stat on each file in the directory. If a file has changed, the stat metadata will almost certainly change as well: the modification time will be different, probably also the size.

We should take care to avoid hazards with mtime comparison. In particular, we should avoid caching stat data that is only a few seconds old because some file systems have low time resolution, and there is a window where several changes can be made without changing the recorded mtime. We may want to enable this only on certain file systems, operating systems, or only for the module cache.

An initial prototype (CL 75295) shows we can reduce go list time by roughly 20%. That may not be a big enough difference to justify added complexity, but profiling shows that more improvement may be possible if we use the cache for module loading and optimize deserialization.

@jayconrod jayconrod added this to the Go1.13 milestone Apr 11, 2019
@jayconrod jayconrod self-assigned this Apr 11, 2019
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@jayconrod jayconrod removed their assignment Oct 14, 2021
@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 15, 2021
@bcmills
Copy link
Contributor

bcmills commented Jul 1, 2022

Oh, hey, @matloob implemented this for Go 1.19 in CL 413815! 😅

@bcmills bcmills closed this as completed Jul 1, 2022
@bcmills bcmills modified the milestones: Backlog, Go1.19 Jul 1, 2022
@golang golang locked and limited conversation to collaborators Jul 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Projects
None yet
Development

No branches or pull requests

7 participants