cmd/go: store package metadata in build cache #31417
Labels
FrozenDueToAge
GoCommand
cmd/go
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
ToolSpeed
Milestone
The Go command currently spends a lot of time gathering package metadata using
go/build.Import
andcmd/go/internal/imports.ScanDir
. Both functions read and parse .go files in directories in order to gather imports and evaluate build constraints. This is expensive both in I/O and CPU time. Source files rarely change, especially in the module cache.We should investigate saving the package metadata we gather in the build cache. The cache key for a package would be derived from
readdir
on a package directory andstat
on each file in the directory. If a file has changed, thestat
metadata will almost certainly change as well: the modification time will be different, probably also the size.We should take care to avoid hazards with mtime comparison. In particular, we should avoid caching stat data that is only a few seconds old because some file systems have low time resolution, and there is a window where several changes can be made without changing the recorded mtime. We may want to enable this only on certain file systems, operating systems, or only for the module cache.
An initial prototype (CL 75295) shows we can reduce
go list
time by roughly 20%. That may not be a big enough difference to justify added complexity, but profiling shows that more improvement may be possible if we use the cache for module loading and optimize deserialization.The text was updated successfully, but these errors were encountered: