cmd/go: load packages in parallel #29758
Labels
FrozenDueToAge
GoCommand
cmd/go
NeedsFix
The path to resolution is known, but the work has not been done.
Performance
ToolSpeed
Milestone
Comments
|
cc @bcmills |
|
Also just to note https://go-review.googlesource.com/c/tools/+/158097. Chatting with @heschik, ideally So hopefully this change will go a good step towards that goal. |
bcmills
added
the
NeedsFix
|
Change https://golang.org/cl/161397 mentions this issue: |
|
Change https://golang.org/cl/167748 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Apr 1, 2019
LoadPackage was used to load a *load.Package for a command line argument, after pattern expansion. It provided two special cases on top of LoadImport. First, it ensured that "cmd/" packages in GOROOT were installed in "$GOROOT/bin" or "$GOROOT/pkg/tool". Second, it translated absolute paths to packages in GOROOT and GOPATH into regular import paths. With this change, LoadImport now ensures "cmd/" packages have the right Target (without the need for a special case) and search.ImportPaths translates absolute paths. LoadPackage no longer handles these special cases and has been renamed to LoadImportWithFlags, since it's still useful for loading implicit dependencies. Updates #29758 Change-Id: I9d54036f90c3ccd9b3a0fe0eaddaa7749593cc91 Reviewed-on: https://go-review.googlesource.com/c/go/+/167748 Run-TryBot: Jay Conrod <jayconrod@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Currently, when the Go command loads a set of packages and their dependencies, it does so on a single goroutine. Tracing shows that most cores are idle when loading packages for a large build, so there's significant performance opportunity here. On the one core that's busy, most of the time is spent in
internal/load.LoadImportandinternal/load.Package.load(which are mutually recursive).One may expect the package loading process to be dominated by I/O, but the files loaded are usually in the kernel file cache, and we do a lot of work in userspace to parse files, extract imports, and apply build constraints. We actually spend very little time in the kernel, at least on Linux (system call overhead is very low).
Based on this information, it seems that we can significantly speed up the package loading process by parallelizing it. This will benefit
go listin particular, which is heavily used bygolang.org/x/tools/go/packages.go buildand other commands may see some improvement, too.The text was updated successfully, but these errors were encountered: