cmd/go: module graph pruning and lazy module loading #36460

bcmills · 2020-01-08T17:02:11Z

I propose to change cmd/go to avoid loading transitive module dependencies
that have no observable effect on the packages to be built.

The key insights that lead to this approach are:

If no package in a given dependency module is ever (even transitively)
imported by any package loaded by an invocation of the go command, then an
incompatibility between any package in that dependency and any other package
has no observable effect in the resulting program(s). Therefore, we can
safely ignore the (transitive) requirements of any module that does not
contribute any package to the build.
We can use the explicit requirements of the main module as a coarse filter
on the set of modules relevant to the main module and to previous
invocations of the go command.

Based on those insights, I propose to change the go command to retain more
transitive dependencies in go.mod files and to avoid loading go.mod files
for “irrelevant” modules, while still maintaining high reproducibility for build
and test operations.

Design doc: http://golang.org/design/36460-lazy-module-loading.

The text was updated successfully, but these errors were encountered:

gopherbot · 2020-02-03T18:46:05Z

Change https://golang.org/cl/217557 mentions this issue: cmd/go: add a regression test for package import cycles guarded by build tags

gopherbot · 2020-02-19T19:45:09Z

Change https://golang.org/cl/220080 mentions this issue: design/36460: add design for lazy module loading

bcmills · 2020-02-19T19:54:17Z

A detailed design is in CL 220080.

(You can view the rendered Markdown by opening the .md file at the latest patch set and clicking the gitiles link.)

ChrisHines · 2020-02-20T18:02:36Z

For modules that are tidy:

The module versions recorded in the go.mod file would be exactly those listed in vendor/modules.txt, if present.

👍 I think that is a nice property to have. I have some projects for which go mod vendor is sane, but go mod tidy updates the go.mod file in undesirable ways due to test-of-test dependencies. If I understand this proposal correctly that problem would go away.

bcmills · 2020-02-20T21:28:17Z

I added a few examples (in our usual cmd/go script test format) to the design doc.

@ChrisHines, the third example explicitly covers the behavior for test-of-test dependencies.

bcmills · 2020-02-21T14:09:10Z

CC @julieqiu since this may affect how we display dependencies on pkg.go.dev.

CC @heschik, @dmitshur because this may affect the package search algorithms used by goimports and godoc.

CC @bep because I seem to recall that Hugo is doing something unusual with module dependencies independent of Go package imports. (But note that we are unlikely to actively support the use of Go modules for purposes other than compiling Go source code.)

bep · 2020-02-21T16:51:38Z

But note that we are unlikely to actively support the use of Go modules for purposes other than compiling Go source code.

Our unusual usage is fairly well anchored in the "Go project". That said, I have read your design document twice, and I think this is a good thing (even for our unusual use) -- but it would be easier to determine with a prototype (I find this topic to be a little bit of a mind twister and I don't fully understand the examples in the document).

…ild tags I've been thinking about the relationship between the package import graph and the module import graph, and realized that the package import graph is not always acyclic. (The package import graph must be acyclic given a specific set of build tags, but the 'mod' subcommands intentionally ignore build tags.) I'm not sure whether we have any existing regression tests that cover this sort of cycle, so I'm adding one now. Thankfully, it passes! Updates #36460 Change-Id: I7679320994ee169855241efa51cd45f71315f7f5 Reviewed-on: https://go-review.googlesource.com/c/go/+/217557 Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>

myitcv · 2020-02-24T11:02:03Z

@bcmills This is excellent, thank you.

I think I follow the detail, but have a couple of questions which might well prove otherwise 😄

Since the all pattern is based on package imports (more-or-less independent of module dependencies), this change should be independent of the go version specified in the go.mod file.

I'm not sure what this sentence means. In the context of changing the definition of the all package pattern, does this not imply the change to the definition is contingent on the go version specified in the go.mod file? Indeed, doesn't the next paragraph go on to say exactly that, in addition to saying the cmd/go version is also relevant?

Again related to the proposed change in the definition of the all package pattern, given the proposed changes to maintenance via go mod tidy of the main module's go.mod file, does this mean that go test all (assuming the previous, i.e. current, definition of all for one second) is no longer reproducible in the case of incomplete go.mod files? Do we need to retain a special pattern for the previous definition of package all? I'm not sure what the main driver was for the original all definition so can't really offer much other than vague questions here.

What about go.mod files that are left intentionally untidy? Are there any situation where this is still required/desirable? If so, how does that influence (or not influence) the proposed (behaviour) changes?

Regarding the transition to use the new behaviour. People will, per your proposal, need to specify go 1.15 in their go.mod files. I'm not currently aware of any side effects of making that change (from an earlier version), but perhaps that would be worth confirming via an FAQ?

Thinking about the compatibility section and CI use, if I've understood things correctly, running go mod tidy with go1.13.x on a go.mod file that was otherwise stable with go1.15.x will result in changes, but running with go1.14.x will not. I guess that's another potential FAQ entry for people who will likely add go1.15beta1 to their build matrix and then start seeing surprising diffs post the go mod tidy and go mod verify steps of their CI build?

bcmills · 2020-02-25T20:07:12Z

In the context of changing the definition of the all package pattern, does this not imply the change to the definition is contingent on the go version specified in the go.mod file?

Not really, no: the all package pattern (both before and after this proposal) is a function of the import graph, and only interacts with modules (and go.mod files) to the extent that those modules provide imported packages.

But not that the all module pattern does depend on the version, and for a go 1.15 module, the all module pattern as defined in Go 1.14 does not necessarily cover the all package pattern as defined in Go 1.14.

bcmills · 2020-02-25T20:09:04Z

What about go.mod files that are left intentionally untidy?

Don't do that? 😅

Are there any situation where this is still required/desirable? If so, how does that influence (or not influence) the proposed (behaviour) changes?

I would argue that there weren't any situations where that was ever required (or particularly desirable). But if you have specific examples, I'd like to look into them in more depth.

bcmills · 2020-02-26T03:44:20Z

People will, per your proposal, need to specify go 1.15 in their go.mod files. I'm not currently aware of any side effects of making that change (from an earlier version), but perhaps that would be worth confirming via an FAQ?

Currently the only other effect on module-mode semantics is that the transition from go 1.13 or lower to go 1.14 or higher also enables automatic use of -mod=vendor when the main module has a vendor subtree. Beyond that, the new language features that come along with the version are all strict additions so far.

bcmills · 2020-02-26T03:48:47Z

Thinking about the compatibility section and CI use, if I've understood things correctly, running go mod tidy with go1.13.x on a go.mod file that was otherwise stable with go1.15.x will result in changes, but running with go1.14.x will not.

go mod tidy with any older version, including 1.14, will demolish the go 1.15 invariants.

The difference between 1.13 and 1.14 is that 1.14 will not demolish those invariants unless you invoke go mod tidy explicitly.

Note that we have fixed bugs in go mod tidy in nearly every release so far, so I hope that by now folks have realized that go mod tidy-ness is version-dependent, much like go fmt cleanness. (And note that go fmt with an older Go version may choke on new language features, so it, too, is not something you want to run with older releases in CI.)

myitcv · 2020-02-26T04:59:48Z

I would argue that there weren't any situations where that was ever required (or particularly desirable). But if you have specific examples, I'd like to look into them in more depth.

This was the comment I had in mind when asking:

https://go-review.googlesource.com/c/tools/+/184018/2#message-7612508883ea3ee1f11a807241422e9b4e138da6

which refers through to #27899. I've not followed that in detail, so just raising for completeness; the point may now be moot.

myitcv · 2020-02-26T05:09:48Z

Currently the only other effect on module-mode semantics is that the transition from go 1.13 or lower to go 1.14 or higher also enables automatic use of -mod=vendor when the main module has a vendor subtree.

Very much a tangent, but are these changes documented somewhere centrally? That would be an incredibly useful guide to start and maintain (vs having the details scattered in various parts of the docs).

bcmills · 2020-02-26T15:07:20Z

This was the comment I had in mind when asking: …

Ah, now I remember. The untidiness there was “larger than tidy” rather than “smaller than tidy”, and I think we have basically settled on the // +build approach for that use-case.

bcmills · 2020-02-26T15:16:21Z

are these [go version] changes documented somewhere centrally?

Not yet. That's arguably part of #30791...

gopherbot · 2021-07-09T20:48:25Z

Change https://golang.org/cl/333629 mentions this issue: _content/ref/mod: document Go 1.17 lazy loading and module graph pruning

gopherbot · 2021-07-13T20:16:04Z

Change https://golang.org/cl/334389 mentions this issue: _content/doc/modules/pruning: add a placeholder redirecting to “Managing dependencies”

…ing dependencies” I plan to add a more in-depth troubleshooting guide at this URL. For now, I just need to URL to work so that I can point to it in the 'go' command in Go 1.17 (in CL 332573). For golang/go#36460 Change-Id: I95f03f76519dfb196ed6c9c13003b2ad9becf6c9 Reviewed-on: https://go-review.googlesource.com/c/website/+/334389 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Steve Traut <straut@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com>

The /doc link is currently a redirect (CL 334389), but I plan to update it soon with a more detailed guide. Updates #36460 Change-Id: I9e4a47ad0c8bcb7361cfa3e5b9d07ad241b13ba6 Reviewed-on: https://go-review.googlesource.com/c/go/+/332573 Trust: Bryan C. Mills <bcmills@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>

gopherbot · 2021-07-15T19:50:27Z

Change https://golang.org/cl/334889 mentions this issue: cmd/go: update error messages in tests to match CL 332573

I neglected to run the 'longtest' builder, and the tests that cover the error message changed in CL 332573 apparently do not run in short mode. Updates #36460 Updates #42661 Change-Id: I53500ddaca8ac9f0dfaab538923b3c9a4f71665e Reviewed-on: https://go-review.googlesource.com/c/go/+/334889 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com> TryBot-Result: Go Bot <gobot@golang.org>

gopherbot · 2021-07-16T21:00:41Z

Change https://golang.org/cl/335135 mentions this issue: doc/go1.17: clarify lazy-loading changes

For golang/go#36460 Change-Id: I36b3657103f069412b225356d011a19c1a10109e Reviewed-on: https://go-review.googlesource.com/c/website/+/333629 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>

gopherbot · 2021-08-27T20:22:25Z

Change https://golang.org/cl/345391 mentions this issue: cmd/go/internal/modload: remove go117EnableLazyLoading

gopherbot · 2021-08-27T20:22:26Z

Change https://golang.org/cl/345393 mentions this issue: cmd/go/internal/modload: use "pruned" instead of "lazy" to describe pruned module graphs

Updates #36460 Change-Id: I19f375f58f118e83a2615a29bbbb3853f059f0bb Reviewed-on: https://go-review.googlesource.com/c/go/+/345391 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>

…runed module graphs The level of support for pruning — not the lazy/eager loading behavior — is the more fundamental property, and what matters in terms of what invariants we need to maintain. If the main module supports pruned module graphs we load its dependencies lazily, and if it does not support pruned module graphs we load its dependencies eagerly. However, in principle we could also load the module graph lazily even in modules that do not support graph pruning — we would just be more likely to overlook inconsistent requirements introduced by hand-edits or bad VCS merges to the go.mod file. (After this change, a “lazy” module is just one in which we happen not to have loaded the module graph, and an “eager” one is one in which we happen to load the module graph more aggressively.) Updates #36460 For #47397 Change-Id: I0d2ffd21acc913f72ff56b59a6bdc539ebc3d377 Reviewed-on: https://go-review.googlesource.com/c/go/+/345393 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>

This change was produced using 'go mod tidy -go=1.17' with a go command built at CL 315210. This activates lazy loading, and updates the go.mod file to maintain the lazy-loading invariants (namely, including an explicit requirement for every package transitively imported by the main module). Note that this does *not* prevent users with earlier go versions from successfully building packages from this module. For golang/go#36460. Change-Id: I6697dd2fd740e59b959ee7b0458227374915293e Reviewed-on: https://go-review.googlesource.com/c/xerrors/+/316114 Run-TryBot: Bryan Mills <bcmills@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

This change was produced using 'go mod tidy -go=1.17' with a go command built at CL 315210. This activates lazy loading, and updates the go.mod file to maintain the lazy-loading invariants (namely, including an explicit requirement for every package transitively imported by the main module). Note that this does *not* prevent users with earlier go versions from successfully building packages from this module. For golang/go#36460 Change-Id: Iabb65fc3ed9727abecc3926abcecd445c967d0a9 Reviewed-on: https://go-review.googlesource.com/c/text/+/315571 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>

…ing dependencies” I plan to add a more in-depth troubleshooting guide at this URL. For now, I just need to URL to work so that I can point to it in the 'go' command in Go 1.17 (in CL 332573). For golang/go#36460 Change-Id: I95f03f76519dfb196ed6c9c13003b2ad9becf6c9 Reviewed-on: https://go-review.googlesource.com/c/website/+/334389 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Steve Traut <straut@google.com> Reviewed-by: Jay Conrod <jayconrod@google.com>

For golang/go#36460 Change-Id: I36b3657103f069412b225356d011a19c1a10109e Reviewed-on: https://go-review.googlesource.com/c/website/+/333629 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com> Reviewed-by: Michael Matloob <matloob@golang.org>

This change was produced using 'go mod tidy -go=1.17' with a go command built at CL 315210. This activates lazy loading, and updates the go.mod file to maintain the lazy-loading invariants (namely, including an explicit requirement for every package transitively imported by the main module). Note that this does *not* prevent users with earlier go versions from successfully building packages from this module. For golang/go#36460. Change-Id: I28a6f54b1e41e9e18b726cacba25a52069b8a4e9 Reviewed-on: https://go-review.googlesource.com/c/crypto/+/316109 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org> TryBot-Result: Go Bot <gobot@golang.org>

This change was produced using 'go mod tidy -go=1.17' with a go command built at CL 315210. This activates lazy loading, and updates the go.mod file to maintain the lazy-loading invariants (namely, including an explicit requirement for every package transitively imported by the main module). Note that this does *not* prevent users with earlier go versions from successfully building packages from this module. For golang/go#36460. Change-Id: Idb3d8d056d41bb254d0b5d9b2699ac33a51081f3 Reviewed-on: https://go-review.googlesource.com/c/sync/+/316130 Auto-Submit: Bryan Mills <bcmills@google.com> Commit-Queue: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Jonathan Amsterdam <jba@google.com> Run-TryBot: Bryan Mills <bcmills@google.com>

bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. modules labels Jan 8, 2020

bcmills added this to the Go1.15 milestone Jan 8, 2020

gopherbot added the Proposal label Jan 8, 2020

bcmills closed this as completed Jan 8, 2020

jayconrod mentioned this issue Feb 11, 2020

x/exp: reconsider using single module for all unrelated experimental packages of various maturity levels #37175

Closed

bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 19, 2020

bcmills reopened this Feb 19, 2020

bcmills added the early-in-cycle A change that should be done early in the 3 month dev cycle. label Feb 19, 2020

bcmills added this to Incoming in Proposals (old) Feb 19, 2020

bcmills closed this as completed Feb 25, 2020

bcmills reopened this Feb 25, 2020

rsc moved this from Incoming to Active in Proposals (old) Feb 26, 2020

cueckoo mentioned this issue Jul 3, 2021

Proposal: package management cue-lang/cue#851

Closed

maurerbot mentioned this issue Jul 22, 2021

feat: add cue encoder (#1) spf13/viper#1175

Closed

gopherbot closed this as completed in 63b968f Aug 6, 2021

bcmills changed the title ~~cmd/go: lazy module loading~~ cmd/go: module graph pruning and lazy module loading Aug 6, 2021

jeffwidman mentioned this issue Nov 30, 2021

Unfixable failure when the dep tree includes a dependency circle and one of those deps at one time referenced a broken module dependabot/dependabot-core#3526

Closed

jeffwidman mentioned this issue Feb 17, 2022

An entry in go.sum does not represent a dependency dependabot/dependabot-core#4740

Closed

bcmills mentioned this issue Apr 12, 2022

proposal: cmd/go: go mod: limit version resolution to packages that are consumed #52296

Closed

rsc unassigned bcmills Jun 23, 2022

zikaeroh mentioned this issue Dec 5, 2022

extended forwards compatibility for Go #57001

Closed

joshklop mentioned this issue Jun 23, 2023

Consider updating declared Go version in go.mod nhooyr/websocket#359

Closed

golang locked and limited conversation to collaborators Jun 23, 2023

gopherbot added the FrozenDueToAge label Jun 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: module graph pruning and lazy module loading #36460

cmd/go: module graph pruning and lazy module loading #36460

bcmills commented Jan 8, 2020 •

edited by rsc

gopherbot commented Feb 3, 2020

gopherbot commented Feb 19, 2020

bcmills commented Feb 19, 2020 •

edited

ChrisHines commented Feb 20, 2020

bcmills commented Feb 20, 2020

bcmills commented Feb 21, 2020

bep commented Feb 21, 2020

myitcv commented Feb 24, 2020

bcmills commented Feb 25, 2020

bcmills commented Feb 25, 2020

bcmills commented Feb 26, 2020

bcmills commented Feb 26, 2020

myitcv commented Feb 26, 2020

myitcv commented Feb 26, 2020

bcmills commented Feb 26, 2020

bcmills commented Feb 26, 2020

gopherbot commented Jul 9, 2021

gopherbot commented Jul 13, 2021

gopherbot commented Jul 15, 2021

gopherbot commented Jul 16, 2021

gopherbot commented Aug 27, 2021

gopherbot commented Aug 27, 2021

cmd/go: module graph pruning and lazy module loading #36460

cmd/go: module graph pruning and lazy module loading #36460

Comments

bcmills commented Jan 8, 2020 • edited by rsc

gopherbot commented Feb 3, 2020

gopherbot commented Feb 19, 2020

bcmills commented Feb 19, 2020 • edited

ChrisHines commented Feb 20, 2020

bcmills commented Feb 20, 2020

bcmills commented Feb 21, 2020

bep commented Feb 21, 2020

myitcv commented Feb 24, 2020

bcmills commented Feb 25, 2020

bcmills commented Feb 25, 2020

bcmills commented Feb 26, 2020

bcmills commented Feb 26, 2020

myitcv commented Feb 26, 2020

myitcv commented Feb 26, 2020

bcmills commented Feb 26, 2020

bcmills commented Feb 26, 2020

gopherbot commented Jul 9, 2021

gopherbot commented Jul 13, 2021

gopherbot commented Jul 15, 2021

gopherbot commented Jul 16, 2021

gopherbot commented Aug 27, 2021

gopherbot commented Aug 27, 2021

bcmills commented Jan 8, 2020 •

edited by rsc

bcmills commented Feb 19, 2020 •

edited