Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: support vendoring in workspace mode #60056

Closed
matloob opened this issue May 8, 2023 · 15 comments
Closed

cmd/go: support vendoring in workspace mode #60056

matloob opened this issue May 8, 2023 · 15 comments

Comments

@matloob
Copy link
Contributor

matloob commented May 8, 2023

This is a proposal to support vendoring in workspace mode.

We originally didn't support vendoring of a workspace to simplify the implementation of workspaces, but there's recently been demand by users of vendoring who want to workspaces for that support, so now is a good time to consider adding that support in.

go work vendor subcommand

Under this proposal, a new subcommand, go work vendor, will be added to the go command. go work vendor will check to make sure that workspace mode is active when it starts, (and return an error if it isn't) and if it is, it will write a vendor directory for the workspace. The vendor directory will be created in the same directory as the go.work file. The vendor directory will contain packages taken from the modules in the build list of the workspace. vendor/modules.txt will contain the set of packages from each of the modules in the build list as usual, will list modules with # explicit annotations for each module required by one of the workspace modules, and will annotate replacements for each of the effective replacements in the workspace (That is, if the go.work file overrides replacements in workspace only the replacement in the go.work file will be annotated.)

go work vendor otherwise behaves the same as and takes the same flags as go mod vendor.

go mod vendor subcommand

go mod vendor will explicitly fail if a go.work file is found and GOWORK is not set to off. See discussion below.

-mod=vendor in workspace mode.

Under this proposal, when workspace mode is active, if -mod=vendor is set explicitly, or it is set implicitly through the existence of a vendor directory in the same directory as the go.work The go command will enter workspace mode and interpret the vendor directory as reflecting the workspace's modules. The go command will use the vendor directory in the same directory as go.work. vendor/modules.txt is interpreted as reflecting the modules in the workspace. That interpretation is done through the vendoring consistency check, and the structure of the internal module graph built.

The vendoring consistency check would check vendor/modules.txt against the workspace modules. It will check that all explicitly required modules appear in vendor/modules.txt with an # explicit annotation and all modules in vendor/modules.txt with # explicit are required by a workspace module. Similarly, all effective replacements in the workspace are expected to appear in vendor/modules.txt and all replacements in vendor/modules.txt are expected to appear in a workspace module go.mod or the go.work file.

The way the module graph is built is similar to to that of an unpruned module. The module graph contains the main modules as its roots, and each of the main modules would require their module requirements in addition to a fake vendor/modules.txt module that in turn requires the set of modules in the vendorList (the set of modules in vendor/modules.txt` that provide packages.

Otherwise, vendor mode with a workspace behaves the same as vendor mode with a module. The packages in the vendor directory are used to satisfy imports in the main modules when loading packages and doing builds.

Other considerations.

go work vendor and go mod vendor.

Under the proposal as it is, go mod vendor works to vendor the module it's in, but fails if there's a go.work file present and GOWORK is not set to off. If that wasn't the case, there could be confusion if a go.mod and go.work files exist in the same directory. Then go mod vendor and go work vendor would create vendor directories in the same place, but with different contents. Since in workspace mode, the vendor directory is interpreted as a reflecting the workspace, if go mod vendor is run before another command in workspace mode, the vendor consistency checks will likely fail. When they fail the error message will give the user the appropriate command to run go mod vendor or go work vendor depending on whether modules.txt was checked against a single module or a the workspace. Even so, the user might still get confused about why running go mod vendor and immediately running another command might result in an error. To reduce the confusion, we require a user to not have a go.work file present or have GOWORK set to off when go mod vendor is run, so that when a following command is run under the same conditions, it doesn't interpret the vendor directory as reflecting the workspace.

The behavior might break already existing workflows where go mod vendor is run when a go.work is present and module vendoring is being used, but it seems better to do so than to introduce the potential confusion of the user creating a vendor directory that fails consistency checks.

@cherrymui cherrymui added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 8, 2023
@cherrymui cherrymui added this to the Backlog milestone May 8, 2023
@thockin
Copy link

thockin commented May 8, 2023

The lack of vendor support makes go workspaces very difficult for Kubernetes to adopt. I don't care if it is exactly identical to classic vendor support, but we need something...

  • We actually review all changes to pinned deps.
  • K8s has been burned more than once by upstream repos which disappear, force-push and obliterate history, or change their vanity go-get URLs to non-code things to intentionally break people.
  • I have personally been burned by trying to work on an airplane with what I THOUGHT was a full set of downloaded modules, but it turns out it was not really the full set.
  • The public Go module proxy does not retain every SHA forever.
  • The protoc runs we do require access to gogo.proto, which is currently satisfied via vendor/.../gogo.proto

@gopherbot
Copy link

Change https://go.dev/cl/495801 mentions this issue: cmd/go: add support for vendoring in workspace mode

@matloob matloob changed the title cmd/go: support vendoring in workspace mode proposal: cmd/go: support vendoring in workspace mode May 31, 2023
@matloob matloob modified the milestones: Backlog, Proposal May 31, 2023
@matloob matloob removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 1, 2023
@gopherbot
Copy link

Change https://go.dev/cl/499977 mentions this issue: gopls/internal/regtest/codelens: set GOWORK=off for go mod vendor

gopherbot pushed a commit to golang/tools that referenced this issue Jun 13, 2023
We might be introducing vendoring for workspace mode. Set GOWORK=off
when running go mod vendor in a single module context to make sure it
has the right behavior and doesn't return an error.

For golang/go#60056

Change-Id: I703d74d579aec6e4dad86ca092e3651e0b2e4eb0
Reviewed-on: https://go-review.googlesource.com/c/tools/+/499977
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Michael Matloob <matloob@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
@rsc
Copy link
Contributor

rsc commented Jun 28, 2023

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc
Copy link
Contributor

rsc commented Jul 12, 2023

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc
Copy link
Contributor

rsc commented Jul 19, 2023

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: cmd/go: support vendoring in workspace mode cmd/go: support vendoring in workspace mode Jul 19, 2023
@rsc rsc modified the milestones: Proposal, Backlog Jul 19, 2023
gopherbot pushed a commit that referenced this issue Jul 24, 2023
In most cases this change removes assumptions that there is a single
main module in vendor mode and iterates over the workspace modules
when doing checks. The go mod vendor command will now, if in workspace
mode, create a vendor directory in the same directory as the go.work
file, containing the packages (and modules in modules.txt) loaded from
the workspace. When reassembling the module graph from the vendor
directory, an edges are added from each of the main modules to their
requirements, plus additionally to a fake 'vendor/modules.txt' module
with edges to all the modules listed in vendor/modules.txt.

For #60056

Change-Id: I4a485bb39836e7ab35cdc7726229191c6599903e
Reviewed-on: https://go-review.googlesource.com/c/go/+/495801
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Michael Matloob <matloob@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/513756 mentions this issue: cmd/go: enter vendor mode depending on new modules.txt workspace line

gopherbot pushed a commit that referenced this issue Aug 11, 2023
modules.txt gets a new ## workspace line at the start of the file if
it's generated in workspace mode. Then, when deciding whether the go
command runs in mod=vendor, we only do so if we're in the same mode
(workspace or not) as the modules.txt specifies.

For #60056

Change-Id: If478a9891a7135614326fcb80c4c33a431e4e531
Reviewed-on: https://go-review.googlesource.com/c/go/+/513756
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
@bcmills bcmills modified the milestones: Backlog, Go1.22 Sep 7, 2023
@findleyr
Copy link
Contributor

findleyr commented Oct 4, 2023

@bcmills what is remaining to do here?

@hyangah
Copy link
Contributor

hyangah commented Oct 4, 2023

I personally think it will be helpful if there is a sketch of how this proposed (and now accepted) feature is going to be used in k8s. My current concern is the same vendor directory is interpreted in different ways depending on the environment. @thockin @matloob

@pohly
Copy link

pohly commented Oct 4, 2023

Is your concern about "depending on the environment" regarding switching between workspace mode on vs. off? I believe all Go invocations in k8s will be done in workspace mode. @thockin knows best, but I've been following along and was trying to help at one point by implementing a hacky solution for a pseudo-vendor dir in workspace mode. That hack thankfully became obsolete with this feature here!

@thockin
Copy link

thockin commented Oct 4, 2023

Sigh, I really need to finish this work.

Kubernetes has a top-level module and workspace. vendor lives there. We expect pretty much everything to use the same vendor tree and to run in workspace mode.

@findleyr
Copy link
Contributor

findleyr commented Oct 5, 2023

Per discussion with @rsc, we think this is done.

@findleyr findleyr closed this as completed Oct 5, 2023
@rsc
Copy link
Contributor

rsc commented Oct 5, 2023

Fine to close now. @thockin if you do run into any problems, please let us know.

@hyangah
Copy link
Contributor

hyangah commented Oct 5, 2023

Is your concern about "depending on the environment" regarding switching between workspace mode on vs. off?

Yes, I was worried this change could interfere with some existing workspace mode users' current workflow who want to keep vendor in non-workspace mode while keeping their go.work files in the repo root. @rsc informed me that the latest change cl 513756 addressed it. The modules.txt has now a new ## workspace line, so the go command can distinguish the mode, instead of failing hard as the top comment implied.

I tested with my existing workflow, and it seems ok so far. :-)

@gopherbot
Copy link

Change https://go.dev/cl/548815 mentions this issue: doc/go1.22: document workspace vendoring

gopherbot pushed a commit that referenced this issue Dec 11, 2023
For #61422.
Updates #60056.

Change-Id: Ie7c7e12acc173f1ff1644555016e51a52509bd6d
Reviewed-on: https://go-review.googlesource.com/c/go/+/548815
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
ezz-no pushed a commit to ezz-no/go-ezzno that referenced this issue Feb 18, 2024
For golang#61422.
Updates golang#60056.

Change-Id: Ie7c7e12acc173f1ff1644555016e51a52509bd6d
Reviewed-on: https://go-review.googlesource.com/c/go/+/548815
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Accepted
Development

No branches or pull requests

9 participants