Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: "go install" ignores vendor folder #48332

Closed
as opened this issue Sep 11, 2021 · 8 comments
Closed

cmd/go: "go install" ignores vendor folder #48332

as opened this issue Sep 11, 2021 · 8 comments

Comments

@as
Copy link
Contributor

as commented Sep 11, 2021

The go install command ignores my vendor folder. This behavior differs from go get.

Below is a command line utility with a vendor folder.

go install github.com/as/ccp@v0.0.2

When run, the output suggests it is downloading existing content redundantly.

The go.mod, go.sum, and vendor folder exist and are configured. This behavior is surprising and prevents easy packaging of versioned command line tools using go modules, especially in a locked down environment where packages are hosted on an air gapped or heavily firewalled network.

; docker build - <<EOF
FROM golang:1.17-alpine
RUN go install github.com/as/ccp@v0.0.1
ENTRYPOINT ["/bin/sh"]
EOF

What did you expect to see?

Step 4/5 : RUN go install github.com/as/ccp@v0.0.1
go: downloading github.com/as/ccp v0.0.1

What did you see instead?

Step 4/5 : RUN go install github.com/as/ccp@v0.0.1
go: downloading github.com/as/ccp v0.0.1
go: downloading google.golang.org/genproto v0.0.0-20210624174822-c5cf32407d0a
go: downloading google.golang.org/api v0.49.0
...
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading golang.org/x/oauth2 v0.0.0-20210615190721-d04028783cf1

/cc @bcmills

@theckman
Copy link
Contributor

theckman commented Sep 11, 2021

That feels like this is an oversight, with the official Modules proxy being unable to guarantee that all packages it cached will remain cached forever. Considering that, vendoring is the only mechanism we have for 100% reproducible builds, with it being especially important for tools that choose to be distributed this way and not precompiled and distributed via the OS's package manager.

@jayconrod
Copy link
Contributor

This is working as intended, though it looks like it needs to be documented better. Neither https://golang.org/ref/mod#go-install nor go help install mention that vendor directories are not used. They should.

go install cmd@version is meant to be a simple replacement for go get that works in most situations. In particular, when go install is used with a version suffix, no module is considered the main module, which means replace and exclude directives are not applied (actually it's an error if they're present in the module providing the packages named on the command line), and the vendor directory is not used. Additionally, the vendor directory can't be used because it's not present in module zip files. #40276 is the proposal where this was discussed. See the rationale and FAQs sections there.

I understand that doesn't really solve your problem though. I'd suggest one of the following solutions:

  • Use go install cmd instead of go install cmd@version. The form without the @version suffix still works and will still use the vendor directory from the main module. You'll need to checkout the module from a VCS though, and if it's not a module you develop, you'll need to mirror the VCS somewhere locally.
  • Run a local proxy behind your firewall. This could be something as simple as downloading dependencies ahead of time into a module cache, then using the module cache as a proxy with a file:// URL. Or something like Athens.

@jayconrod jayconrod added this to the Go1.18 milestone Sep 13, 2021
@jayconrod jayconrod self-assigned this Sep 13, 2021
@gopherbot
Copy link

Change https://golang.org/cl/349590 mentions this issue: _content/ref/mod: document 'go install cmd@version' ignores vendor

gopherbot pushed a commit to golang/website that referenced this issue Sep 13, 2021
For golang/go#48332

Change-Id: I3e1b6228ce2ef6c78b851bffb79b48e9ac06aad0
Reviewed-on: https://go-review.googlesource.com/c/website/+/349590
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/349591 mentions this issue: cmd/go: document 'go install cmd@version' ignores vendor directories

gopherbot pushed a commit that referenced this issue Sep 13, 2021
For #48332

Change-Id: I708eb3e8f3f386f03210b7117d9ab8b0be2125bb
Reviewed-on: https://go-review.googlesource.com/c/go/+/349591
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@theckman
Copy link
Contributor

@jayconrod can you point to the document where that was discussed? That's the distribution command for many applications, that cannot be what we intend. :(

@mvdan
Copy link
Member

mvdan commented Sep 14, 2021

@theckman see "Why can't vendor directories be used?" and the other mentions of module zips in #40276

@jayconrod
Copy link
Contributor

@theckman As @mvdan pointed out, this is mentioned in #40276. We definitely thought about it. Please read through the design doc and the discussion there.

Supporting vendor directories would require a completely different implementation than what any other module-aware subcommand does, including go get, which go install cmd@version was meant to replace. We decided not to go in that direction.

@vsapronov
Copy link

@as I needed the same to work in my project. Saw this issue created by you.
My situation is even more complicated since I have replace directives in my go.mod of the module that I want to install.
I just went to the option of releasing my tool that I want to be go-installable on the side - in the different repo.
And I have my tool that is just "localizing" dependencies code, including vendored dependencies. So the output of my tool is just a self-contained pile of code ready to be placed into "release" repository. The tool is here: https://github.com/specgen-io/goven/blob/main/README.md

passionSeven added a commit to passionSeven/website that referenced this issue Oct 18, 2022
For golang/go#48332

Change-Id: I3e1b6228ce2ef6c78b851bffb79b48e9ac06aad0
Reviewed-on: https://go-review.googlesource.com/c/website/+/349590
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
liam-verta added a commit to VertaAI/kaniko that referenced this issue Apr 8, 2023
See golang/go#48332 for background.

Calling `go install <package>@<commit>` bypasses the local `go.mod`
dependency versioning and also ignores dependencies in the `vendor` directory.

To address this we must add a file that imports the packages we
intend to install - it only needs to import them not do anything else with them.

With that done, calling `go install <package>@<commit>` and then
`go mod vendor` will properly add the package and its dependencies to
the `go.mod` file, `go.sum` file, and `vendor` directory.

Finally, future calls must be in the form `go install <package>` and then
the package and dependencies will be looked up using the local settings.
liam-verta added a commit to VertaAI/kaniko that referenced this issue Apr 10, 2023
**Desired Behavior**

* Installed tools versions are locked the versions of installed tools to specific versions
* Versions of the dependencies used by installed tools are controlled by `go.mod` and `go.sum`

**Actual behavior**

The Dockerfile includes tool installations locked to a specific commit by referencing that commit in the command `go install <package>@<commit>`:

https://github.com/GoogleContainerTools/kaniko/blob/24846d22b11f20edc03dff7ac3b3959019969873/deploy/Dockerfile#L25-L32

However, this form of the `go install` command bypasses the local `go.mod`
dependency versioning and also ignores dependencies in the `vendor` directory.  Instead it goes to the project commit referenced and downloads the versions of the dependencies listed in that external package and also uses those at runtime.  This documented but easily overlooked behavior.

See golang/go#48332 for background.

**Proposed fix**

To address this we must add a file that imports the packages we
intend to install - it only needs to import them not do anything else with them.

With that done, manually calling `go install <package>@<commit>` will properly add `<package>` and its dependencies to the `go.mod` and `go.sum` files.
Then calling `go mod vendor` will update the `vendor` directory to include the updated dependency.
In both calls the dependency versions in `go.mod` will take precedence and can be modified if needed.

Finally, future scripted calls must be in the form `go install <package>` so that `<package>` and dependencies will be looked up using the local configuration.

**Side effects of proposed fix**

* Installed tools and their dependencies are are included in the `vendor` directory  - `Docker-slim` is less slim due the additional vendoring, but that scenario is not built by Verta.
@golang golang locked and limited conversation to collaborators Aug 5, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants