Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: no way to vendor symlink files with Go modules #39600

Closed
chizhg opened this issue Jun 15, 2020 · 6 comments
Closed

cmd/go: no way to vendor symlink files with Go modules #39600

chizhg opened this issue Jun 15, 2020 · 6 comments
Labels
FrozenDueToAge GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Unfortunate
Milestone
Unplanned

Comments

@chizhg
Copy link

chizhg commented Jun 15, 2020 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.14.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/chizhg/Library/Caches/go-build"
GOENV="/Users/chizhg/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/chizhg/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/chizhg/go/src/k8s.io/test-infra/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/n1/3yzf5ppx0_9c0vnsmf8k6k2c00lvhy/T/go-build213053347=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

  1. In https://github.com/knative/serving/tree/master/config, we have added a few symlinks to aggregate files from different folders to one place.
  2. One of our downstream projects needs to vendor the knative.dev/serving project using Go modules, and needs these symlinks in config/ as well.

What did you expect to see?

The symlinks can be vendored by the downstream project by either

  • adding a dummy.go file under the config/ package, and adding import _ "knative.dev/serving/config in one of the Go files

or

What did you see instead?

The symlinks cannot be vendored by using the first approach (adding a dummy.go file), and it cannot be vendored by force copying either, as when we run go list -f '{{.Dir}}' -mod=readonly -m knative.dev/serving, the local mod directory does not contain the symlinks in the first place.
@andybons andybons changed the title We don't have a way to vendor symlink files with Go modules cmd/go: no way to vendor symlink files with Go modules Jun 15, 2020
@andybons andybons added GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 15, 2020
@andybons andybons added this to the Unplanned milestone Jun 15, 2020
@andybons
Copy link
Member

@bcmills @jayconrod @matloob

@bcmills
Copy link
Contributor

bcmills commented Jun 16, 2020

Not all filesystems in current use support symlinks. The module file formats are intended to be portable, so symlinks are intentionally skipped (see #27093 and #24057 (comment)).

@bcmills
Copy link
Contributor

bcmills commented Jun 16, 2020

when we run go list -f '{{.Dir}}' -mod=readonly -m knative.dev/serving, the local mod directory does not contain the symlinks in the first place.

What did you do to try to populate it?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 16, 2020
@chizhg
Copy link
Author

chizhg commented Jun 16, 2020

Below is what we tried:

  1. Run go mod vendor to reset the main module's vendor directory and download the dependent modules to the local cache.
  2. Run go list -f '{{.Dir}}' -mod=readonly -m knative.dev/serving to find out local cache path for the module, which is for example ${GOPATH}/pkg/mod/knative.dev/serving@v0.15.1
  3. Go to the config files path ${GOPATH}/pkg/mod/knative.dev/serving@v0.15.1/config and try to copy the symlinks, but they are not there.
    So this approach does not work.

Right now we are using a very hacky way to get these symlinks:

  1. Run go list -f '{{.Version}}' -mod=readonly -m knative.dev/serving to find out the version of this module, e.g. v0.15.1
  2. Run git clone https://github.com/knative/serving.git --branch v0.15.1 to clone the repo to a temp directory, copy these symlinks from the temp directory to vendor/knative.dev/serving/config, and them delete the temp directory.
    This approach only works well when we depend on a tagged version of that module, but does not if we depend on a specific commit, or HEAD of master branch. So we really wish there could be a solution to vendor symlinks with Go modules.

@bcmills
Copy link
Contributor

bcmills commented Jun 16, 2020

You should be able to find the actual files underlying the links using go list, but the symlinks themselves will not be present: as mentioned above, they are intentionally omitted from the module zip file, and the zip files are checksummed, so their contents cannot change without a really compelling reason.

Your git clone alternative seems like the right approach here. These symlinks are not needed for the go tool itself, and the go tool's requirements are what drive the module file formats.

@bcmills
Copy link
Contributor

bcmills commented Jun 16, 2020

Closing as not providing a large enough benefit to outweigh the churn of invalidating checksums.

@bcmills bcmills closed this as completed Jun 16, 2020
@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 16, 2020
@golang golang locked and limited conversation to collaborators Jun 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Unfortunate
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
4 participants
@andybons @bcmills @gopherbot @chizhg