Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: build fails with sym links ("no Go files") #52406

Closed
jhgit opened this issue Apr 18, 2022 · 4 comments
Closed

cmd/go: build fails with sym links ("no Go files") #52406

jhgit opened this issue Apr 18, 2022 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@jhgit
Copy link

jhgit commented Apr 18, 2022

What version of Go are you using (go version)?

$ go version
go version go1.18 freebsd/amd64

Does this issue reproduce with the latest release?

Unknown

What operating system and processor architecture are you using (go env)?

env GOPATH=/tmp/go go env Output
$ env GOPATH=/tmp/go go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/xxx/.cache/go-build"
GOENV="/home/xxx/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOINSECURE=""
GOMODCACHE="/home/xxx/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="freebsd"
GOPATH="/tmp/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="cc"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2409260475=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am trying to build some go projects (e.g., https://github.com/FiloSottile/age, https://github.com/muesli/duf).

If I start with an GOPATH pointing to an empty directory tree or one already populated with downloaded files, the build succeeds. If GOPATH is a directory with symbolic links to files in another directory, the build fails.

Here is a script to reproduce the problem (using 'age' as an example, but this happens with other go projects as well). This script uses a tool called 'lndir' to create a hierarchy of sym links. It uses /tmp/src and /tmp/go and /tmp/go-links as working directories.

$ cat repro
rm -rf /tmp/src
mkdir /tmp/src
cd /tmp/src
wget --quiet https://github.com/FiloSottile/age/archive/refs/tags/v1.0.0.zip
unzip v1.0.0.zip > /dev/null
cd age-1.0.0

echo "=== first build starting with no downloaded packages"
echo "=== download packages to /tmp/go"
find /tmp/go -type d -exec chmod u+w {} +
rm -rf /tmp/go
env GOPATH=/tmp/go go mod vendor
env GOPATH=/tmp/go go build -v -o age ./cmd/age
ls -al age

echo "=== now do another build starting with the packages downloaded already"
rm age
env GOPATH=/tmp/go go build -v -o age ./cmd/age
ls -al age

echo "=== now build using tree of links to source packages in /tmp/go-links"
cd ..
mv age-1.0.0 age-1.0.0.no-links
unzip v1.0.0.zip > /dev/null
cd age-1.0.0
rm -rf /tmp/go-links
mkdir /tmp/go-links
lndir /tmp/go /tmp/go-links > /dev/null
env GOPATH=/tmp/go-links go mod vendor
env GOPATH=/tmp/go-links go build -v -o age ./cmd/age
ls -al age

$ sh repro

Observe errors (as shown below) from running repro script.

I'm not sure where to start looking for the source of this error. Apologies if this issue is mis-categorized, but I'm also not sure if this is the best place to start.

What did you expect to see?

No errors.

What did you see instead?

Errors like the following (for a build of age):

package filippo.io/age/cmd/age
        imports filippo.io/age/agessh
        imports filippo.io/edwards25519: no Go files in /tmp/src/age-1.0.0/vendor/filippo.io/edwards25519
package filippo.io/age/cmd/age
        imports filippo.io/age
        imports golang.org/x/crypto/chacha20poly1305: no Go files in /tmp/src/age-1.0.0/vendor/golang.org/x/crypto/chacha20poly1305
package filippo.io/age/cmd/age
        imports golang.org/x/crypto/cryptobyte: no Go files in /tmp/src/age-1.0.0/vendor/golang.org/x/crypto/cryptobyte
.
.

Notice that vendor packages are missing (silently not downloaded) from the source tree after 'go mod vendor' when using the GOPATH hierarchy with symbolic links.
@ianlancetaylor
Copy link
Contributor

In general the go tool intentionally does not support symlinks, but I don't know whether we've written that down anywhere.

CC @bcmills

@ianlancetaylor ianlancetaylor added Documentation NeedsFix The path to resolution is known, but the work has not been done. labels Apr 19, 2022
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Apr 19, 2022
@jhgit
Copy link
Author

jhgit commented Apr 19, 2022 •
edited

It seems to me that you generally have to do something special to "fail" to read a link that ultimately points to the desired file (for instance, call lstat(2) instead of the typical stat(2) when determing file status information). If you just "read" the filename (using the name of the link), it (the system that allows symlinks to exist) would normally follow the link and things would "just work". I certainly don't know what go is doing under the covers, so I imagine it's more complicated than that. But I wouldn't think go would need to do anything special to do the typical "right" thing for symlinks, which is to just let the system follow the sym link to access the contents of the file in question.

That said, this is an extremely difficult thing to debug and something of a cryptic failure. Go does not emit any information indicating why 'go mod vendor' behaves differently. If the decision is to "not support symlinks", at least go could provide more information so a user can see what is happening. Maybe there is a better way to trace what's going on, and I am just not aware of it yet. My first instinct was to add extra -v's, but that didn't help (for that matter a single -v didn't help).

@bcmills
Copy link
Contributor

bcmills commented Apr 19, 2022

We don't go out of our way to support symlinks, and don't walk symlink directories when matching patterns like ./..., but in general I would expect that a tree of symlinked files would work.

(Compare to the approach taken in https://cs.opensource.google/go/go/+/master:misc/cgo/test/overlaydir_test.go;drc=476395cb3e97cf7d9c50d8fe57c2c7926f1cff48.)

@jhgit, that repro script has a lot going on. Can you reduce the behavior to a cmd/go script test? (See https://cs.opensource.google/go/go/+/master:src/cmd/go/testdata/script/README for documentation for that format.)

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed Documentation labels Apr 19, 2022
@gopherbot gopherbot removed the NeedsFix The path to resolution is known, but the work has not been done. label Apr 19, 2022
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators May 19, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@jhgit @ianlancetaylor @bcmills @gopherbot