Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: GOPATH mode rejects directory arguments outside of GOPATH/src that contain '@' characters #45944

Open
metux opened this issue May 4, 2021 · 12 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@metux
Copy link

metux commented May 4, 2021

What version of Go are you using (go version)?

go version go1.14 linux/386

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE=""
GOARCH="386"
GOBIN=""
GOCACHE="/home/nekrad/.cache/go-build"
GOENV="/home/nekrad/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="386"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/lib/go-1.13"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.14"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.14/pkg/tool/linux_386"
GCCGO="gccgo"
GO386="387"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m32 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build963122998=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Building w/ 'go build' and setting GOPATH to a directory that happens to have @ in its name.

What did you expect to see?

Build w/o errors

What did you see instead?

/usr/lib/go-1.14/bin/go build -ldflags "-s -w" -o /home/nekrad/src/deb-pkg/pkg/moby-engine@master.git/tmp/build/dockerd-linux-i386 /home/nekrad/src/deb-pkg/pkg/moby-engine@master.git/cmd/dockerd
can't load package: package /home/nekrad/src/deb-pkg/pkg/moby-engine@master.git/cmd/dockerd: cannot use path@version syntax in GOPATH mode

It tries to do some magic interpretation on an opaque directory name, which it should NOT do.
The directory names are externally given by some infrastructure.
'go build' just should take it as it is (and cope w/ all characters that are allowed for file names on Linux/Unix platforms).

@cagedmantis cagedmantis changed the title go build breaks with @ in GOPATH path names cmd/go: build breaks with @ in GOPATH path names May 4, 2021
@cagedmantis cagedmantis added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 4, 2021
@cagedmantis
Copy link
Contributor

go1.14 is not one of the actively supported versions (go1.15, go1.16). Can you reproduce the issue with one of the supported versions?

@metux
Copy link
Author

metux commented May 5, 2021

go1.14 is not one of the actively supported versions (go1.15, go1.16). Can you reproduce the issue with one of the supported versions?

Unfortunately, I have to cope with whats delivered by (stable) distros. Even 1.14 is only delivered by additional backport repos, no 1.15 available.

@mvdan
Copy link
Member

mvdan commented May 5, 2021

Like @cagedmantis said, 1.14.x is no longer supported, so even if a bug was spotted here, at most there would be a backport to 1.15.x.

I imagine that what's happening here is that @ is not allowed in package paths, as it clashes with the modules syntax to specify a version, like go get foo/bar@v1.2.3.

Consider escaping that special character, just like you might usually do for slashes or spaces.

@bcmills where are these restrictions documented? I don't see it in go help importpath or go help packages.

@seankhliao
Copy link
Member

I believe the current set of restrictions are #45549 and @ hasn't been allowed since modules were introduced in 1.11

cc @jayconrod @matloob

@bcmills
Copy link
Contributor

bcmills commented May 5, 2021

Building w/ 'go build' and setting GOPATH to a directory that happens to have @ in its name.

Please provide concrete steps to reproduce the observed behavior using a supported version of the go tool (1.16.3 or 1.15.11).

The posted go env output (GOPATH="/usr/lib/go-1.13") does not appear to correspond to “setting GOPATH”. The specific details are very relevant to the validation that the go command performs, so if we are to address the underlying use-case we need to know how to reproduce it accurately.

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels May 5, 2021
@metux
Copy link
Author

metux commented May 6, 2021

I believe the current set of restrictions are #45549 and @ hasn't been allowed since modules were introduced in 1.11

cc @jayconrod @matloob

I'm confused - this wasn't supposed to be a module name, but a plain path name in GOPATH.

Meanwhile I've found a workaround (pretty dirty hack) to build moby, but autogenerating go.mod via 'go mod init'.
BUT: I need to point GOPATH to an unwritable directly, so it can't download anything - otherwise lots of errors break the build.
This is very ugly, but at least seems to work (somehow).

Is there any way for preventing any downloads and remote access completely ?

--mtx

@jayconrod
Copy link
Contributor

Based on the command you posted earlier, I expect you'll still see this error (with a slightly different message) in Go 1.16 because of this argument:

/home/nekrad/src/deb-pkg/pkg/moby-engine@master.git/cmd/dockerd

To avoid confusion and ambiguous commands, go build disallows @ characters in arguments except when they're used in path@version arguments in module mode.

This isn't a problem for import paths, since the language spec does not allow @ characters in import declarations. It may be a problem (as in this case) for relative and absolute directory paths, but I think it's better to keep the current strict behavior rather than change anything.

I'm confused - this wasn't supposed to be a module name, but a plain path name in GOPATH.

/home/nekrad/src/deb-pkg/pkg/moby-engine@master.git/cmd/dockerd is a directory path. You may be able to build dockerd using its normal package path instead, but it looks like this has been customized enough that I'm not sure what that would be. cmd/dockerd maybe?

Is there any way for preventing any downloads and remote access completely ?

In module-aware mode, go mod vendor will build a vendor directory (fetching what it needs from the network), then you can build offline with -mod=vendor (which may be enabled automatically, depending on the go version in go.mod). There are a few ways to build offline, but that's probably the easiest.

@bcmills
Copy link
Contributor

bcmills commented May 7, 2021

To avoid confusion and ambiguous commands, go build disallows @ characters in arguments except when they're used in path@version arguments in module mode.

The go command should (and does!) allow arguments that contain the @ character if they unambiguously refer to a directory rather than a package import path:

$ go version
go version devel go1.17-1108cbe60 Thu May 6 02:21:55 2021 +0000 linux/amd64

$ mkdir foo@bar

$ cd foo@bar

foo@bar$ go mod init example.com/bar
go: creating new go.mod: module example.com/bar

foo@bar$ cat > main.go
package main

func main() {}

foo@bar$ pwd
/tmp/tmp.eEQh4n8Unb/foo@bar

foo@bar$ go build $(pwd)

foo@bar$ ls
bar  go.mod  main.go

foo@bar$

However, in GOPATH mode when outside of GOPATH/src, the go command synthesizes an import path for the working directory based on its file path. It appears that when the directory includes an @ symbol, that synthesized import path also includes an @ symbol and is then rejected as an argument:

foo@bar$ go list .
example.com/bar

foo@bar$ export GO111MODULE=off

foo@bar$ go list .
_/tmp/tmp.eEQh4n8Unb/foo@bar

foo@bar$ go build .

foo@bar$ go build $(pwd)
package /tmp/tmp.eEQh4n8Unb/foo@bar: can only use path@version syntax with 'go get' and 'go install' in module-aware mode

@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels May 7, 2021
@bcmills bcmills added this to the Backlog milestone May 7, 2021
@bcmills bcmills changed the title cmd/go: build breaks with @ in GOPATH path names cmd/go: GOPATH mode rejects directory arguments outside of GOPATH/src that contain '@' characters May 7, 2021
@metux
Copy link
Author

metux commented May 10, 2021

To avoid confusion and ambiguous commands, go build disallows @ characters in arguments except when they're used in path@version arguments in module mode.

This is a huge problem for automated build engines, if a programming language / compiler imposes restrictions on the source root directories. This also applies to the inherent "src" prefix. That's a heavy violation of layers that make maintaining automatic build infrastructures a pretty hard problem.

In my case, the generic deb/rpm packaging engine (*1) puts the source trees (git repos) into subdirs that may have '@' in their names, when explicit package versions (note: package versions have nothing do to w/ go module versions). If the go compiler (just one of a dozen different compilers used in a distro context) has it's own funny ideas how source directory names may look like, I have to do invasive changes in the packaging engine and the surrounding infrastructures.

This isn't a problem for import paths, since the language spec does not allow @ characters in import declarations. It may be a problem (as in this case) for relative and absolute directory paths, but I think it's better to keep the current strict behavior rather than change anything.

By "import path" you mean the arguments of the import statement ?
These are not of my concern, but the place where the whole code tree is placed in the file system.

I'm confused - this wasn't supposed to be a module name, but a plain path name in GOPATH.

/home/nekrad/src/deb-pkg/pkg/moby-engine@master.git/cmd/dockerd is a directory path. You may be able to build dockerd using its normal package path instead, but it looks like this has been customized enough that I'm not sure what that would be. cmd/dockerd maybe?

I played a lot with GOPATH, but always lead to lots of error messages, some indicate clashes with the standard library (don't recall the exact messages). So, I tried to generate go.mod (via go mod init), which also just works under certain special circumstances. (see below)

The upstream builds docker within a docker container (exactly what I CAN NOT do) and copying around their source tree so /go/src. Without having 'src' prefix in the path name does not seem to work - go wont find the correct sources.
According to the official documentation (*2), that sounds pretty much mandatory - haven't found any documentation on how to do that w/o having that special source tree layout.

Is there any way for preventing any downloads and remote access completely ?

In module-aware mode, go mod vendor will build a vendor directory (fetching what it needs from the network), then you can build offline with -mod=vendor (which may be enabled automatically, depending on the go version in go.mod). There are a few ways to build offline, but that's probably the easiest.

My problem here is 'go mod init'. Documentation suggest that this can be used to create a go.mod from vendor.conf (which does exist in the docker/moby source). BUT: it tries to download a lot of stuff (another thing I CAN NOT have in a distro build context) and uses that stuff instead of from vendor dir - which in turn breaks a lot of things. I can prevent that by pointing GOPATH to an unwritable directory - a very weird hack.

Is there any way to prevent 'go mod init' from doing any downloads and just take what is already there ?
Or is there any other way of creating a go.mod file from the existing sources.

Finally, all I wanna do is building exactly the sources in that tree, using the distro's toolchain (and it's standard library), and no external interactions.

--mtx

*1) https://github.com/metux/deb-pkg
*2) https://golang.org/doc/gopath_code

@jayconrod
Copy link
Contributor

For better or worse, cmd/go is a very opinionated tool. This make it easier for different projects to work together, since strong conventions are enforced. However, if you have a lot of external constraints (sounds like that's the case here), you may run into problems.

It seems like a fairly simple change is possible: cmd/go could allow @ in arguments if they appear to be absolute or relative file paths (starting with ./ or ../). Version suffixes have never been allowed on file paths, so there doesn't seem much risk of confusion here.

@bcmills
Copy link
Contributor

bcmills commented May 10, 2021

Is there any way to prevent 'go mod init' from doing any downloads and just take what is already there ?
Or is there any other way of creating a go.mod file from the existing sources.

You can use replace directives to stitch multiple local modules together into a build (note that you will also need to require every module that provides a directly-imported package or, for Go 1.17 or later, a transitively-imported package).

Then you can set GOPROXY=off in the environment to prevent the go command from downloading other modules.

This sort of stitching-together should be easier using Go 1.17 (which is still under development) with a go.mod file that specifies go 1.17 or later. That will avoid loading go.mod files for module dependencies that don't provide imported packages, which may substantially decrease the number of replace directives that you need in order to produce a fully-local configuration.

@ianlancetaylor
Copy link
Contributor

Just a note that we should never accept paths that start with @ characters, as the system linker and other tools can treat them specially, as response files (https://docs.microsoft.com/en-us/windows/win32/midl/response-files).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

7 participants