Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go build' in module mode rebuilds vendored dependencies in GOROOT #27285

Closed
vincentbernat opened this issue Aug 27, 2018 · 24 comments
Closed
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone

Comments

@vincentbernat
Copy link

What version of Go are you using (go version)?

go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bernat/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/bernat/src/gocode"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/bernat/tmp/reproduce/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build345235277=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Using the following main.go file:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	helloHandler := func(w http.ResponseWriter, req *http.Request) {
		io.WriteString(w, "Hello, world!\n")
	}

	http.HandleFunc("/hello", helloHandler)
	http.ListenAndServe(":8080", nil)
}

And the following commands:

go mod init anything
go build -i -o /dev/null

I get:

go: finding golang.org/x/text v0.3.0
go build golang_org/x/crypto/cryptobyte/asn1: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/crypto/cryptobyte/asn1.a: permission denied
go build golang_org/x/crypto/poly1305: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/crypto/poly1305.a: permission denied
go build golang_org/x/crypto/internal/chacha20: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/crypto/internal/chacha20.a: permission denied
go build golang_org/x/net/dns/dnsmessage: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/net/dns/dnsmessage.a: permission denied
go build golang_org/x/crypto/curve25519: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/crypto/curve25519.a: permission denied
go build golang_org/x/text/transform: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/text/transform.a: permission denied
go build golang_org/x/text/unicode/bidi: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/text/unicode/bidi.a: permission denied
go build golang_org/x/net/http2/hpack: open /usr/lib/go-1.11/pkg/linux_amd64/vendor/golang_org/x/net/http2/hpack.a: permission denied

What did you expect to see?

go build shouldn't try to erase stuff shipped with Go. This only seems to happen for vendorized stuff. Non-vendorized stuff don't get compiled. Without -i, this doesn't happen. The provided command is the one issued by a flycheck plugin for Emacs.

@vincentbernat vincentbernat changed the title go build 1.11 tries to rebuild some vendorized dependencies go build 1.11 tries to rebuild shipped vendorized dependencies Aug 27, 2018
@bradfitz
Copy link
Contributor

I think this is a duplicate of #26988.

@ianlancetaylor ianlancetaylor changed the title go build 1.11 tries to rebuild shipped vendorized dependencies cmd/go: go build 1.11 tries to rebuild shipped vendorized dependencies Aug 28, 2018
@ianlancetaylor
Copy link
Contributor

I thought #26998 was fixed by https://golang.org/cl/130138, although I see that the CL does not mention the issue.

This should be fixed, but, in the meantime, don't use go build -i. Now that we have the build cache, go build -i isn't useful.

@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 28, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Aug 28, 2018
@FiloSottile
Copy link
Contributor

@gopherbot please file this for backport against 1.11. This is a regression.

@gopherbot
Copy link

Backport issue(s) opened: #27389 (for 1.11).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@ianlancetaylor
Copy link
Contributor

Honestly I'm not sure it helps to file an issue for backport when we don't have a fix yet.

@FiloSottile
Copy link
Contributor

@ianlancetaylor the point of opening a backport issue is that it will not get closed when a "Fixes" patch gets merged. We used to lose track of things that way. Opening it immediately avoids having to track an extra "should backport but don't have a fix yet" state separately, and having to take a GitHub action when it transitions to "it has a fix now" by a Gerrit action.

releasebot pushes all CherryPickCandidate issues to the next minor release, so we don't even have to manage them, making them basically zero overhead.

@ianlancetaylor
Copy link
Contributor

But the downside is that when we are looking at a minor release, we have to look at these issues again and again, each time going back to the original issue, and working out that it has not been fixed and there is nothing to backport.

Since it is gopherbot that is closing the issues, we don't have to be hostage to it. I suggest that if gopherbot sees that the issue has a milestone of an earlier minor release, that the issue not be closed when the patch is committed to tip.

@FiloSottile
Copy link
Contributor

@ianlancetaylor It's GitHub, not gopherbot, that closes issues when a patch is submitted to master. (gopherbot only takes care of merges to release branches.) The other issue is when something needs backport to two releases, which we also used to lose track of. It sounds like a gopherbot feature that only applies the CherryPickCandidate label when the main issue is closed would solve this? In any case, we are OT, so if you'd like to propose that or a different change, please open a separate issue.

@FiloSottile
Copy link
Contributor

More details about this rebuilding issue at #27482.

@bcmills bcmills self-assigned this Sep 6, 2018
jessepeterson added a commit to jessepeterson/micromdm that referenced this issue Sep 7, 2018
groob pushed a commit to micromdm/micromdm that referenced this issue Sep 7, 2018
@godwhoa
Copy link

godwhoa commented Sep 7, 2018

For anyone running into this issue, try the current master. Fixed in my case.

➜  bug27285 go version
go version go1.11rc1 linux/amd64
➜  bug27285 go build -i
go build golang_org/x/crypto/poly1305: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/poly1305.a: permission denied
go build golang_org/x/crypto/curve25519: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/curve25519.a: permission denied
go build golang_org/x/text/transform: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/text/transform.a: permission denied
go build golang_org/x/text/unicode/bidi: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/text/unicode/bidi.a: permission denied
go build golang_org/x/net/http2/hpack: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/net/http2/hpack.a: permission denied
go build golang_org/x/net/dns/dnsmessage: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/net/dns/dnsmessage.a: permission denied
go build golang_org/x/crypto/cryptobyte/asn1: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/cryptobyte/asn1.a: permission denied
go build golang_org/x/crypto/internal/chacha20: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/internal/chacha20.a: permission denied

➜  bug27285 go version
go version devel +ceb7745 Fri Sep 7 18:58:57 2018 +0000 linux/amd64
➜  bug27285 go build -i
# anything
./main.go:4:2: imported and not used: "fmt"
./main.go:10:3: undefined: io

@bcmills
Copy link
Contributor

bcmills commented Sep 7, 2018

@godwhoa, I see traces there for rc1 and devel. Have you tried the actual 1.11 release? (I believe some module bugs were fixed in between rc1 and rc2.)

@godwhoa
Copy link

godwhoa commented Sep 7, 2018

➜  bug27285 go version
go version go1.11 linux/amd64
➜  bug27285 go build -i
go build golang_org/x/crypto/cryptobyte/asn1: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/cryptobyte/asn1.a: permission denied
go build golang_org/x/net/dns/dnsmessage: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/net/dns/dnsmessage.a: permission denied
go build golang_org/x/crypto/curve25519: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/curve25519.a: permission denied
go build golang_org/x/crypto/internal/chacha20: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/internal/chacha20.a: permission denied
go build golang_org/x/crypto/poly1305: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/crypto/poly1305.a: permission denied
go build golang_org/x/text/transform: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/text/transform.a: permission denied
go build golang_org/x/net/http2/hpack: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/net/http2/hpack.a: permission denied
go build golang_org/x/text/unicode/bidi: open /usr/local/go/pkg/linux_amd64/vendor/golang_org/x/text/unicode/bidi.a: permission denied

@bcmills
Copy link
Contributor

bcmills commented Sep 7, 2018

Thanks!

@bcmills
Copy link
Contributor

bcmills commented Sep 13, 2018

My first clue so far is that the effective ImportPath of the vendored dependencies varies between GOPATH mode and module mode: in GOPATH mode they have the prefix vendor/, where in module mode they do not.

I suspect that that results in different artifacts and/or different cache keys.

@bcmills
Copy link
Contributor

bcmills commented Sep 13, 2018

(And I have a test case that replicates this.)

prydie added a commit to oracle/oci-cloud-controller-manager that referenced this issue Oct 28, 2018
owainlewis pushed a commit to oracle/oci-cloud-controller-manager that referenced this issue Oct 29, 2018
* Run VP E2E tests in CI
* Fix FSS not found bug
* Use mount target in AD2 in FSS E2E
* Use cloud-provider config in volume provisioner
* Don't use go's -i flag anymore. See: golang/go#27285
* Use cloud-provider config in flexvolume driver
@gopherbot
Copy link

Change https://golang.org/cl/147443 mentions this issue: vendor/golang_org: move to internal/golang_org

@bcmills bcmills changed the title cmd/go: go build 1.11 tries to rebuild shipped vendorized dependencies cmd/go: go build 1.11 tries to rebuild vendored dependencies in GOROOT Nov 15, 2018
@bcmills bcmills changed the title cmd/go: go build 1.11 tries to rebuild vendored dependencies in GOROOT cmd/go: 'go build' in module mode rebuilds vendored dependencies in GOROOT Nov 15, 2018
wusendong added a commit to wusendong/bk-cmdb that referenced this issue Nov 21, 2018
hectorj2f pushed a commit to hectorj2f/virtual-kubelet that referenced this issue Apr 25, 2019
hectorj2f pushed a commit to hectorj2f/virtual-kubelet that referenced this issue May 2, 2019
hectorj2f pushed a commit to hectorj2f/virtual-kubelet that referenced this issue May 8, 2019
hectorj2f pushed a commit to hectorj2f/virtual-kubelet that referenced this issue May 9, 2019
hectorj2f pushed a commit to hectorj2f/virtual-kubelet that referenced this issue May 9, 2019
hectorj2f pushed a commit to hectorj2f/virtual-kubelet that referenced this issue May 9, 2019
hectorj2f pushed a commit to hectorj2f/virtual-kubelet that referenced this issue May 11, 2019
hectorj2f pushed a commit to hectorj2f/virtual-kubelet that referenced this issue May 26, 2019
cpuguy83 pushed a commit to cpuguy83/virtual-kubelet that referenced this issue Jun 18, 2019
siggy added a commit to linkerd/linkerd2 that referenced this issue Jul 25, 2019
more info:
golang/go#27285

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Sr4l added a commit to Sr4l/gitea that referenced this issue Aug 7, 2019
this flag is not needed for Go versions >=1.10 and creates problems
while building. see golang/go#27285 (comment)
lunny pushed a commit to go-gitea/gitea that referenced this issue Aug 7, 2019
this flag is not needed for Go versions >=1.10 and creates problems
while building. see golang/go#27285 (comment)
@golang golang locked and limited conversation to collaborators Nov 29, 2019
@rsc rsc unassigned bcmills Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Development

No branches or pull requests

10 participants