cmd/go: build cache not safe for concurrent builds #43645
Comments
seankhliao
changed the title
seankhliao
added
the
NeedsInvestigation
|
@zimmski, are the errors always for that specific source file ( |
Could you give the output of running |
Will take a look at a view pipelines and update this comment with what i find: i see "can't open import:" which both reference "/usr/local/go" which is the Go archive from https://golang.org/dl/ unpacked. So far this seems isolated there. And other errors are of the form "can't import facts for package" which are StdLib packages so far. Here is a full list of one such failing build. One thing to note is that this failure comes from a subtest where we compile 10 other test cases in parallel with different literals in the code. One fails but the others do not. |
Of course, sorry that i posted my local one. It obviously does not make much sense to do that. The follow output is directly from one of the failing CI pipelines. Edit: update the issue description. |
|
I went now through some failures and it looks like that the failures even though they are everywhere in our build always come from the "net" package. I will add now a new build with the GOCACHE again used from the node itself. However, since now all branches do not use that cache anymore i am wondering if this happens with that branch at all. |
|
In the configuration that fails, how is the Do the same commands exhibit the same failure mode when run outside of Docker? (I'm trying to rule out the possibility of a bug introduced by a Docker filesystem hook somewhere, or at least distill down the bug to something that we can reproduce independent of Docker.) |
bcmills
added
the
WaitingForInfo
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
What version of Go are you using (
go version)?1.15.6 before that we used 1.12.*
Does this issue reproduce with the latest release?
1.15.6 is latest, so yes.
What operating system and processor architecture are you using (
go env)?GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS="-v -trimpath"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/builds/symflower/symflower/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/builds/symflower/symflower"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang-11"
CXX="clang++-11"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build695024912=/tmp/go-build -gno-record-gcc-switches"
What did you do?
(This issue is created because of #40461 where i asked if should create an issue or if it fits the referenced one.)
Gist: Parallel running "go build" processes that use the same GOCACHE directory lead to errors such as
crypto/tls\n/usr/local/go/src/crypto/tls/common.go:22:2: can't open import: \"net\": open /cache/go-build/87/8713176dd630051cd4b09a7fa534da9a291da82ba5c84c02966f3ee18dc3a206-d: no such file or directory.Longer: We switched from Go 1.12 to 1.15.6 for one project and are now running into constant non-deterministic build/test problems because of what I think is that Go's cache is not concurrent-safe. Basically our CI does one "go build" over a whole branch and uses one common GOCACHE per CI node. Hence, multiple CI jobs could write to the same cache directory. That part was never a problem even with the newer Go version (i do not know why). After the "go build" CI job multiple jobs run in parallel to do linting and testing (via "go test"). And these run into constant issues e.g.
crypto/tls\n/usr/local/go/src/crypto/tls/common.go:22:2: can't open import: \"net\": open /cache/go-build/87/8713176dd630051cd4b09a7fa534da9a291da82ba5c84c02966f3ee18dc3a206-d: no such file or directory.Some additional information that might be interesting:
What did you expect to see?
We expected that all builds can use the same cache directory at the same time and the builds that worked with an older Go version still work with the latest version.
What did you see instead?
Hundreds of errors such as
crypto/tls\n/usr/local/go/src/crypto/tls/common.go:22:2: can't open import: \"net\": open /cache/go-build/87/8713176dd630051cd4b09a7fa534da9a291da82ba5c84c02966f3ee18dc3a206-d: no such file or directory. That lead to a failing build. We never had this problem with Go 1.12. The cache just worked. With 1.15.6 it does not. Basically because of this problem we are back to having no cache at all.We do not run "go clean" at all, so this is hopefully not #31948.
Our current workaround:
The text was updated successfully, but these errors were encountered: