cmd/dist: no way to disable buildvcs stamping during bootstrap #54852

williamh · 2022-09-03T00:17:00Z

Hi,

I am the packager for Go on Gentoo linux.

We have an on-going issue with the VCS stamping which has appeared in another way.

Since go 1.17 and older are not in our tree any longer, we are able to add "-buildvcs=false" to our default build flags for go-based packages to prevent vcs stamping from breaking our package builds.

However, if go is being bootstrapped and the bootstrap binaries are newer than version 1.18, it is possible for vcs stamping to crash the build as shown below.

What version of Go are you using (`go version`)?

$ go version
go version go1.18.5 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/william/.cache/go-build"
GOENV="/home/william/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/william/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/william/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="x86_64-pc-linux-gnu-gcc"
CXX="x86_64-pc-linux-gnu-g++"
CGO_ENABLED="1"
GOMOD="/home/william/build/go/src/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build162734369=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ cd
$ mkdir build
$ cd build
$ git init
$ chmod 000 .git
$ tar -xf /path/to/go1.18.5.tar.gz
$ cd go/src
$ export GOROOT_BOOTSTRAP=/usr/lib/go
$ ./all.bash

What did you expect to see?

Successful build.

What did you see instead?

Building Go cmd/dist using /usr/lib/go. (go1.18.5 linux/amd64)
Building Go toolchain1 using /usr/lib/go.
error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping.
go tool dist: FAILED: /usr/lib/go/bin/go install -gcflags=-l -tags=math_big_pure_go compiler_bootstrap bootstrap/cmd/...: exit status 1

The text was updated successfully, but these errors were encountered:

williamh · 2022-09-03T00:25:28Z

This was reported by a user here, but I wanted to reproduce it outside of our package manager before I reported it to you.

seankhliao · 2022-09-03T09:40:33Z

Note that we currently have 1.17 as our minimum bootstrap version.

If you only want to support 1.18+ as bootstrap you could add -buildvcs=false to the flags passed in src/cmd/dist/buildtool.go

bcmills · 2022-09-06T15:38:43Z

Hmm. As of CL 398855 (backported to Go 1.18.2), the default behavior is to stamp VCS information when the VCS tool is present, and to omit it when the VCS tool is not found.

Moreover, we intentionally avoid stamping VCS information for programs found in the standard library:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/load/pkg.go;l=2431;drc=e64c87157d1e8fbc512a670b8c0af8abc3afa7c1

So one option might be to change that condition from !p.Standard to !p.Goroot, but then that would also omit VCS information from binaries in other GOROOT modules (such as GOROOT/misc).

bcmills · 2022-09-06T15:45:55Z

That said, looking at https://bugs.gentoo.org/836261 I don't understand why this would affect Go bootstrapping.

The issue reported there is that the VCS stamping is somehow coming up with /gentoo/fossil_repo/var.fossil as the VCS metadata to stamp into the binaries, but that should not be possible: cmd/dist/buildtool.go places the bootstrap module in $GOROOT/pkg/bootstrap/src/bootstrap, in which case the presence of the $GOROOT/.git directory should cause that (not /gentoo/fossil_repo/var.fossil) to be treated as the repo root.

So I guess a contributing factor is that $GOROOT/.git is stripped from the repo during Gentoo builds?

seankhliao · 2022-09-06T16:20:06Z

I believe Gentoo builds using a source tarball rather than a vcs checkout

williamh · 2022-09-09T01:04:36Z

That's correct, we build using a source tarball.

williamh · 2022-09-17T18:07:24Z

I confirmed that the following sequence produces a successful build, so this is definitely caused by building from a release tarball instead of the git repository.

$ cd
$ mkdir build
$ cd build
$ git init
$ chmod 000 .git
$ git clone git@github.com:golang/go
$ cd go
$ git checkout go1.19.1
$ cd src
$ export GOROOT_BOOTSTRAP=/usr/lib/go
$ ./all.bash

gopherbot · 2022-09-21T15:47:44Z

Change https://go.dev/cl/432435 mentions this issue: cmd/go: omit VCS stamping during bootstrap

paralin · 2023-07-27T21:36:36Z

Hi @williamh @bcmills - has there been a regression with this in Go 1.20?

In Buildroot we have a build failure due to not having the .git directory in the Go sources: https://gitlab.com/buildroot.org/buildroot/-/jobs/4725186525

Building Go toolchain1 using /builds/buildroot.org/buildroot/test-output/TestMender/host/lib/go-1.19.10.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
go tool dist: FAILED: /builds/buildroot.org/buildroot/test-output/TestMender/host/lib/go-1.19.10/bin/go install -tags=math_big_pure_go compiler_bootstrap purego bootstrap/cmd/...: exit status 1

It seems that -buildvcs=false is not set in the "go install" command regardless of isGitRepo():

https://github.com/golang/go/blob/go1.20.6/src/cmd/dist/buildtool.go#L205

Applying two patches fixes the issue within Buildroot:

Set -buildvcs=false in the "go install" command if not isGitDir(): paralin@1e0a65f
Add an environment variable GOBOOTSTRAP_BUILDVCS to force isGitDir() to return false: paralin@7e65e8e

This worked fine with go1.19.x but regressed with go1.20.x.

When compiling without a Git repository, "go install" returns an error: error obtaining VCS status: exit status 128 Use -buildvcs=false to disable VCS stamping. Set -buildvcs=false in go install when isGitDir() returns false. Fixes: golang#61620 Related: golang#54852 Signed-off-by: Christian Stewart <christian@aperture.us>

Add an environment variable to bypass the isGitRepo() check and force it to return false, thereby also setting -buildvcs=false during "go install." This is useful in situations where the .git directory may be present but we still want to force cmd/dist to ignore it. Fixes: golang#61620 Related: golang#54852 Signed-off-by: Christian Stewart <christian@aperture.us>

When compiling without a Git repository, "go install" returns an error: error obtaining VCS status: exit status 128 Use -buildvcs=false to disable VCS stamping. Set -buildvcs=false in go install when isGitDir() returns false. Fixes: golang#61620 Related: golang#54852

Add an environment variable to bypass the isGitRepo() check and force it to return false, thereby also setting -buildvcs=false during "go install." This is useful in situations where the .git directory may be present but we still want to force cmd/dist to ignore it. Fixes: golang#61620 Related: golang#54852

When compiling without a Git repository, "go install" returns an error: error obtaining VCS status: exit status 128 Use -buildvcs=false to disable VCS stamping. Set -buildvcs=false in go install when isGitDir() returns false. Fixes: golang#61620 Related: golang#54852

Add an environment variable to bypass the isGitRepo() check and force it to return false, thereby also setting -buildvcs=false during "go install." This is useful in situations where the .git directory may be present but we still want to force cmd/dist to ignore it. Fixes: golang#61620 Related: golang#54852

When compiling without a Git repository, "go install" returns an error: error obtaining VCS status: exit status 128 Use -buildvcs=false to disable VCS stamping. Set -buildvcs=false in go install when isGitDir() returns false. Fixes: golang#61620 Related: golang#54852

Add an environment variable to bypass the isGitRepo() check and force it to return false, thereby also setting -buildvcs=false during "go install." This is useful in situations where the .git directory may be present but we still want to force cmd/dist to ignore it. Fixes: golang#61620 Related: golang#54852

gopherbot · 2023-07-27T22:30:49Z

Change https://go.dev/cl/513835 mentions this issue: cmd/dist: set buildvcs=false if not within a git repository

When building go-bootstrap as part of the make.bash process, the cmd/dist invokes the bootstrap Go compiler to build the go_bootstrap tool: ${GOROOT_BOOTSTRAP}/bin/go install -tags=math_big_pure_go compiler_bootstrap purego bootstrap/cmd/... If there is an invalid .git directory in a parent of ${GOROOT_BOOTSTRAP}, make.bash will fail. Reproduction of the issue: mkdir go-issue-61620 cd ./go-issue-61620 wget https://go.dev/dl/go1.19.11.src.tar.gz mkdir go-bootstrap tar -xf go1.19.11.src.tar.gz -C ./go-bootstrap --strip-components=1 cd ./go-bootstrap/src/ bash make.bash cd ../../ wget https://go.dev/dl/go1.20.6.src.tar.gz mkdir go tar -xf go1.20.6.src.tar.gz -C ./go/ --strip-components=1 printf "gitdir: ../../does/not/exist/.git" > ./.git cd ./go/src/ GOROOT_BOOTSTRAP=$(pwd)/../../go-bootstrap/ bash make.bash The build fails with the following error: Building Go toolchain1 using [snip]/go-1.19.10. error obtaining VCS status: exit status 128 Use -buildvcs=false to disable VCS stamping. go tool dist: FAILED: [snip]/go-1.19.10/bin/go install -tags=math_big_pure_go \ compiler_bootstrap purego bootstrap/cmd/...: exit status 1 This change unconditionally sets -buildvcs=false when compiling go-bootstrap. We don't need the revision information in those binaries anyway. Setting this flag was previously not done as we were unsure if the go-bootstrap compiler would be new enough to support the buildvcs build flag. However, since Go 1.20.x, Go 1.19.x is the minimum version for go-bootstrap, and supports -buildvcs=false. We can now set -buildvcs=false without worrying about compatibility. Related: golang#54852 Fixes: golang#61620

When building go-bootstrap as part of the make.bash process, the cmd/dist invokes the bootstrap Go compiler to build the go_bootstrap tool: ${GOROOT_BOOTSTRAP}/bin/go install -tags=math_big_pure_go compiler_bootstrap purego bootstrap/cmd/... If there is an invalid .git directory in a parent of ${GOROOT_BOOTSTRAP}, make.bash will fail. Reproduction of the issue: mkdir go-issue-61620 cd ./go-issue-61620 wget https://go.dev/dl/go1.19.11.src.tar.gz mkdir go-bootstrap tar -xf go1.19.11.src.tar.gz -C ./go-bootstrap --strip-components=1 cd ./go-bootstrap/src/ bash make.bash cd ../../ wget https://go.dev/dl/go1.20.6.src.tar.gz mkdir go tar -xf go1.20.6.src.tar.gz -C ./go/ --strip-components=1 printf "gitdir: ../../does/not/exist/.git" > ./.git cd ./go/src/ GOROOT_BOOTSTRAP=$(pwd)/../../go-bootstrap/ bash make.bash The build fails with the following error: Building Go toolchain1 using [snip]/go-1.19.10. error obtaining VCS status: exit status 128 Use -buildvcs=false to disable VCS stamping. go tool dist: FAILED: [snip]/go-1.19.10/bin/go install -tags=math_big_pure_go \ compiler_bootstrap purego bootstrap/cmd/...: exit status 1 This change unconditionally sets -buildvcs=false when compiling go-bootstrap. We don't need the revision information in those binaries anyway. Setting this flag was previously not done as we were unsure if the go-bootstrap compiler would be new enough to support the buildvcs build flag. Since Go 1.20.x, Go 1.19.x is the minimum version for go-bootstrap, and supports -buildvcs=false. We can now set -buildvcs=false without worrying about compatibility. Related: golang#54852 Fixes: golang#61620

seankhliao changed the title ~~vcs stamping interfeers with bootstrapping using go 1.18.5 or newer~~ cmd/dist: no way to disable buildvcs stamping during bootstrap Sep 3, 2022

seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 3, 2022

seankhliao added this to the Unplanned milestone Sep 3, 2022

bcmills modified the milestones: Unplanned, Backlog Sep 6, 2022

bcmills added the help wanted label Sep 6, 2022

bcmills self-assigned this Sep 9, 2022

bcmills modified the milestones: Backlog, Go1.20 Sep 9, 2022

bcmills removed the help wanted label Sep 9, 2022

gopherbot closed this as completed in 42a4620 Sep 22, 2022

paralin mentioned this issue Jul 27, 2023

cmd/dist: fails with buildvcs=false errors if invalid .git exists in parent of GOROOT_BOOTSTRAP #61620

Open

paralin mentioned this issue Jul 27, 2023

cmd/dist: set buildvcs=false when building go-bootstrap #61621

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/dist: no way to disable buildvcs stamping during bootstrap #54852

cmd/dist: no way to disable buildvcs stamping during bootstrap #54852

williamh commented Sep 3, 2022

williamh commented Sep 3, 2022

seankhliao commented Sep 3, 2022

bcmills commented Sep 6, 2022

bcmills commented Sep 6, 2022

seankhliao commented Sep 6, 2022

williamh commented Sep 9, 2022

williamh commented Sep 17, 2022

gopherbot commented Sep 21, 2022

paralin commented Jul 27, 2023 •

edited

gopherbot commented Jul 27, 2023

cmd/dist: no way to disable buildvcs stamping during bootstrap #54852

cmd/dist: no way to disable buildvcs stamping during bootstrap #54852

Comments

williamh commented Sep 3, 2022

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

williamh commented Sep 3, 2022

seankhliao commented Sep 3, 2022

bcmills commented Sep 6, 2022

bcmills commented Sep 6, 2022

seankhliao commented Sep 6, 2022

williamh commented Sep 9, 2022

williamh commented Sep 17, 2022

gopherbot commented Sep 21, 2022

paralin commented Jul 27, 2023 • edited

gopherbot commented Jul 27, 2023

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

paralin commented Jul 27, 2023 •

edited