cmd/go: go tool dist test fail inside repo #26980

iwdgo · 2018-08-14T07:58:20Z

When building Go, the run.bat part cannot be run correctly. It seems related to GOTOOLDIR which is incorrectly read or updated. On GOPATH, no issue occurs. Inside the repo, problems occur.
run.ba* checks that it is inside the repo (go\src).
After investigating, I suppose that this is why go tool dist cannot run inside the repo.

What version of Go are you using (`go version`)?

C:\Users{username is removed below}\Documents\Google>go version
go version go1.11rc1 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

C:\Users\{}\Documents\Google>go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\{}\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\{}\Documents\Google\
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\{}\AppData\Local\Temp\go-build523053410=/tmp/go-build -gno-record-gcc-switches

What did you do?

C:\Users\{}\Documents\Google>set GOTOOLDIR=C:\Users\{}\Documents\Google\golang\go\pkg\tool\windows_amd64

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

C:\Users\{}\Documents\Google>go env GOTOOLDIR
C:\Users\{}\Documents\Google\golang\go\pkg\tool\windows_amd64

What did you see instead?

C:\Users\{}\Documents\Google>go env GOTOOLDIR
C:\Go\pkg\tool\windows_amd64

C:\Users\{}\Documents\Google>set GO
GOPATH=C:\Users\{}\Documents\Google\
GOROOT=C:\Go\
GOTOOLDIR=C:\Users\{}\Documents\Google\golang\go\pkg\tool\windows_amd64

C:\Users\Costa\Documents\Google>set GOTOOLDIR=

C:\Users\Costa\Documents\Google>set GO
GOPATH=C:\Users\Costa\Documents\Google\
GOROOT=C:\Go\

C:\Users\Costa\Documents\Google>go env GOTOOLDIR
C:\Go\pkg\tool\windows_amd64

If you apply the same recipe to GOTMPDIR for instance, everything occurs as expected.
make.bat succeeds and displays the installation message. all.bat fails on the run.bat part with:

(lines removed)
Building packages and commands for windows/amd64.

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Users\\Costa\\Documents\\Google\\golang\\go\\go.exe": file does not exist

Starting run.bat with and w/o rebuild fails similarly:

C:\Users\Costa\Documents\Google\golang\go\src>run.bat --no-rebuild

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Go\\go.exe": file does not exist
C:\Users\{}\Documents\Google\golang\go\src>run.bat --no-rebuild --no-local

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Go\\go.exe": file does not exist
C:\Users\{}\Documents\Google\golang\go\src>set GO
GOARCH=amd64
GOBUILDFAIL=1
GOCACHE=C:\Users\{}\AppData\Local\go-build
GOEXE=.exe
GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\{}\AppData\Local\Temp\go-build370009818=/tmp/go-build -gno-record-gcc-switches
GOHOSTARCH=amd64
GOHOSTOS=windows
GOOS=windows
GOPATH=C:\Users\{username}\go
GOROOT=C:\Go
GOROOT_BOOTSTRAP=C:\Go\
GOTOOLDIR=C:\Go\pkg\tool\windows_amd64

After failure, GOPATH is set to a directory that does not exist.

I updated the base installation using rc1. The rest is run on tip.
I gave it a try but could not locate the GOTOOLDIR environment variable handling.

The text was updated successfully, but these errors were encountered:

agnivade · 2018-08-14T09:13:25Z

GOTOOLDIR is not meant to be updated.

Please see #26886 and #10264

davecheney · 2018-08-14T09:19:36Z

Closing as duplicate of #26886.

Unlike many projects on GitHub, the Go project does not use its bug tracker for general discussion or asking questions. We only use our bug tracker for tracking bugs and tracking proposals going through the Proposal Process.

Please see https://golang.org/wiki/Questions for good places to ask questions.

iwdgo · 2018-08-14T09:19:55Z

Thanks for the confirmation. This is not explicit in the documentation of go env. It would be an improvement to confirm that GOTOOLDIR appears but cannot be changed to avoid any further issue.

The issue is then that run.bat fails and go tool dist test fails when run inside the repo. I updated the title accordingly. I still suspect that an invalid GOTOOLDIR is used or as underlined above the GOPATH is incorrectly set.

iwdgo · 2018-08-14T09:22:13Z

@davecheney I understand your point but go tool dist test still fails. I updated the title accordingly.

davecheney · 2018-08-14T09:22:52Z

It looks like the root failure is

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\Users\Costa\Documents\Google\golang\go\go.exe": file does not exist

This is not an expected error, and is not seen on our build dashboard or other users in the wild. Could some program you have installed on your computer, possibly an AV program, be interfering with the Go build process?

iwdgo · 2018-08-14T09:30:03Z

I build regularly and the issue started early last week. I re-installed Go with rc1. I clean where there are traces of the build. I have no alarm from AV and no error during build.
I confirm that out of the repo, the command succeeds. I can also run the build from inside go\src\bin.
I understand your argument but It might be related to Windows 10.
As you can see after failure, GOPATH is set to a non-existent directory.
I found this issue #18583 which is about the build process of Windows and revolves around GOPATH and USERPROFILE.

davecheney · 2018-08-14T10:00:20Z

@iwdgo building Go from source does not rely on GOPATH or being inside a repo.

Can you please post the complete output from your build invocation. Please do not update the issue description, that makes it hard for people to follow the flow of the issue as it evolves. Thank you.

iwdgo · 2018-08-14T10:11:40Z

Sorry for jumping the gun on the title. My focus is indeed this command inside the repo.
I re-started by computer as I can't remember when was the last time. The output of the command is identical.
GOPATH invalid value leads me to think that something occurs an intermediate environment variable.
The output of all.bat on tip this morning is as follows. I'm not sure that this is what you ask.

C:\Users\{}\Documents\Google\golang\go\src>go clean -cache

C:\Users\{}\Documents\Google\golang\go\src>time
The current time is:  9:03:17,91
Enter the new time:

C:\Users\{}\Documents\Google\golang\go\src>all.bat
Building Go cmd/dist using C:\Go\
Building Go toolchain1 using C:\Go\.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for windows/amd64.

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Users\\{}\\Documents\\Google\\golang\\go\\go.exe": file does not exist

C:\Users\{}\Documents\Google\golang\go\src>

C:\Users\{}\Documents\Google\golang\go\pkg\tool\windows_amd64 14-08-18 9:06
C:\Go\pkg\tool\windows_amd64 14-08-18 8:41

davecheney · 2018-08-14T10:25:38Z

Thanks for your update. All I can suggest is something on your computer is blocking C:\Users\{}\Documents\Google\golang\go\go.exe" from being written. Not being a windows user myself I understand how frustrating it must be to be told "please turn off your AV", but this is often the cause of issues like this.

iwdgo · 2018-08-14T10:38:07Z

I believe that it is not trying to update to trying to execute because of the "exec:" word but I could not locate the related source code. There is never been a go.exe in the \go directory but only in the go\bin. And the installation has already occurred from what I can read. I confirm that make.bat finishes w/o error.
The structure of go directory is as expected:

14-08-18  09:11    <DIR>          .
14-08-18  09:11    <DIR>          ..
24-03-18  20:32               347 .gitattributes
30-03-18  21:01               270 .gitcookies
22-07-18  14:59    <DIR>          .github
24-03-18  20:32               928 .gitignore
24-07-18  12:59    <DIR>          api
13-08-18  16:10            55.284 AUTHORS
14-08-18  09:10    <DIR>          bin
24-03-18  20:32             1.339 CONTRIBUTING.md
13-08-18  16:10            71.070 CONTRIBUTORS
13-08-18  16:10    <DIR>          doc
24-03-18  20:32             5.686 favicon.ico
24-03-18  20:32    <DIR>          lib
24-03-18  20:32             1.479 LICENSE
22-07-18  14:54    <DIR>          misc
24-03-18  20:32             1.303 PATENTS
14-08-18  09:11    <DIR>          pkg
22-07-18  14:59             1.607 README.md
24-03-18  20:32                26 robots.txt
14-08-18  09:20    <DIR>          src
04-08-18  11:04    <DIR>          test
14-08-18  09:11                48 VERSION.cache

and go\bin:

 Directory of C:\Users\{}\Documents\Google\golang\go\bin

14-08-18  09:10    <DIR>          .
14-08-18  09:10    <DIR>          ..
14-08-18  09:13        12.742.656 go.exe
14-08-18  09:13         3.483.648 gofmt.exe

And from inside go\bin, everything occurs as expected:

C:\Users\Costa\Documents\Google\golang\go\bin>go version
go version devel +4fc7b93ad7 Mon Aug 13 19:04:16 2018 +0000 windows/amd64

davecheney · 2018-08-14T10:42:29Z

Can you please try to eliminate any local issues by unsettling any environment variable starting with GO

…

On 14 Aug 2018, at 20:38, Constantin Konstantinidis ***@***.***> wrote: I believe that it is not trying to update to trying to execute because of the "exec:" word but I could not locate the related source code. There is never been a go.exe in the \go directory but only in the go\bin. And the installation has already occurred from what I can read. I confirm that make.bat finishes w/o error. The structure of go directory is as expected: 14-08-18 09:11 <DIR> . 14-08-18 09:11 <DIR> .. 24-03-18 20:32 347 .gitattributes 30-03-18 21:01 270 .gitcookies 22-07-18 14:59 <DIR> .github 24-03-18 20:32 928 .gitignore 24-07-18 12:59 <DIR> api 13-08-18 16:10 55.284 AUTHORS 14-08-18 09:10 <DIR> bin 24-03-18 20:32 1.339 CONTRIBUTING.md 13-08-18 16:10 71.070 CONTRIBUTORS 13-08-18 16:10 <DIR> doc 24-03-18 20:32 5.686 favicon.ico 24-03-18 20:32 <DIR> lib 24-03-18 20:32 1.479 LICENSE 22-07-18 14:54 <DIR> misc 24-03-18 20:32 1.303 PATENTS 14-08-18 09:11 <DIR> pkg 22-07-18 14:59 1.607 README.md 24-03-18 20:32 26 robots.txt 14-08-18 09:20 <DIR> src 04-08-18 11:04 <DIR> test 14-08-18 09:11 48 VERSION.cache and go\bin: Directory of C:\Users\{}\Documents\Google\golang\go\bin 14-08-18 09:10 <DIR> . 14-08-18 09:10 <DIR> .. 14-08-18 09:13 12.742.656 go.exe 14-08-18 09:13 3.483.648 gofmt.exe And from inside go\bin, everything occurs as expected: C:\Users\Costa\Documents\Google\golang\go\bin>go version go version devel +4fc7b93ad7 Mon Aug 13 19:04:16 2018 +0000 windows/amd64 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

iwdgo · 2018-08-14T10:49:45Z

A couple of details to confirm that the build works as go tool dist test -h and go tool dist -list both work as expected.
Here is the outcome of your suggestion:

C:\Users\Costa\Documents\Google\golang\go\src>set GOPATH=

C:\Users\Costa\Documents\Google\golang\go\src>set GOROOT=

C:\Users\Costa\Documents\Google\golang\go\src>
C:\Users\Costa\Documents\Google\golang\go\src>
C:\Users\Costa\Documents\Google\golang\go\src>go tool dist test
go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Users\\Costa\\Documents\\Google\\golang\\go\\go.exe": file does not exist

C:\Users\Costa\Documents\Google\golang\go\src>set GO
Environment variable GO not defined

The last message confirms that there are no environment variables set which have a name starting with GO and I did the same for CGO. I add go env to confirm that values after failure. Thanks for keeping the issue open.

C:\Users\Costa\Documents\Google\golang\go\src>go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Costa\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\Costa\go
set GOPROXY=
set GORACE=
set GOROOT=C:\Users\Costa\Documents\Google\golang\go
set GOTMPDIR=
set GOTOOLDIR=C:\Users\Costa\Documents\Google\golang\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\Costa\AppData\Local\Temp\go-build408101810=/tmp/go-build -gno-record-gcc-switches

davecheney · 2018-08-14T10:57:39Z

I’m going to butt out now, my input Inst useful as I can’t debug this without access to a windows machine.

…

On 14 Aug 2018, at 20:49, Constantin Konstantinidis ***@***.***> wrote: A couple of details to confirm that the build works as go tool dist test -h and go tool dist -list both work as expected. Here is the outcome of your suggestion: C:\Users\Costa\Documents\Google\golang\go\src>set GOPATH= C:\Users\Costa\Documents\Google\golang\go\src>set GOROOT= C:\Users\Costa\Documents\Google\golang\go\src> C:\Users\Costa\Documents\Google\golang\go\src> C:\Users\Costa\Documents\Google\golang\go\src>go tool dist test go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Users\\Costa\\Documents\\Google\\golang\\go\\go.exe": file does not exist C:\Users\Costa\Documents\Google\golang\go\src>set GO Environment variable GO not defined The last message confirms that there are no environment variables set which have a name starting with GO and I did the same for CGO. I add go env to confirm that values after failure. Thanks for keeping the issue open. C:\Users\Costa\Documents\Google\golang\go\src>go env set GOARCH=amd64 set GOBIN= set GOCACHE=C:\Users\Costa\AppData\Local\go-build set GOEXE=.exe set GOFLAGS= set GOHOSTARCH=amd64 set GOHOSTOS=windows set GOOS=windows set GOPATH=C:\Users\Costa\go set GOPROXY= set GORACE= set GOROOT=C:\Users\Costa\Documents\Google\golang\go set GOTMPDIR= set GOTOOLDIR=C:\Users\Costa\Documents\Google\golang\go\pkg\tool\windows_amd64 set GCCGO=gccgo set CC=gcc set CXX=g++ set CGO_ENABLED=1 set GOMOD= set CGO_CFLAGS=-g -O2 set CGO_CPPFLAGS= set CGO_CXXFLAGS=-g -O2 set CGO_FFLAGS=-g -O2 set CGO_LDFLAGS=-g -O2 set PKG_CONFIG=pkg-config set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\Costa\AppData\Local\Temp\go-build408101810=/tmp/go-build -gno-record-gcc-switches — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

iwdgo · 2018-08-14T11:06:09Z

I'll keep searching anyway. Thanks for your replies. If anybody can point me to the relevant part of the code, it would be helpful as I already searched cmd/go in and out.

iwdgo · 2018-08-21T13:02:36Z

Here is a working solution. I post a complete explanation to detail the options. A key contradiction is that all.ba* (bash/bat) checks for the presence of make.bat. The understanding is that it must run from inside the repo used for build.

After build using make.bat , the build is usable from outside the repo which sits under the original GOPATH as documentation requires.

Create your workspace directory, %USERPROFILE%\go. (If you'd like to use a different directory, you will need to set the GOPATH environment variable.)

To illustrate the issue, here is what happen in various cases including what works. %USERPROFILE% is used everywhere and has been replaced manually:

C:\Users\Costa\Documents\Google\golang\go\src>set GO
GOPATH=%USERPROFILE%\Documents\Google\
GOROOT=C:\Go\
GOROOT_BOOTSTRAP=C:\Go\

GOPATH\golang\go\bin>go tool dist version
go1.11rc1

The version displayed in the GOROOT version.

Result here is as expected:

GOPATH\golang\go\bin>go version
go version devel +30d3ebe367 Tue Aug 21 09:05:51 2018 +0000 windows/amd64

GOPATH\golang\go>set GOROOT=%CD%

GOPATH\golang\go\src>set GO
GOPATH=%USERPROFILE%\Documents\Google\
GOROOT=%USERPROFILE%\Documents\Google\golang\go
GOROOT_BOOTSTRAP=C:\Go\

GOPATH\golang\go>cd src

GOPATH\golang\go\src>go tool dist version
devel +30d3ebe367 Tue Aug 21 09:05:51 2018 +0000

run.bat still fails

GOPATH\golang\go\src>run.bat --no-rebuild

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "%USERPROFILE%\\Documents\\Google\\golang\\go\\go.exe": file does not exist

set GOPATH=%USERPROFILE%\Download

C:\Users\Costa\Documents\Google\golang\go\src>set GO
GOPATH=%USERPROFILE%\Downloads
GOROOT=%USERPROFILE%\Documents\Google\golang\go
GOROOT_BOOTSTRAP=C:\Go\

Here, you can run go build:

%USERPROFILE%\Documents\Google\golang\go\src>cd %GOPATH%

%USERPROFILE%\Downloads>go tool dist version
devel +30d3ebe367 Tue Aug 21 09:05:51 2018 +0000

%USERPROFILE%\Downloads>go tool dist test -no-rebuild

##### Testing packages.
ok      archive/tar     2.088s
ok      archive/zip     4.410s

%USERPROFILE%\Downloads>go tool dist test -no-rebuild -run=^go_test:cmd/go$

##### Testing packages.
ok      cmd/go  380.826s

ALL TESTS PASSED (some were excluded)

It is feasible to fix the build, all.bat, run.bat or the documentation but I'm not certain on how it is supposed to work now or in the near future.

iwdgo · 2018-10-21T14:57:25Z

Working on tip, issue is solved. Verbose flag must still be passed through GOFLAGS. Issue can be closed.

golang\go\src>go tool dist version
devel +035f9e8102 Sat Oct 20 14:19:29 2018 +0000
golang\go\src>set GOFLAGS=-v

golang\go\src>go tool dist test -no-rebuild -run=^go_test:encoding/xml$

##### Testing packages.
=== RUN   TestMarshal
=== RUN   TestMarshal/0
...

iwdgo changed the title ~~cmd/go: GOTOOLDIR cannot be updated~~ cmd/go: go tool dist test fail inside repo Aug 14, 2018

davecheney closed this as completed Aug 14, 2018

davecheney added ExpertNeeded OS-Windows labels Aug 14, 2018

davecheney reopened this Aug 14, 2018

agnivade closed this as completed Oct 21, 2018

golang locked and limited conversation to collaborators Oct 21, 2019

gopherbot added the FrozenDueToAge label Oct 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: go tool dist test fail inside repo #26980

cmd/go: go tool dist test fail inside repo #26980

iwdgo commented Aug 14, 2018

agnivade commented Aug 14, 2018

davecheney commented Aug 14, 2018

iwdgo commented Aug 14, 2018

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018 via email

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018 via email

iwdgo commented Aug 14, 2018

iwdgo commented Aug 21, 2018

iwdgo commented Oct 21, 2018

cmd/go: go tool dist test fail inside repo #26980

cmd/go: go tool dist test fail inside repo #26980

Comments

iwdgo commented Aug 14, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

agnivade commented Aug 14, 2018

davecheney commented Aug 14, 2018

iwdgo commented Aug 14, 2018

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018 via email

iwdgo commented Aug 14, 2018

davecheney commented Aug 14, 2018 via email

iwdgo commented Aug 14, 2018

iwdgo commented Aug 21, 2018

iwdgo commented Oct 21, 2018

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?