Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/gomote: default SSH environment is still not usable on Windows #56015

Open
bcmills opened this issue Oct 3, 2022 · 5 comments
Open
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Oct 3, 2022

What version of Go are you using (go version)?

~/go/src$ go version
go version devel go1.20-b150c629ab Mon Oct 3 15:00:34 2022 -0400 linux/amd64
~/go/src$ go version -m $(which gomote)
/usr/local/google/home/bcmills/bin/gomote: devel go1.20-b150c629ab Mon Oct 3 15:00:34 2022 -0400
        path    golang.org/x/build/cmd/gomote
        mod     golang.org/x/build      v0.0.0-20220930212151-6d81d672f5d8      h1:n9vXkRm+rbxzB4YPd3iUolVmrIu3MN3ZjkMM9siIrIw=
        dep     cloud.google.com/go/compute     v1.3.0  h1:mPL/MzDDYHsh5tHRS9mhmhWlcgClCrCa6ApQCU6wnHI=
        dep     github.com/aws/aws-sdk-go       v1.30.15        h1:Sd8QDVzzE8Sl+xNccmdj0HwMrFowv6uVUx9tGsCE1ZE=
        dep     github.com/golang/groupcache    v0.0.0-20200121045136-8c9f03a8e57e      h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
        dep     github.com/golang/protobuf      v1.5.2  h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
        dep     github.com/googleapis/gax-go/v2 v2.1.1  h1:dp3bWCh+PPO1zjRRiCSczJav13sBvG4UhNyVTa1KqdU=
        dep     github.com/jmespath/go-jmespath v0.4.0  h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
        dep     go.opencensus.io        v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M=
        dep     golang.org/x/net        v0.0.0-20220127200216-cd36cc0744dd      h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
        dep     golang.org/x/oauth2     v0.0.0-20211104180415-d3ed0bb246c8      h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg=
        dep     golang.org/x/sync       v0.0.0-20210220032951-036812b2e83c      h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
        dep     golang.org/x/sys        v0.0.0-20220722155257-8c9f86f7a55f      h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
        dep     golang.org/x/text       v0.3.7  h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
        dep     golang.org/x/time       v0.0.0-20210723032227-1f47c861a9ac      h1:7zkz7BUtwNFFqcowJ+RIgu2MaV/MapERkDIy+mwPyjs=
        dep     google.golang.org/api   v0.70.0 h1:67zQnAE0T2rB0A3CwLSas0K+SbVzSxP+zTLkQLexeiw=
        dep     google.golang.org/genproto      v0.0.0-20220222213610-43724f9ea8cf      h1:SVYXkUz2yZS9FWb2Gm8ivSlbNQzL2Z/NpPKE3RG2jWk=
        dep     google.golang.org/grpc  v1.44.0 h1:weqSxi/TMs1SqFRMHCtBgXRs8k3X39QIDEZ0pRcttUg=
        dep     google.golang.org/protobuf      v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
        build   -buildmode=exe
        build   -compiler=gc
        build   CGO_ENABLED=1
        build   CGO_CFLAGS=
        build   CGO_CPPFLAGS=
        build   CGO_CXXFLAGS=
        build   CGO_LDFLAGS=
        build   GOARCH=amd64
        build   GOOS=linux
        build   GOAMD64=v1

What did you do?

Use gomote create to create a Windows instance, then gomote putbootstrap and gomote push to populate it.

Then, use gomote ssh to connect to it.

What did you expect to see?

Since #32430 was closed, I expected to see an environment in which I could run make.bat without additional configuration.

What did you see instead?

I still had to discover and set the correct values for GOROOT_BOOTSTRAP and PATH manually, and even then I suspect that my PATH may be missing some things.

~/go/src$ mote ssh
$ /usr/bin/ssh -o CertificateFile=/tmp/.gomote/id_ed25519-162749511-cert.pub -i /usr/local/google/home/bcmills/.config/gomote/.ssh/id_ed25519 -p 2222 bcmills-windows-amd64-2016-0@farmer.golang.org
Enter passphrase for key '/usr/local/google/home/bcmills/.config/gomote/.ssh/id_ed25519':
# Welcome to the gomote ssh proxy.
# Connecting to/starting remote ssh...
#
# `gomote push` and the builders use:
# - workdir: C:\workdir
# - GOROOT: C:\workdir/go
# - GOPATH: C:\workdir/gopath
# - env: GO_BUILDER_NAME=windows-amd64-2016 GOARCH=amd64 GOHOSTARCH=amd64 GO_TEST_TIMEOUT_SCALE=2
# Happy debugging.
set GO_BUILDER_NAME=windows-amd64-2016
set GOARCH=amd64
set GOHOSTARCH=amd64
set GO_TEST_TIMEOUT_SCALE=2
set GOPATH=C:\workdir\gopath
set PATH=$PATH;C:\workdir\go\bin
cd C:\workdir\go\src
Warning: Permanently added '[localhost]:44503' (ECDSA) to the list of known hosts.
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

gopher@SERVER-2016-V7- C:\Users\gopher>set GO_BUILDER_NAME=windows-amd64-2016

gopher@SERVER-2016-V7- C:\Users\gopher>set GOARCH=amd64

gopher@SERVER-2016-V7- C:\Users\gopher>set GOHOSTARCH=amd64

gopher@SERVER-2016-V7- C:\Users\gopher>set GO_TEST_TIMEOUT_SCALE=2

gopher@SERVER-2016-V7- C:\Users\gopher>set GOPATH=C:\workdir\gopath

gopher@SERVER-2016-V7- C:\Users\gopher>set PATH=$PATH;C:\workdir\go\bin

gopher@SERVER-2016-V7- C:\Users\gopher>cd C:\workdir\go\src

gopher@SERVER-2016-V7- C:\workdir\go\src>.\make.bat
ERROR: Cannot find C:\Users\gopher\Go1.4\bin\go.exe
Set GOROOT_BOOTSTRAP to a working Go tree >= Go 1.17.

gopher@SERVER-2016-V7- C:\workdir\go\src>set GOROOT_BOOTSTRAP=C:\workdir\go1.4

gopher@SERVER-2016-V7- C:\workdir\go\src>.\make.bat
Building Go cmd/dist using C:\workdir\go1.4. (go1.17.13 windows/amd64)
go tool dist: cannot invoke C compiler ["gcc"]: exec: "gcc": executable file not found in %PATH%

Go needs a system C compiler for use with cgo.
To set a C compiler, set CC=the-compiler.
To disable cgo, set CGO_ENABLED=0.

gopher@SERVER-2016-V7- C:\workdir\go\src>set PATH=%PATH%;C:\godep\gcc64\bin

gopher@SERVER-2016-V7- C:\workdir\go\src>.\make.bat
Building Go cmd/dist using C:\workdir\go1.4. (go1.17.13 windows/amd64)
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for windows/amd64.
---
Installed Go for windows/amd64 in C:\workdir\go
Installed commands in C:\workdir\go\bin
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Oct 3, 2022
@gopherbot gopherbot added this to the Unreleased milestone Oct 3, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Oct 3, 2022

(CC @ianlancetaylor @mknyszek from #32430)

@bcmills
Copy link
Contributor Author

bcmills commented Oct 3, 2022

The PATH issue seems to be invalid escaping somewhere:

gopher@SERVER-2016-V7- C:\workdir\go\src>echo %PATH%
$PATH;C:\workdir\go\bin

Note that %PATH% contains the literal string $PATH, as if it were trying to expand a variable on Unix instead of Windows.

That comes from here:
https://cs.opensource.google/go/x/build/+/master:internal/coordinator/remote/ssh.go;l=323;drc=bdfd194e7635d156821351a49bba2bc00a3211ac

@gopherbot
Copy link

Change https://go.dev/cl/437817 mentions this issue: internal/coordinator/remote: use correct syntax to expand PATH on Windows

@bcmills bcmills changed the title x/build/cmd/gomote: default SSH environment is still not usable on Windows x/build/internal/coordinator/remote: default SSH environment is still not usable on Windows Oct 3, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Oct 4, 2022

https://go.dev/cl/437817 is necessary and perhaps sufficient to fix this. (I'll double-check once a coordinator is deployed with that fix.)

@bcmills bcmills self-assigned this Oct 4, 2022
@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsFix The path to resolution is known, but the work has not been done. labels Oct 4, 2022
gopherbot pushed a commit to golang/build that referenced this issue Oct 4, 2022
…dows

For golang/go#56015.

Change-Id: Ic16f9b8e995424f50fda6921228a75683f91f8ff
Reviewed-on: https://go-review.googlesource.com/c/build/+/437817
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@dmitshur dmitshur removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 11, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Oct 13, 2022

(I'll double-check once a coordinator is deployed with that fix.)

With the most recent coordinator push, it appears that:

  • on many (all?) platforms we're missing either a GOROOT_BOOTSTRAP environment variable or a PATH entry that includes the default bootstrap directory
  • on Windows, we're also missing a PATH entry for the C compiler.

From what I can tell, the problem is that these are set in cmd/buildlet.baseEnv instead of provided by the coordinator.

I can think of a few possible solutions:

  1. Move the handling for those environment variables from the buildlet to the coordinator.

    • However, the buildlet logic for GOROOT_BOOTSTRAP is a bit subtle, so it's not obvious to me whether this is feasible.
  2. Have the coordinator execute a script on the buildlet to identify the actual environment and then use that.

    • This seems like the simplest approach to me, but perhaps not the cleanest?
  3. Somehow change the way the ssh daemon is started so that it inherits buildlet environment variables.

    • This seems like the most hermetic approach, but I don't know how to implement it.

I don't plan to follow up on any of these myself at the moment, although approach (2) seems straightforward enough to be shovel-ready when someone gets the time.

@bcmills bcmills removed their assignment Oct 13, 2022
@dmitshur dmitshur changed the title x/build/internal/coordinator/remote: default SSH environment is still not usable on Windows x/build/cmd/gomote: default SSH environment is still not usable on Windows Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Status: Planned
Development

No branches or pull requests

4 participants
@dmitshur @bcmills @gopherbot and others