There are elements of Windows-specific stuff to this issue, but the more general issue (if I risk overloading this one) is that each gomote environment should be audited to be usable. Perhaps more code should be shared so that the gomote environments can't rot, or as easily.

Also, IIRC, gomote to BSDs ssh in as "gopher" or some non-root user with pwd in /home/gopher but the buildlets run as root, in /work ($WORKDIR). That should also be made consistent, one way or another.

Tests would be nice too, even if they were manual integration tests we run occasionally by hand.

Change https://golang.org/cl/181542 mentions this issue: cmd/go: in TestScript, set $GOEXE instead of $exe

Change https://golang.org/cl/207459 mentions this issue: dashboard, env, cmd/buildlet/testssh: fix gomote ssh for a number of buidlers

Related: I filed #35629 to make the Windows environments have their GUIs enabled, so RDP into them is more useful than a black cmd.exe on a black background.

On a freebsd-amd64-12_2 gomote, the workdir (/tmp/workdir) is owned by and only writable by root, but gomote ssh logs in as gopher. I have to switch to root just to write files in the workdir.

An experience report from debugging #50892 via gomote.

1. When I gomote ssh, I get a greeting message listing an environment, but that environment isn't actually set for me:

# `gomote push` and the builders use:
# - workdir: /Users/gopher/workdir
# - GOROOT: /Users/gopher/workdir/go
# - GOPATH: /Users/gopher/workdir/gopath
# - env: GO_BUILDER_NAME=darwin-amd64-12_0 GOROOT_BOOTSTRAP=/Users/gopher/goboot
…
gopher@gophers-Mac src % ./make.bash
ERROR: Cannot find /Users/gopher/go1.4/bin/go.

2. The terminal mapping in gomote ssh from Linux is completely broken. When I press the ⌫ key on my keyboard, the cursor advances ahead a space, but appears to maybe also delete a character from its internal buffer? When I press the left and right arrow keys the cursor doesn't move. When I press the ⌦ key on my keyboard, the console prints ~ instead of deleting a character. Maybe something in the $TERM settings is off?

3. When I try to work around (2) using gomote run, it apparently has an empty $PATH, even though gomote run (without other arguments) claims that for the -path flag “The empty string (default) does not modify the $PATH”.

4. gomote run gives an inscrutable error when I try to give it a (possibly-not-found?) relative directory:

~/go/src$ gomote run $(<~/.gomote) /bin/pwd
/Users/gopher/workdir

~/go/src$ gomote run $(<~/.gomote) ../bin/go
Error running run: Error trying to execute ../bin/go: buildlet: HTTP status 400 Bad Request: requires 'cmd' parameter

@bcmills , for (2), what's your local TERM and what TERM do you get once sshed? The output of stty -a on both sides may also be relevant (not that I understand most of that output).

for (2), what's your local TERM and what TERM do you get once sshed?

Local $TERM is tmux-256color. Remote $TERM is also tmux-256color, which is likely related to the problem.

~$ stty -a
speed 38400 baud; rows 105; columns 100; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>;
swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O;
min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel
iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho
-extproc

gopher@gophers-Mac-74 ~ % stty -a
speed 38400 baud; 105 rows; 100 columns;
lflags: icanon isig iexten echo echoe echok echoke -echonl echoctl
        -echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo
        -extproc
iflags: -istrip icrnl -inlcr -igncr ixon -ixoff -ixany -imaxbel -iutf8
        -ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
        -dtrflow -mdmbuf
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = ^@; eol2 = ^@;
        erase = ^?; intr = ^C; kill = ^U; lnext = ^V; min = 1; quit = ^\;
        reprint = ^R; start = ^Q; status = ^T; stop = ^S; susp = ^Z;
        time = 0; werase = ^W;

~/go-review$ gomote run $(<~/.gomote) /tmp/buildlet/go/src/make.bash
make.bash must be run from $GOROOT/src
Error running run: exit status 1

~/go-review$ gomote run -dir=/tmp/buildlet/go/src $(<~/.gomote) make.bash
Error running run: Error trying to execute make.bash: buildlet: HTTP status 400 Bad Request: bogus 'dir' parameter

Apparently I need to repeat the full path to the command twice (once for the command and once for the -dir argument)‽

I wrote https://go.dev/cl/418104 as an attempt at fixing this problem. However, I have no idea how to test it.

Change https://go.dev/cl/418104 mentions this issue: internal/coordinator/remote: set up enviroment for SSH clients

Committed and tested.

Change https://go.dev/cl/418654 mentions this issue: internal/coordinator/remote: export base environment for ssh