Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/gomote: unexpected /connect-ssh response: 502 Bad Gateway when connecting multiple SSH sessions #53394

Closed
bcmills opened this issue Jun 15, 2022 · 3 comments
Closed

x/build/cmd/gomote: unexpected /connect-ssh response: 502 Bad Gateway when connecting multiple SSH sessions #53394

bcmills opened this issue Jun 15, 2022 · 3 comments
Assignees
@cagedmantis
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release Team
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented Jun 15, 2022

go1.19beta1 version -m $(which gomote) 
/usr/local/google/home/bcmills/bin/gomote: go1.19beta1
        path    golang.org/x/build/cmd/gomote
        mod     golang.org/x/build      v0.0.0-20220614223518-59e7a6bb02db      h1:m2l/jsehSm9knEbpJWq+y8CTOTda9SEZoV3MtnaqMv8=
        dep     cloud.google.com/go/compute     v1.3.0  h1:mPL/MzDDYHsh5tHRS9mhmhWlcgClCrCa6ApQCU6wnHI=
        dep     github.com/aws/aws-sdk-go       v1.30.15        h1:Sd8QDVzzE8Sl+xNccmdj0HwMrFowv6uVUx9tGsCE1ZE=
        dep     github.com/golang/groupcache    v0.0.0-20200121045136-8c9f03a8e57e      h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
        dep     github.com/golang/protobuf      v1.5.2  h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
        dep     github.com/googleapis/gax-go/v2 v2.1.1  h1:dp3bWCh+PPO1zjRRiCSczJav13sBvG4UhNyVTa1KqdU=
        dep     github.com/jmespath/go-jmespath v0.4.0  h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
        dep     go.opencensus.io        v0.23.0 h1:gqCw0LfLxScz8irSi8exQc7fyQ0fKQU/qnC/X8+V/1M=
        dep     golang.org/x/net        v0.0.0-20220127200216-cd36cc0744dd      h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
        dep     golang.org/x/oauth2     v0.0.0-20211104180415-d3ed0bb246c8      h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg=
        dep     golang.org/x/sync       v0.0.0-20210220032951-036812b2e83c      h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
        dep     golang.org/x/sys        v0.0.0-20220209214540-3681064d5158      h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
        dep     golang.org/x/text       v0.3.7  h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
        dep     golang.org/x/time       v0.0.0-20210723032227-1f47c861a9ac      h1:7zkz7BUtwNFFqcowJ+RIgu2MaV/MapERkDIy+mwPyjs=
        dep     google.golang.org/api   v0.70.0 h1:67zQnAE0T2rB0A3CwLSas0K+SbVzSxP+zTLkQLexeiw=
        dep     google.golang.org/genproto      v0.0.0-20220222213610-43724f9ea8cf      h1:SVYXkUz2yZS9FWb2Gm8ivSlbNQzL2Z/NpPKE3RG2jWk=
        dep     google.golang.org/grpc  v1.44.0 h1:weqSxi/TMs1SqFRMHCtBgXRs8k3X39QIDEZ0pRcttUg=
        dep     google.golang.org/protobuf      v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
        dep     gopkg.in/inf.v0 v0.9.1  h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
        build   -compiler=gc
        build   CGO_ENABLED=1
        build   CGO_CFLAGS=
        build   CGO_CPPFLAGS=
        build   CGO_CXXFLAGS=
        build   CGO_LDFLAGS=
        build   GOARCH=amd64
        build   GOOS=linux
        build   GOAMD64=v1

What did you do?

In one terminal (in a tmux session):

~$ gomote create linux-amd64
# still creating linux-amd64 after 5s; 0 requests ahead of you
# still creating linux-amd64 after 10s; 0 requests ahead of you
# still creating linux-amd64 after 15s; 0 requests ahead of you
# still creating linux-amd64 after 20s; 0 requests ahead of you
# still creating linux-amd64 after 25s; 0 requests ahead of you
# still creating linux-amd64 after 30s; 0 requests ahead of you
# still creating linux-amd64 after 35s; 0 requests ahead of you
# still creating linux-amd64 after 40s; 0 requests ahead of you
user-bcmills-linux-amd64-0

~$ gomote ssh user-bcmills-linux-amd64-0
$ ssh -p 2222 user-bcmills-linux-amd64-0@farmer.golang.org # auth using https://github.com/bcmills.keys
Enter passphrase for key '/usr/local/google/home/bcmills/.ssh/id_rsa':
# Welcome to the gomote ssh proxy, bcmills.
# Connecting to/starting remote ssh...
#
# `gomote push` and the builders use:
# - workdir: /workdir
# - GOROOT: /workdir/go
# - GOPATH: /workdir/gopath
# - env: GO_BUILDER_NAME=linux-amd64 GOROOT_BOOTSTRAP=/go1.4 GO_DISABLE_OUTBOUND_NETWORK=1
# Happy debugging.
…

Once the first SSH connection is established, attempt another gomote ssh command in a separate terminal:

~$ gomote ssh user-bcmills-linux-amd64-0

What did you expect to see?

Two successful gomote ssh sessions connected to the same VM.

What did you see instead?

$ ssh -p 2222 user-bcmills-linux-amd64-0@farmer.golang.org # auth using https://github.com/bcmills.keys
Enter passphrase for key '/usr/local/google/home/bcmills/.ssh/id_rsa':
# Welcome to the gomote ssh proxy, bcmills.
# Connecting to/starting remote ssh...
#
failed to connect to ssh on user-bcmills-linux-amd64-0: unexpected /connect-ssh response: 502 Bad Gateway, dial tcp [::1]:2200: connect: connection refused

―

To my recollection this used to work — I'm not sure exactly when it regressed.

(CC @golang/release)
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jun 15, 2022
@gopherbot gopherbot added this to the Unreleased milestone Jun 15, 2022
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 15, 2022
@cagedmantis cagedmantis added this to In Progress in Go Release Team Jun 21, 2022
@cagedmantis cagedmantis self-assigned this Jun 21, 2022
@cagedmantis
Copy link
Contributor

I'm investigating if this works via the new gomote path. Once we have that information then we will re-triage the issue.

@cagedmantis
Copy link
Contributor

I've confirmed that this also does not work with the new gomote implementation.

@cagedmantis cagedmantis moved this from In Progress to Planned in Go Release Team Jun 27, 2022
@gopherbot
Copy link

Change https://go.dev/cl/541381 mentions this issue: cmd/buildlet: add an SSH server to the buildlet

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cagedmantis cagedmantis

Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Go Release Team
Planned
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@cagedmantis @dmitshur @bcmills @gopherbot