Skip to content

x/build/cmd/coordinator: some reverse buildlets getting 401 Unauthorized from Athens #30749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bradfitz opened this issue Mar 11, 2019 · 8 comments
Closed

x/build/cmd/coordinator: some reverse buildlets getting 401 Unauthorized from Athens #30749

bradfitz opened this issue Mar 11, 2019 · 8 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

In this failure, the Solaris SmartOS reverse builder on Joyent gets an HTTP 401 error from Athens:

https://build.golang.org/log/46f3157e9df8b8df1165d18ce3924a9a571aa637

solaris-amd64-smartosbuildlet at 486ca37b14f56b2e125fc3afbd900a5369532043 building oauth2 at e64efc72b421e893cbf63f17ba2221e7d6d0b0f3

:: Running /tmp/workdir/go/bin/go with args ["/tmp/workdir/go/bin/go" "test" "-short" "golang.org/x/oauth2/..."] and env ["_=*579310*/opt/buildlet-stage0" "LANG=C" "PATH=/tmp/workdir/go/bin:/usr/sbin:/usr/bin:/opt/local/bin" "PWD=/" "SHLVL=1" "SMF_FMRI=svc:/network/buildlet:default" "SMF_METHOD=start" "SMF_RESTARTER=svc:/system/svc/restarter:default" "SMF_ZONENAME=2d3e7fba-7436-4d0b-e3f6-85ba4a54741d" "TZ=UTC" "A__z=\"*SHLVL" "WORKDIR=/tmp/workdir" "GOROOT_BOOTSTRAP=/tmp/workdir/go1.4" "GO_BUILDER_NAME=solaris-amd64-smartosbuildlet" "GOROOT_BOOTSTRAP=/root/go-solaris-amd64-bootstrap" "HOME=/root" "GOROOT=/tmp/workdir/go" "GOPATH=/tmp/workdir/gopath" "GOPROXY=http://10.240.0.50:30156" "GO_BUILDER_SET_GOPROXY=coordinator" "GO111MODULE=on" "GOPROXY=http://localhost:64065" "TMPDIR=/tmp/workdir/tmp" "GOCACHE=/tmp/workdir/gocache"] in dir /tmp/workdir/gopath/src/golang.org/x/oauth2

go: finding golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e
go: finding cloud.google.com/go v0.34.0
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding google.golang.org/appengine v1.4.0
go: golang.org/x/net@v0.0.0-20190108225652-1e06a53dbb7e: unexpected status (http://localhost:64065/golang.org/x/net/@v/v0.0.0-20190108225652-1e06a53dbb7e.info): 401 Unauthorized
go: google.golang.org/appengine@v1.4.0: unexpected status (http://localhost:64065/google.golang.org/appengine/@v/v1.4.0.info): 401 Unauthorized
go: golang.org/x/sync@v0.0.0-20181221193216-37e7f081c4d4: unexpected status (http://localhost:64065/golang.org/x/sync/@v/v0.0.0-20181221193216-37e7f081c4d4.info): 401 Unauthorized
go: cloud.google.com/go@v0.34.0: unexpected status (http://localhost:64065/cloud.google.com/go/@v/v0.34.0.info): 401 Unauthorized
go: error loading module requirements

Why?

It's obviously talking to Athens fine, but why would Athens return 401? From upstream? Why would upstream return 401? I guess IP rate limiting from Gerrit?

Maybe our Athens IP(s) should be whitelisted. Or we should make their git requests authenticated?

/cc @dmitshur @bcmills
@gopherbot gopherbot added this to the Unreleased milestone Mar 11, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Mar 11, 2019
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 11, 2019
@bcmills
Copy link
Contributor

bcmills commented Mar 11, 2019

Why would upstream return 401?

#30690. 😞

@bcmills
Copy link
Contributor

bcmills commented Mar 11, 2019

Hmm, I guess that results in 500's, though.

@bradfitz
Copy link
Contributor Author

Unless Athens maps 500 to 401 for some reason.

But I suspect we're just being rate limited. It wouldn't be the first time.

@bsiegert
Copy link
Contributor

Happening now on the netbsd-arm-bsiegert builder.

@bradfitz
Copy link
Contributor Author

The big question is why Athens is even getting a 401 itself from Gerrit. These should be cache hits, so Athens shouldn't be even hitting upstream at all.

Either our Athens is in a bad state somehow, or there's a bug. Or maybe it's fixed in upstream Athens and we're behind.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/170620 mentions this issue: cmd/coordinator: add logging around module proxy errors

gopherbot pushed a commit to golang/build that referenced this issue Apr 3, 2019
@bradfitz
cmd/coordinator: add logging around module proxy errors
c12ed7d 
Trying to track down where the 401s like these are coming from:

https://build.golang.org/log/4f9d520650841688b416ade5c205cd10b487f319

Updates golang/go#30749

Change-Id: I2f3e73882619b90a5238040f4d1134d678a058bd
Reviewed-on: https://go-review.googlesource.com/c/build/+/170620
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 3, 2019

Yay new logging. This seems to be happening before we even get to Athens:

2019/04/03 18:37:20 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:37:21 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:37:21 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:37:21 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:38:45 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:38:45 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:38:45 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:38:45 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:38:46 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:38:46 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:38:46 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:38:46 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:38:47 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:38:47 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:38:47 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:38:47 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:38:47 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:38:47 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:38:47 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:38:47 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-s390x-ibm"
2019/04/03 18:40:09 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:40:09 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:40:09 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:40:09 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:40:09 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:40:09 modproxy: sending 401 Unauthorized due to invalid key for builder "linux-ppc64-buildlet"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"
2019/04/03 18:40:10 modproxy: sending 401 Unauthorized due to invalid key for builder "host-linux-ppc64le-osu"

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/170621 mentions this issue: cmd/buildlet: trim space from reverse buildlets' gobuildkey file contents

@golang golang locked and limited conversation to collaborators Apr 2, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@bradfitz @bsiegert @dmitshur @bcmills @gopherbot