x/build: enable builder-specific GOPROXY settings on reverse builders #42971

bcmills · 2020-12-03T15:42:52Z

Looking at the builder log in https://build.golang.org/log/26ea2fd1426a51a224b5d321915717e906ded84a, I see the following line. Note that it lists GOPROXY three different times and GOROOT_BOOTSTRAP twice:

:: Running /tmp/farm/go/bin/go with args ["/tmp/farm/go/bin/go" "test" "-short" "golang.org/x/crypto/..."] and env ["LANG=zh_CN.UTF-8" "PATH=/tmp/farm/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOME=/home/gopher" "LOGNAME=gopher" "USER=gopher" "SHELL=/bin/bash" "INVOCATION_ID=fe969678216d4cceaebbdc436df708c2" "JOURNAL_STREAM=8:4142335" "GO_BUILDER_ENV=host-linux-amd64-wsl" "WORKDIR=/tmp/farm" "HTTPS_PROXY=http://proxy:8123" "HTTP_PROXY=http://proxy:8123" "GOPROXY=https://goproxy.io,direct" "WSL_DISTRO_NAME=Ubuntu" "WSL_INTEROP=/run/WSL/20_interop" "WSLENV=BASH_ENV/u" "GO_STAGE0_NET_DELAY=700ms" "GO_STAGE0_DL_DELAY=600ms" "GOROOT_BOOTSTRAP=/tmp/farm/go1.4" "GO_BUILDER_NAME=linux-amd64-wsl" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/lib/go" "GOROOT=/tmp/farm/go" "GOPATH=/tmp/farm/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=https://proxy.golang.org" "TMPDIR=/tmp/farm/tmp" "GOCACHE=/tmp/farm/gocache"] in dir /tmp/farm/gopath/src/golang.org/x/crypto

The second GOROOT_BOOTSTRAP seems to come from here, and might be working as intended: https://github.com/golang/build/blob/a9f75ee9992b28221ceec0c56af51265ea857ce2/cmd/buildlet/buildlet.go#L263

The second GOPROXY setting comes from here: https://github.com/golang/build/blob/5f1e306b98a112c236997d393adc5c11405377ca/cmd/coordinator/coordinator.go#L2702

The third GOPROXY setting comes from here: https://github.com/golang/build/blob/b6220de1adc000cad31eadf59558e1d3a8e32205/dashboard/builders.go#L921-L925

Given that one of the tests in that log failed with an EOF error communicating with proxy.golang.org, I suspect that it really should be using the local GOPROXY default (https://goproxy.io,direct) instead of the cmd/go default. I'm not sure what needs to happen to fix that configuration.

CC @mengzhuo @cagedmantis @golang/release

The text was updated successfully, but these errors were encountered:

mengzhuo · 2020-12-04T02:27:50Z

Please assign this issue to me. @bcmills
p.s. Why can't I assign this issue to myself?

mengzhuo · 2020-12-04T02:39:01Z

To other builders maintainer, linux-amd64-wsl using systemd to set the enviroment.

[Unit]
Description=Go Builder For WSL
After=network-online.target

[Service]
ExecStartPre=/bin/bash -c 'chmod -R u+w /tmp/farm; rm -rf /tmp/farm; /bin/mkdir -p /tmp/farm; true'
ExecStart=/home/gopher/go/bin/stage0
Environment=GO_BUILDER_ENV=host-linux-amd64-wsl
Environment=WORKDIR=/tmp/farm
Environment=HTTPS_PROXY=http://proxy:8123
Environment=HTTP_PROXY=http://proxy:8123
Environment=GOPROXY=https://goproxy.io,direct

User=gopher
WorkingDirectory=/home/gopher

Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

dmitshur · 2020-12-04T02:39:12Z

p.s. Why can't I assign this issue to myself?

@mengzhuo Take a look at https://golang.org/wiki/GitHubAccess#assignees.

gopherbot · 2020-12-04T03:54:54Z

Change https://golang.org/cl/275332 mentions this issue: buildlet: check $workdir/go1.4 existance before append env

gopherbot · 2020-12-04T07:20:47Z

Change https://golang.org/cl/275412 mentions this issue: cmd/coordinator,dashboard: add privateGoProxy setting

oiooj · 2021-01-22T04:00:09Z

If anything else need to be change, I'd like to push it.

bcmills · 2022-02-04T20:12:54Z

greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2021-01-01

bcmills · 2022-02-04T20:15:00Z

@golang/release: this is a frequent failure mode on multiple reverse builders, and there has been a fix CL (CL 275412) out for review since December 2020. Can we prioritize a review for that change at some point in the near future?

bcmills · 2022-04-18T15:38:14Z

@golang/release, any progress to share on this issue? As far as I can tell the CL from 2020 still hasn't been picked up, and I'm still seeing a lot of errors on the builders that look like they could be resolved by this:

greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2022-02-04

Builders behind firewall may not access to proxy.golang.org but has its own private Go module proxy linux-mips64le-mengzhuo linux-amd64-wsl linux-riscv64-unmatched NeedsGoProxy is for long test used which we should not change. Update golang/go#42971 Change-Id: If300b751487ae08e05f5200581eb4c2719706a56 Reviewed-on: https://go-review.googlesource.com/c/build/+/275412 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Lee Baokun <bk@golangcn.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>

dmitshur · 2022-04-30T03:34:56Z

CL 275412 has now landed. Comparing a recent before/after log shows it's working as intended: there's no more unwanted "GOPROXY=https://proxy.golang.org" entry.

We can check again in a few weeks to make sure the same problem doesn't happen again for another reason, but at least so far there aren't any new hits (other than one yesterday, before the fix):

greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2022-04-28
2022-04-28T16:00:39-dcaea06-17371ee/linux-amd64-wsl

bcmills · 2022-05-02T19:51:27Z

Unfortunately, goproxy.io doesn't seem to be entirely reliable either. 😞

https://build.golang.org/log/58e02729ea56983be383b850ca96f268ba258e4b:

:: Running /tmp/workdir-host-linux-amd64-wsl/go/bin/go with args ["/tmp/workdir-host-linux-amd64-wsl/go/bin/go" "test" "-short" "golang.org/x/website/..."] and env ["LANG=zh_CN.UTF-8" "HOME=/home/gopher" "LOGNAME=gopher" "USER=gopher" "SHELL=/bin/bash" "INVOCATION_ID=4b8fac3e13e44f2ca8c6759ce9cfce12" "JOURNAL_STREAM=8:47829246" "GO_BUILDER_ENV=host-linux-amd64-wsl" "HTTPS_PROXY=http://uplink.golangcn.org:8123" "HTTP_PROXY=http://uplink.golangcn.org:8123" "GOPROXY=https://goproxy.io,direct" "WSL_DISTRO_NAME=Ubuntu" "WSL_INTEROP=/run/WSL/20_interop" "WSLENV=BASH_ENV/u" "GO_STAGE0_NET_DELAY=400ms" "GO_STAGE0_DL_DELAY=1.3s" "WORKDIR=/tmp/workdir-host-linux-amd64-wsl" "GO_BUILDER_NAME=linux-amd64-wsl" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/lib/go" "GOROOT=/tmp/workdir-host-linux-amd64-wsl/go" "GOPATH=/tmp/workdir-host-linux-amd64-wsl/gopath" "GO111MODULE=on" "TMPDIR=/tmp/workdir-host-linux-amd64-wsl/tmp" "GOCACHE=/tmp/workdir-host-linux-amd64-wsl/gocache" "PATH=/tmp/workdir-host-linux-amd64-wsl/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "PWD=/tmp/workdir-host-linux-amd64-wsl/gopath/src/golang.org/x/website"] in dir /tmp/workdir-host-linux-amd64-wsl/gopath/src/golang.org/x/website

…
cmd/events/main.go:16:2: github.com/microcosm-cc/bluemonday@v1.0.2: Get "https://proxy.golang.com.cn/github.com/microcosm-cc/bluemonday/@v/v1.0.2.zip": Service Unavailable

mengzhuo · 2022-05-03T01:46:27Z

cc @oiooj as goproxy.io maintainer

oiooj · 2022-05-03T04:01:21Z

I tested this request and there is no problem so far.

curl -v https://proxy.golang.com.cn/github.com/microcosm-cc/bluemonday/@v/v1.0.2.zip

and goproxy.io use Tencent Cloud global CDN service. 😢

bcmills · 2022-05-03T18:32:17Z

greplogs -l -e 'https://(?:goproxy\.io|proxy.golang.com.cn).*(?:unexpected EOF|Service Unavailable|connection reset)' --since=2022-04-29
2022-05-03T12:34:17-fb9be63-a41e37f/linux-amd64-wsl
2022-05-02T19:53:56-fb9be63-3b01a80/linux-amd64-wsl
2022-04-29T18:37:11-fb9be63-8c5917c/linux-amd64-wsl

bcmills · 2022-05-09T14:40:40Z

At any rate: it looks like the builder-specific GOPROXY settings are working as intended, so I'm going to close this issue.

I'll file a separate issue for the connection errors on the builders using goproxy.io.

bcmills · 2022-05-09T16:12:39Z

(Filed the goproxy.io connectivity issue as #52793.)

bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 3, 2020

gopherbot added this to the Unreleased milestone Dec 3, 2020

bcmills changed the title ~~x/build: environment configuration issue for linux-amd64-wsl builder~~ x/build: GOPROXY environment configuration issue with linux-amd64-wsl builder Dec 3, 2020

dmitshur assigned mengzhuo Dec 4, 2020

oiooj mentioned this issue Jan 22, 2021

x/build/cmd/coordinator: Allow custom GOPROXY environment variable for buildlet #43839

Closed

bcmills changed the title ~~x/build: GOPROXY environment configuration issue with linux-amd64-wsl builder~~ x/build: enable builder-specific GOPROXY settings on reverse builders Feb 4, 2022

toothrot added this to Planned in Go Release Team Feb 8, 2022

bcmills mentioned this issue Feb 24, 2022

x/tools/gopls/internal/regtest/modfile: TestSumUpdateFixesDiagnostics failures with "Unmet: empty diagnostics" #51352

Closed

bcmills mentioned this issue Apr 18, 2022

x/vuln/cmd/govulncheck: TestCommand failures with mismatched exit codes and no other output #52410

Closed

dmitshur moved this from Planned to In Progress in Go Release Team Apr 29, 2022

dmitshur self-assigned this Apr 29, 2022

bcmills closed this as completed May 9, 2022

Go Release Team automation moved this from In Progress to Done May 9, 2022

rsc unassigned mengzhuo and dmitshur Jun 23, 2022

dmitshur mentioned this issue Aug 4, 2022

x/build: linux-amd64-wsl builder runs into "unexpected EOF" errors downloading dependencies #54275

Open

golang locked and limited conversation to collaborators Jun 23, 2023

gopherbot added the FrozenDueToAge label Jun 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: enable builder-specific GOPROXY settings on reverse builders #42971

x/build: enable builder-specific GOPROXY settings on reverse builders #42971

bcmills commented Dec 3, 2020 •

edited

mengzhuo commented Dec 4, 2020

mengzhuo commented Dec 4, 2020

dmitshur commented Dec 4, 2020 •

edited

gopherbot commented Dec 4, 2020

gopherbot commented Dec 4, 2020

oiooj commented Jan 22, 2021

bcmills commented Feb 4, 2022

bcmills commented Feb 4, 2022 •

edited

bcmills commented Apr 18, 2022 •

edited

dmitshur commented Apr 30, 2022 •

edited

bcmills commented May 2, 2022

mengzhuo commented May 3, 2022

oiooj commented May 3, 2022

bcmills commented May 3, 2022

bcmills commented May 9, 2022

bcmills commented May 9, 2022

x/build: enable builder-specific GOPROXY settings on reverse builders #42971

x/build: enable builder-specific GOPROXY settings on reverse builders #42971

Comments

bcmills commented Dec 3, 2020 • edited

mengzhuo commented Dec 4, 2020

mengzhuo commented Dec 4, 2020

dmitshur commented Dec 4, 2020 • edited

gopherbot commented Dec 4, 2020

gopherbot commented Dec 4, 2020

oiooj commented Jan 22, 2021

bcmills commented Feb 4, 2022

bcmills commented Feb 4, 2022 • edited

bcmills commented Apr 18, 2022 • edited

dmitshur commented Apr 30, 2022 • edited

bcmills commented May 2, 2022

mengzhuo commented May 3, 2022

oiooj commented May 3, 2022

bcmills commented May 3, 2022

bcmills commented May 9, 2022

bcmills commented May 9, 2022

bcmills commented Dec 3, 2020 •

edited

dmitshur commented Dec 4, 2020 •

edited

bcmills commented Feb 4, 2022 •

edited

bcmills commented Apr 18, 2022 •

edited

dmitshur commented Apr 30, 2022 •

edited