Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: LUCI builders fail to fetch dependencies for subrepos #65482

Open
cherrymui opened this issue Feb 2, 2024 · 16 comments
Open

x/build: LUCI builders fail to fetch dependencies for subrepos #65482

cherrymui opened this issue Feb 2, 2024 · 16 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Unfortunate
Milestone

Comments

@cherrymui
Copy link
Member

cherrymui commented Feb 2, 2024

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`

(placeholder for watchflakes)

@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Feb 2, 2024
@gopherbot gopherbot added this to the Unreleased milestone Feb 2, 2024
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-01-08 17:40 x_vulndb-gotip-windows-amd64-longtest vulndb@3a907017 go@10a66d68 [build] (log)
go: cloud.google.com/go@v0.111.0: Get "https://proxy.golang.org/cloud.google.com/go/@v/v0.111.0.mod": dial tcp: lookup proxy.golang.org: getaddrinfow: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server.
2024-01-19 15:09 x_tools-go1.20-linux-riscv64 tools@83c6c25d release-branch.go1.20@a95136a8 [build] (log)
go: github.com/yuin/goldmark@v1.4.13: Get "https://proxy.golang.org/github.com/yuin/goldmark/@v/v1.4.13.mod": unexpected EOF
2024-01-19 16:29 x_tools-go1.22-linux-riscv64 tools@39a25458 release-branch.go1.22@66f8e1e8 [build] (log)
ERROR: rpc error: code = NotFound desc = stream path not found
2024-01-19 18:38 x_tools-go1.21-linux-riscv64 tools@c7ccb517 release-branch.go1.21@2540b143 [build] (log)
go: mvdan.cc/gofumpt@v0.4.0 requires
	github.com/pkg/diff@v0.0.0-20210226163009-20ebb0f2a09e: Get "https://proxy.golang.org/github.com/pkg/diff/@v/v0.0.0-20210226163009-20ebb0f2a09e.mod": read tcp 10.0.0.237:40720->142.250.72.241:443: read: connection reset by peer
2024-01-19 22:06 x_tools-go1.21-linux-riscv64 tools@d0930dbc release-branch.go1.21@2540b143 [build] (log)
go: golang.org/x/vuln@v1.0.1: Get "https://proxy.golang.org/golang.org/x/vuln/@v/v1.0.1.mod": read tcp 10.0.0.237:45686->142.251.46.209:443: read: connection timed out
2024-01-19 22:06 x_tools-go1.22-linux-riscv64 tools@d0930dbc release-branch.go1.22@9a70e17e [build] (log)
ERROR: rpc error: code = NotFound desc = stream path not found
2024-01-19 23:15 x_tools-go1.21-linux-riscv64 tools@e1555a36 release-branch.go1.21@2540b143 [build] (log)
ERROR: rpc error: code = NotFound desc = stream path not found
2024-01-31 16:48 x_net-gotip-darwin-arm64_11 net@b2208d04 go@b8ac61e6 [build] (log)
ERROR: rpc error: code = NotFound desc = stream path not found

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-02-05 18:07 x_tools-go1.21-linux-arm tools@df03d7d4 release-branch.go1.21@2fdad8af [build] (log)
-> unzip /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/cache/download/golang.org/x/text/@v/v0.14.0.zip: open /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/text@v0.14.0/unicode/bidi/bidi_test.go: no space left on device
2024-02-05 18:07 x_tools-go1.22-linux-arm tools@df03d7d4 release-branch.go1.22@b0957cfc [build] (log)
-> unzip /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/cache/download/honnef.co/go/tools/@v/v0.4.6.zip: open /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/honnef.co/go/tools@v0.4.6/go.mod: no space left on device
-> unzip /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v1.2.1.zip: mkdir /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/github.com/!burnt!sushi/toml@v1.2.1/cmd: no space left on device
-> unzip /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/cache/download/gopkg.in/check.v1/@v/v1.0.0-20190902080502-41f04d3bba15.zip: open /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/gopkg.in/check.v1@v1.0.0-20190902080502-41f04d3bba15/checkers.go: no space left on device
-> unzip /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/cache/download/golang.org/x/exp/typeparams/@v/v0.0.0-20221212164502-fae10dda9338.zip: mkdir /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/exp/typeparams@v0.0.0-20221212164502-fae10dda9338/example/genericmethods: no space left on device
-> unzip /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/cache/download/golang.org/x/text/@v/v0.14.0.zip: open /home/swarming/.swarming/w/ir/x/w/gopath/pkg/mod/golang.org/x/text@v0.14.0/cases/cases.go: no space left on device

watchflakes

@cherrymui
Copy link
Member Author

Possibly related to #57372 and/or #55162.

@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 8, 2024
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-02-06 16:04 x_tools-go1.21-linux-riscv64 tools@aee2e764 release-branch.go1.21@f2920803 [build] (log)
go: github.com/BurntSushi/toml@v1.2.1: Get "https://proxy.golang.org/github.com/%21burnt%21sushi/toml/@v/v1.2.1.mod": dial tcp 74.125.137.141:443: i/o timeout
2024-02-06 16:04 x_tools-gotip-linux-riscv64 tools@aee2e764 go@6076edc5 [build] (log)
go: github.com/BurntSushi/toml@v1.2.1: Get "https://proxy.golang.org/github.com/%21burnt%21sushi/toml/@v/v1.2.1.mod": dial tcp 74.125.137.141:443: i/o timeout
2024-02-08 16:25 x_build-go1.22-linux-amd64-misccompile build@b6d280a2 release-branch.go1.22@20107e05 [build] (log)
2024/02/08 08:29:35 run starting
[I2024-02-08T08:29:35.021015-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:29:35.021075-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:29:35.021703-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:29:35.023953-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:29:35.024144-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:29:35.025670-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:29:35.026722-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:29:35.098126-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:29:35.098223-08:00 7463 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
...
**FAILURE**

* go on release-branch.go1.22 ([commit 20107e0](https://go.googlesource.com/go/+/20107e05a609b8f2e61a6b5e8dc258237ad046e7))
* build on master ([commit b6d280a](https://go.googlesource.com/build/+/b6d280a2fb40af865f1985f94638efc5ef28b69f))

Error:
	shard 2: Failed to run "fetch \"golang.org/x/build\" dependencies": exit status 1

Links:
* [shard 2 build page](https://ci.chromium.org/b/8756644795768960417)

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-02-08 15:56 x_vulndb-go1.22-linux-amd64-misccompile vulndb@0c44de67 release-branch.go1.22@20107e05 [build] (log)
2024/02/08 08:31:24 run starting
[I2024-02-08T08:31:24.712075-08:00 3293 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:31:24.712986-08:00 3293 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:31:24.713013-08:00 3293 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:31:24.713088-08:00 3293 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:31:24.713147-08:00 3293 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:31:24.715278-08:00 3293 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:31:24.716020-08:00 3293 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
[I2024-02-08T08:31:26.386977-08:00 3293 0 options.go:162] dropping Batch(len(Data): 1, Meta: <nil>)
2024/02/08 08:31:26 installed tools
...
**FAILURE**

* go on release-branch.go1.22 ([commit 20107e0](https://go.googlesource.com/go/+/20107e05a609b8f2e61a6b5e8dc258237ad046e7))
* vulndb on master ([commit 0c44de6](https://go.googlesource.com/vulndb/+/0c44de67e81f1c6b5091cc4da646a78d676fd1b6))

Error:
	shard 1: Failed to run "fetch \"golang.org/x/vulndb\" dependencies": exit status 1

Links:
* [shard 1 build page](https://ci.chromium.org/b/8756644680858371409)

watchflakes

@bcmills
Copy link
Contributor

bcmills commented Feb 16, 2024

Might this be caused by port exhaustion on the builder machines?

(Compare #52545 (comment).)

@mengzhuo
Copy link
Contributor

ESTAB  0       550164            10.0.0.140:45756           34.98.107.15:443     users:(("rdb",pid=563753,fd=8))

I found this weird connection to Google with RDB, undocumented require service in wiki?

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-03-08 15:33 x_tools-go1.22-linux-riscv64 tools@93c0ca56 release-branch.go1.22@db6097f8 [build] (log)
go: golang.org/x/tools@v0.18.0 requires
	golang.org/x/net@v0.22.0 requires
	golang.org/x/crypto@v0.21.0 requires
	golang.org/x/net@v0.21.0 requires
	golang.org/x/crypto@v0.19.0 requires
	golang.org/x/net@v0.10.0 requires
	golang.org/x/sys@v0.8.0: Get "https://proxy.golang.org/golang.org/x/sys/@v/v0.8.0.mod": read tcp 10.0.0.203:35238->142.250.189.177:443: read: connection timed out
2024-03-08 16:23 x_tools-go1.22-linux-riscv64 tools@9b64301e release-branch.go1.22@db6097f8 [build] (log)
go: golang.org/x/telemetry@v0.0.0-20240228155512-f48c80bd79b2 requires
	golang.org/x/sys@v0.17.0: Get "https://proxy.golang.org/golang.org/x/sys/@v/v0.17.0.mod": read tcp 10.0.0.203:42128->142.250.72.177:443: read: connection timed out

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-03-13 17:06 x_telemetry-go1.21-linux-riscv64 telemetry@d1a75fc0 release-branch.go1.21@140b37d6 [build] (log)
go: golang.org/x/mod@v0.16.0: Get "https://goproxy.cn/golang.org/x/mod/@v/v0.16.0.mod": dial tcp: lookup goproxy.cn on 10.0.0.1:53: read udp 10.0.0.203:45180->10.0.0.1:53: i/o timeout

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-03-13 20:17 x_telemetry-go1.21-linux-arm64 telemetry@ffa7a4a4 release-branch.go1.21@140b37d6 [build] (log)
ERROR: rpc error: code = NotFound desc = stream path not found

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-03-14 17:37 x_telemetry-go1.21-darwin-amd64_12 telemetry@1cb5a0da release-branch.go1.21@140b37d6 [build] (log)
ERROR: start fetch: rpc error: code = NotFound desc = stream path not found

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-03-13 17:06 x_telemetry-gotip-linux-riscv64 telemetry@d1a75fc0 go@0a6f05e3 [build] (log)
go: cloud.google.com/go@v0.110.8: Get "https://goproxy.cn/cloud.google.com/go/@v/v0.110.8.mod": dial tcp: lookup goproxy.cn on 10.0.0.1:53: no such host

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-03-19 19:46 x_tools-go1.21-linux-riscv64 tools@dd526462 release-branch.go1.21@140b37d6 [build] (log)
go: github.com/yuin/goldmark@v1.4.13: Get "https://goproxy.cn/github.com/yuin/goldmark/@v/v1.4.13.mod": dial tcp: lookup goproxy.cn on 10.0.0.1:53: read udp 10.0.0.140:47475->10.0.0.1:53: i/o timeout
2024-03-19 20:46 x_tools-go1.21-linux-riscv64 tools@d0f7dcef release-branch.go1.21@140b37d6 [build] (log)
go: github.com/yuin/goldmark@v1.4.13: Get "https://goproxy.cn/github.com/yuin/goldmark/@v/v1.4.13.mod": dial tcp: lookup goproxy.cn on 10.0.0.1:53: read udp 10.0.0.237:33595->10.0.0.1:53: i/o timeout

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-04-19 17:10 x_build-go1.22-darwin-amd64_14 build@46790409 release-branch.go1.22@d6c972ad [build] (log)
go: dmitri.shuralyov.com/app/changes@v0.0.0-20191223015216-e22f40b36873: Get "https://proxy.golang.org/dmitri.shuralyov.com/app/changes/@v/v0.0.0-20191223015216-e22f40b36873.mod": dial tcp: lookup proxy.golang.org: no such host

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-04-03 20:36 x_oauth2-gotip-linux-riscv64 oauth2@d0e617c5 go@58c5db31 [build] (log)
go: cloud.google.com/go/compute@v1.20.1: Get "https://goproxy.cn/cloud.google.com/go/compute/@v/v1.20.1.mod": dial tcp: lookup goproxy.cn on 10.0.0.1:53: read udp 10.0.0.203:47720->10.0.0.1:53: i/o timeout

watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- mode == "build" && log ~ `Failed to run "fetch .*" dependencies`
2024-04-26 17:54 x_tools-go1.21-linux-riscv64 tools@bcec0994 release-branch.go1.21@891ac91e [build] (log)
go: github.com/google/go-cmp@v0.6.0: Get "https://goproxy.cn/github.com/google/go-cmp/@v/v0.6.0.mod": dial tcp: lookup goproxy.cn on 10.0.0.1:53: read udp 10.0.0.237:40213->10.0.0.1:53: i/o timeout

watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Unfortunate
Projects
Status: No status
Development

No branches or pull requests

5 participants