New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: darwin-amd64
LUCI builders are sometimes too slow
#65040
Comments
The "darwin-amd64-13" builder on the old dashboard doesn't have Another possibility is that we're running more tests on Mac builders in LUCI than we are on the old dashboard. For one, the old dashboard uses the |
We discussed and looked into this a bit. This turns out not to be a consistent slowness: looking at https://ci.chromium.org/ui/p/golang/builders/ci/gotip-darwin-amd64_13 briefly shows many builds complete successfully in 15-20 min, with some taking 10 or 30 min, but the failures are taking an hour or longer and reaching timeout. This builder at this time has two different providers. The two recent timeouts happened on the same |
darwin-amd64_13
LUCI builder is too slowdarwin-amd64_13
LUCI builder is sometimes too slow or stalls
Update on this. I took a look at median, worst case, etc build times for the main repo builder by bot hostname and there definitely seems to be a lot of variance between different hosts: nearly 4x difference in median runtime between the fastest and slowest hosts. My first thought was slow network on some hosts, so I eyeballed the difference between a run on the fastest and slowest hosts and found that the slowdown seems to impact pretty much all packages. e.g., cmd/go 139s -> 540s Even small packages, like cmd/gofmt 0.064s -> 0.79s Some are similar though crypto/md5 0.021s -> 0.024s So it seems unrelated to network. That slowest machine is https://chromium-swarm.appspot.com/bot?id=darwin-amd64-13--ac2a2248-1530-4adb-bb91-ae854aa7c79c.golang.ca.macservice.goog. Interestingly, it seems to be much faster today. Previously x/tools was timing out, now it is ~15m. x/build is faster, etc. This definitely still needs more investigation. |
Found new dashboard test flakes for:
2023-12-08 00:50 gotip-darwin-amd64-nocgo go@4bf1ca4b cmd/fix.TestRewrite/jni.ArrayElement (log)
2024-01-08 16:00 gotip-darwin-amd64_13 go@881869dd cmd/go.TestScript (log)
2024-01-09 15:29 gotip-darwin-amd64_13 go@aba18d5b cmd/fix.TestRewrite (log)
2024-01-09 15:29 gotip-darwin-amd64_13 go@aba18d5b cmd/go.TestScript (log)
2024-01-10 17:35 gotip-darwin-amd64_11 go@e9b3ff15 cmd/cgo/internal/testplugin.TestIssue22175 (log)
2024-01-10 17:35 gotip-darwin-amd64_11 go@e9b3ff15 cmd/compile/internal/loopvar.TestLoopVarHashes (log)
2024-01-10 17:35 gotip-darwin-amd64_11 go@e9b3ff15 cmd/compile/internal/ssa.TestFmaHash (log)
2024-01-17 19:47 gotip-darwin-amd64_13 go@92d7169a cmd/fix.TestRewrite (log)
2024-01-17 19:47 gotip-darwin-amd64_13 go@92d7169a cmd/go.TestScript (log)
|
There seem to be timeouts on other LUCI darwin builders (other than 13). I put them all here for now. Feel free to split if this is really specific to 13. Thanks. |
Found new dashboard test flakes for:
2024-01-22 21:47 gotip-darwin-amd64_13 go@b7ae16e0 cmd/fix.TestRewrite (log)
2024-01-22 21:47 gotip-darwin-amd64_13 go@b7ae16e0 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2024-01-04 17:31 go1.22-darwin-amd64_13 release-branch.go1.22@15dcdeb5 cmd/fix.TestRewrite [ABORT] (log)
2024-01-04 17:31 go1.22-darwin-amd64_13 release-branch.go1.22@15dcdeb5 cmd/go.TestScript (log)
2024-01-08 18:52 go1.22-darwin-amd64_11 release-branch.go1.22@8eaa7935 cmd/cgo/internal/testplugin.TestIssue22175 [ABORT] (log)
2024-01-08 18:52 go1.22-darwin-amd64_11 release-branch.go1.22@8eaa7935 cmd/compile/internal/loopvar.TestLoopVarHashes [ABORT] (log)
2024-01-08 18:52 go1.22-darwin-amd64_11 release-branch.go1.22@8eaa7935 cmd/compile/internal/ssa.TestFmaHash (log)
2024-01-10 20:16 go1.21-darwin-amd64_11 release-branch.go1.21@2540b143 cmd/cgo/internal/testplugin.TestIssue22295 [ABORT] (log)
2024-01-10 20:16 go1.21-darwin-amd64_11 release-branch.go1.21@2540b143 cmd/compile/internal/loopvar.TestLoopVarHashes [ABORT] (log)
2024-01-10 20:16 go1.21-darwin-amd64_11 release-branch.go1.21@2540b143 cmd/compile/internal/ssa.TestFmaHash (log)
|
Found new dashboard test flakes for:
2024-01-18 23:21 x_build-gotip-darwin-amd64_13 build@911ff433 go@66d34c7d x/build/internal/workflow.TestWatchdog/success (log)
|
Found new dashboard test flakes for:
2024-01-08 18:41 x_tools-gotip-darwin-amd64_13 tools@25a0e9d3 go@b702e043 x/tools/gopls/internal/test/integration/misc.TestStaticcheckRelatedInfo/default [ABORT] (log)
|
darwin-amd64_13
LUCI builder is sometimes too slow or stallsdarwin-amd64
LUCI builders are sometimes too slow
Found new dashboard test flakes for:
2024-01-30 18:58 gotip-darwin-amd64_13 go@0b1b6c15 runtime/pprof.TestVMInfo (log)
2024-01-30 23:37 gotip-darwin-amd64_12 go@ae457e81 runtime/pprof.TestVMInfo (log)
2024-01-30 23:37 gotip-darwin-amd64_14 go@ae457e81 runtime/pprof.TestVMInfo (log)
2024-01-31 14:20 gotip-darwin-amd64-nocgo go@13766fe7 runtime/pprof.TestVMInfo (log)
2024-01-31 16:50 gotip-darwin-amd64_13 go@1bb947b2 runtime/pprof.TestVMInfo (log)
2024-01-31 19:56 gotip-darwin-amd64_12 go@79738217 runtime/pprof.TestVMInfo (log)
2024-01-31 20:51 gotip-darwin-amd64_12 go@5c0d0929 runtime/pprof.TestVMInfo (log)
2024-01-31 20:51 gotip-darwin-amd64_13 go@5c0d0929 runtime/pprof.TestVMInfo (log)
2024-01-31 23:21 gotip-darwin-amd64_12 go@2f6a25f4 runtime/pprof.TestVMInfo (log)
2024-01-31 23:21 gotip-darwin-amd64_14 go@2f6a25f4 runtime/pprof.TestVMInfo (log)
2024-02-01 15:34 gotip-darwin-amd64_13 go@5b6cd3d0 runtime/pprof.TestVMInfo (log)
2024-02-01 18:09 gotip-darwin-amd64_12 go@18840865 runtime/pprof.TestVMInfo (log)
2024-02-01 19:29 gotip-darwin-amd64_13 go@29746b48 runtime/pprof.TestVMInfo (log)
2024-02-01 20:09 gotip-darwin-amd64_12 go@d864eac2 runtime/pprof.TestVMInfo (log)
2024-02-01 21:02 gotip-darwin-amd64_12 go@829f2ce3 runtime/pprof.TestVMInfo (log)
2024-02-01 21:42 gotip-darwin-amd64_13 go@c9d88ea2 runtime/pprof.TestVMInfo (log)
2024-02-01 21:42 gotip-darwin-amd64_14 go@c9d88ea2 runtime/pprof.TestVMInfo (log)
2024-02-01 22:47 gotip-darwin-amd64_13 go@ac08c05d runtime/pprof.TestVMInfo (log)
2024-02-01 22:57 gotip-darwin-amd64-nocgo go@6d3c1ce8 runtime/pprof.TestVMInfo (log)
2024-02-01 22:57 gotip-darwin-amd64_13 go@6d3c1ce8 runtime/pprof.TestVMInfo (log)
2024-02-01 22:59 gotip-darwin-amd64-nocgo go@117164f9 runtime/pprof.TestVMInfo (log)
2024-02-01 22:59 gotip-darwin-amd64_12 go@117164f9 runtime/pprof.TestVMInfo (log)
2024-02-02 15:13 gotip-darwin-amd64_12 go@244ccf47 runtime/pprof.TestVMInfo (log)
2024-02-02 15:22 gotip-darwin-amd64_13 go@149db960 runtime/pprof.TestVMInfo (log)
2024-02-02 17:06 gotip-darwin-amd64_12 go@3b83560b runtime/pprof.TestVMInfo (log)
|
Filed #65468 for |
Found new dashboard test flakes for:
2024-02-02 18:35 gotip-darwin-amd64_12 go@3d20a327 runtime/pprof.TestVMInfo (log)
2024-02-02 22:08 gotip-darwin-amd64_12 go@9819d826 runtime/pprof.TestVMInfo (log)
|
TestVMInfo failure is #62352. It seems that it matches this because it keeps retrying until it timed out, so |
I changed the pattern to "default", so the other issue should take priority. |
Found new dashboard test flakes for:
2024-02-07 19:42 x_tools-gotip-darwin-amd64-longtest tools@76ef6b6a go@1400b268 x/tools/gopls/internal/test/integration/misc.TestWindowsVendoring_Issue56291/default [ABORT] (log)
|
Found new dashboard test flakes for:
2024-02-07 21:58 x_tools-go1.22-darwin-amd64-longtest tools@5fcc6273 release-branch.go1.22@dd31ad7e x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckError/default [ABORT] (log)
2024-02-07 21:58 x_tools-go1.22-darwin-amd64-longtest tools@5fcc6273 release-branch.go1.22@dd31ad7e x/tools/gopls/internal/test/integration/misc.TestWindowsVendoring_Issue56291/default (log)
2024-02-08 13:20 x_tools-go1.22-darwin-amd64-longtest tools@f4fa7a75 release-branch.go1.22@20107e05 x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckStd/experimental [ABORT] (log)
2024-02-08 13:20 x_tools-gotip-darwin-amd64-longtest tools@f4fa7a75 go@58fea5df x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckError2/experimental [ABORT] (log)
|
Found new dashboard test flakes for:
2024-02-09 17:28 x_tools-gotip-darwin-amd64-longtest tools@a5af84e3 go@4a7f3ac8 x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckStd/experimental [ABORT] (log)
|
Found new dashboard test flakes for:
2024-02-12 19:37 x_tools-go1.22-darwin-amd64-longtest tools@50b4f1b1 release-branch.go1.22@20107e05 x/tools/gopls/internal/test/integration/misc.TestGovulncheckInfo/default [ABORT] (log)
2024-02-14 21:26 x_tools-go1.22-darwin-amd64-longtest tools@e3254052 release-branch.go1.22@20107e05 x/tools/gopls/internal/test/integration/misc.TestRunVulncheckPackageDiagnostics/default/forwarded [ABORT] (log)
|
Found new dashboard test flakes for:
2024-02-16 15:51 go1.22-darwin-amd64_10.15 release-branch.go1.22@6fbd01a7 cmd/cgo/internal/testplugin.TestMethod [ABORT] (log)
2024-02-16 15:51 go1.22-darwin-amd64_10.15 release-branch.go1.22@6fbd01a7 cmd/compile/internal/loopvar.TestLoopVarHashes (log)
2024-02-16 15:51 go1.22-darwin-amd64_10.15 release-branch.go1.22@6fbd01a7 cmd/compile/internal/ssa.TestFmaHash (log)
|
Found new dashboard test flakes for:
2024-02-22 15:37 gotip-darwin-amd64-longtest go@638b9023 cmd/go.TestScript/repro_build (log)
2024-02-22 19:32 gotip-darwin-amd64-longtest go@db57e5a0 cmd/go.TestScript/build_cc_cache_issue64423 (log)
|
Found new dashboard test flakes for:
2024-02-14 19:47 x_pkgsite-metrics-go1.21-darwin-amd64-longtest pkgsite-metrics@ffcab83d release-branch.go1.21@f38fca30 x/pkgsite-metrics.Test [ABORT] (log)
2024-02-23 21:21 gotip-darwin-amd64-longtest go@72030635 cmd/go.TestScript/build_trimpath_cgo (log)
2024-02-23 21:21 gotip-darwin-amd64_11 go@b899e0b8 cmd/cgo/internal/testcarchive.TestExtar [ABORT] (log)
2024-02-23 21:21 gotip-darwin-amd64_11 go@b899e0b8 cmd/cgo/internal/testplugin.TestIssue18676 [ABORT] (log)
2024-02-23 21:21 gotip-darwin-amd64_11 go@b899e0b8 cmd/compile/internal/loopvar.TestLoopVarHashes [ABORT] (log)
|
Found new dashboard test flakes for:
2024-02-28 00:05 gotip-darwin-amd64-longtest go@ff8a2c0a cmd/go.TestScript/test_trimpath_test_suffix (log)
|
Found new dashboard test flakes for:
2024-02-28 17:41 gotip-darwin-amd64-longtest go@28e0052e reflect:mayMoreStackMove.TestChanOfGC [ABORT] (log)
2024-02-28 20:21 gotip-darwin-amd64-longtest go@88b2f250 reflect:mayMoreStackMove.TestChanOfGC [ABORT] (log)
|
Found new dashboard test flakes for:
2024-02-28 22:57 gotip-darwin-amd64-longtest go@288615dd reflect:mayMoreStackMove.TestChanOfGC [ABORT] (log)
|
Found new dashboard test flakes for:
2024-03-04 17:27 x_net-go1.21-darwin-amd64_12 net@c289c7ab release-branch.go1.21@3643147a x/net/quic.TestUDPSourceUnspecified/udp4/udp/unspec [ABORT] (log)
|
Found new dashboard test flakes for:
2024-03-08 17:01 gotip-darwin-amd64-longtest go@a46285f8 net.TestCloseRead [ABORT] (log)
|
Found new dashboard test flakes for:
2024-03-18 16:57 gotip-darwin-amd64-longtest go@f40bf748 cmd/go.TestScript/mod_vendor_auto (log)
|
Found new dashboard test flakes for:
2024-03-19 15:19 gotip-darwin-amd64-longtest go@a73af5d9 cmd/go.TestScript/build_dash_x (log)
|
Found new dashboard test flakes for:
2024-03-20 10:09 gotip-darwin-amd64-longtest go@fba54f63 cmd/go.TestScript/build_dash_x (log)
2024-03-20 19:08 gotip-darwin-amd64-longtest go@6af27c49 cmd/go.TestScript/repro_build (log)
2024-03-21 16:12 gotip-darwin-amd64-longtest go@d9f97467 cmd/go.TestScript/build_cache_output (log)
2024-03-21 19:45 gotip-darwin-amd64-longtest go@802473cf cmd/go.TestScript/test_vet (log)
2024-03-21 20:00 gotip-darwin-amd64-longtest go@aa1b50e1 cmd/go.TestScript/test_rebuildall (log)
2024-03-21 22:02 gotip-darwin-amd64-longtest go@41bd9a51 cmd/go.TestScript/build_GOTMPDIR (log)
2024-03-21 22:02 gotip-darwin-amd64-longtest go@41bd9a51 cmd/go.TestScript/test_race_install (log)
|
Found new dashboard test flakes for:
2024-03-22 16:39 gotip-darwin-amd64-longtest go@5f5b20c4 cmd/go.TestScript/build_GOTMPDIR (log)
|
Found new dashboard test flakes for:
2024-03-25 09:38 gotip-darwin-amd64-longtest go@c4792e60 cmd/go.TestScript/build_pgo (log)
|
Found new dashboard test flakes for:
2024-03-25 19:21 gotip-darwin-amd64-longtest go@2c677361 cmd/go.TestScript/build_cc_cache_issue64423 (log)
|
Found new dashboard test flakes for:
2024-03-26 15:26 gotip-darwin-amd64_11 go@340f73ce cmd/api:check.TestCheck [ABORT] (log)
2024-03-26 19:41 go1.22-darwin-amd64_11 release-branch.go1.22@e23707b5 cmd/cgo/internal/testplugin.TestIssue22175 [ABORT] (log)
2024-03-26 19:41 go1.22-darwin-amd64_11 release-branch.go1.22@e23707b5 cmd/compile/internal/loopvar.TestLoopVarHashes [ABORT] (log)
2024-03-26 19:41 go1.22-darwin-amd64_11 release-branch.go1.22@e23707b5 cmd/compile/internal/ssa.TestFmaHash (log)
2024-03-26 19:58 gotip-darwin-amd64-longtest go@42178776 cmd/go.TestScript/build_pgo (log)
2024-03-27 18:23 gotip-darwin-amd64_11 go@d4cc35c4 cmd/api:check.TestCheck [ABORT] (log)
2024-03-27 21:58 gotip-darwin-amd64_11 go@2e1003e2 cmd/cgo/internal/testplugin.TestIssue24351 [ABORT] (log)
2024-03-27 21:58 gotip-darwin-amd64_11 go@2e1003e2 cmd/compile/internal/loopvar.TestLoopVarHashes (log)
2024-03-27 21:58 gotip-darwin-amd64_11 go@2e1003e2 cmd/compile/internal/ssa.TestDebugLines_53456 (log)
2024-03-29 16:41 gotip-darwin-amd64-longtest go@8f59b28f cmd/go.TestScript/build_cache_output (log)
2024-03-29 16:46 gotip-darwin-amd64_11 go@9a028e14 cmd/cgo/internal/testplugin.TestIssue25756 [ABORT] (log)
2024-03-29 16:46 gotip-darwin-amd64_11 go@9a028e14 cmd/compile/internal/loopvar.TestLoopVarHashes (log)
2024-03-29 16:46 gotip-darwin-amd64_11 go@9a028e14 cmd/compile/internal/ssa.TestFmaHash (log)
2024-03-29 21:55 gotip-darwin-amd64_11 go@7979c8f5 cmd/internal/testdir.Test/fixedbugs/issue46234.go (log)
2024-04-08 21:43 gotip-darwin-amd64-longtest go@e995aa95 cmd/go.TestScript/build_cache_output (log)
|
Found new dashboard test flakes for:
2024-04-22 20:21 gotip-darwin-amd64-longtest go@6737f4ce cmd/go.TestScript/test_vet (log)
|
Found new dashboard test flakes for:
2024-04-22 22:06 gotip-darwin-amd64-longtest go@674657e1 cmd/go.TestScript/vet_flags (log)
|
Found new dashboard test flakes for:
2024-04-22 22:06 gotip-darwin-amd64-longtest go@5a3b6400 cmd/go.TestScript/reuse_git (log)
|
Found new dashboard test flakes for:
2024-04-23 02:53 gotip-darwin-amd64-longtest go@0304d035 cmd/go.TestScript/vet_flags (log)
|
Found new dashboard test flakes for:
2024-04-24 18:15 gotip-darwin-amd64-longtest go@e6891188 cmd/go.TestScript/cover_build_simple (log)
2024-04-24 21:50 gotip-darwin-amd64-longtest go@9b9de261 cmd/go.TestScript/test_vet (log)
|
Found new dashboard test flakes for:
2024-04-26 22:55 gotip-darwin-amd64-longtest go@1c470499 cmd/go.TestScript/build_pgo (log)
|
Found new dashboard test flakes for:
2024-04-29 13:58 gotip-darwin-amd64-longtest go@a81c8b3b cmd/go.TestScript/build_pgo (log)
|
Found new dashboard test flakes for:
2024-05-01 03:56 gotip-darwin-amd64-longtest go@c5698e31 cmd/go.TestScript/test_fuzz_test_race (log)
2024-05-01 17:37 gotip-darwin-amd64-longtest go@660a0719 cmd/go.TestScript/cover_atomic_pkgall (log)
|
Found new dashboard test flakes for:
2024-05-02 22:18 gotip-darwin-amd64-longtest go@e602625c cmd/go.TestScript/build_trimpath_cgo (log)
|
Found new dashboard test flakes for:
2024-05-02 22:14 gotip-darwin-amd64-longtest go@b64e5e38 cmd/go.TestScript/build_tag_goexperiment (log)
|
Found new dashboard test flakes for:
2024-05-03 12:00 gotip-darwin-amd64-longtest go@2f5b420f cmd/go.TestScript/test_race_install (log)
|
Found new dashboard test flakes for:
2024-05-03 16:40 gotip-darwin-amd64-longtest go@18dd6826 cmd/go.TestScript/vet_flags (log)
2024-05-03 17:28 gotip-darwin-amd64-longtest go@7b5206bd cmd/go.TestScript/build_cc_cache_issue64423 (log)
|
Found new dashboard test flakes for:
2024-05-03 19:05 gotip-darwin-amd64-longtest go@99fc3179 cmd/go.TestScript/cgo_path (log)
|
Found new dashboard test flakes for:
2024-05-04 07:51 gotip-darwin-amd64-longtest go@20130cc3 cmd/go.TestScript/build_pgo (log)
|
Go version
aba18d5
Output of
go env
in your module/workspace:What did you do?
Check status of https://ci.chromium.org/p/golang/g/go-gotip/console.
What did you see happen?
Multiple failures involving timeouts on the
darwin-amd64_13
builder:What did you expect to see?
No timeouts:
go test -short cmd/go
takes <20s to run locally, so either the builders should be faster, or they should set aGO_TEST_TIMEOUT_SCALE
that is long enough to reliably run the tests.The only timeouts on the builders should be for (a) true deadlocks, and (b) tests that take a similarly long time (approaching 3 minutes) when run locally.
The text was updated successfully, but these errors were encountered: