-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: misc-compile TryBots are too slow #32632
Comments
On https://farmer.golang.org/try?commit=c29ef957, I'm seeing multiple |
On https://farmer.golang.org/try?commit=5b2140c8, I'm again seeing |
I too have observed this: When these trybots take the longest, they become a bottleneck for overall TryBot completion speed (e.g., #17104). The "misc-compile" trybots do much less work than normal trybots: they check only that compilation is successful, but don't execute tests. They're generally split by OS, matching all architectures available: Those architectures are currently tested sequentially. So one possible explanation is if newly added architectures start to get matched, and there become too many for sequential testing to be viable. If that's the case, we can fix this by adjusting the split to avoid putting too much work into any single trybot. |
I observed a case where Looking at its logs confirmed it was building as many as 5 different architectures, all sequentially:
So the root problem here is pretty clear: we need to parallelize the workload better for OSes that have many architectures. A fix can look like this: -addMiscCompile("-openbsd", "^openbsd-") // 4: amd64, 386, arm, arm64
+addMiscCompile("-openbsd-amd", "^openbsd-(386|amd64)$") // 2: 386, amd64
+addMiscCompile("-openbsd-arm", "^openbsd-(arm|arm64)$") // 2: arm, arm64
+addMiscCompile("-openbsd-mips", "^openbsd-(mips64)$") // 1: mips64 (We have good test coverage for missing trybots, so we'll know if something needs updating.) |
I am looking at a TryBot run where misc-compile takes longer than 15 minutes, and longer than linux-amd64-longtest. If there is an easy fix, even if hacky, I would recommend applying it, because this is adding significant latency to everyone's development cycle. TryBots are supposed to take at most 5 minutes, IIRC, so even running two architectures in a row would probably be too slow. (15min / 4 * 2 = 7.5min) |
Change https://golang.org/cl/313210 mentions this issue: |
With the work done to migrate x/build to OpenCensus in CL 303669, we're closer to starting to measure and track TryBot completion time, which has been a blocker for making progress on #17104. However, the problem here is quite clear and a simple fix is available, so I've sent CL 313210 that implements the suggestion in #32632 (comment). Our updated target time for TryBot completion time is 10 minutes (see subject of issue #17104), so going with at most |
Change https://golang.org/cl/313533 mentions this issue: |
Some TryBot builders are disabled or configured with a custom try policy that doesn't test the master branch of the main Go repository. For example, openbsd-386-68 is configured with: tryBot: explicitTrySet("sys"), So it's only a TryBot for the x/sys repo, not the main one. Modify TestTryBotsCompileAllPorts to not consider such TryBots when determining whether a port has sufficient pre-submit test coverage. This will help in the next change which re-arranges target selection between misc-compile TryBots. For golang/go#32632. Change-Id: If393610d11b04b75133a4f10f89cfdfff9dbaece Reviewed-on: https://go-review.googlesource.com/c/build/+/313533 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Heschi Kreinick <heschi@google.com>
I am waiting for try-bots on https://go-review.googlesource.com/c/go/+/177959, and generally try-bots are finished withing about 5 minutes, but misc-compile-mips takes more than 11 minutes from https://farmer.golang.org/try?commit=fa91b668
Every bot is finished, except misc-compile-mips. Maybe misc-compile-mips should not be part of try-bots, if it is soo slow.
Alex
The text was updated successfully, but these errors were encountered: