-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: linux-arm.*-aws
TryBots are often stragglers
#54679
Comments
I did some measurements a while ago analyzing builds from May 1 - June 10. Paraphrasing the results I found:
Rerunning the analysis since August 1, I get:
I think the main difference is that https://go.dev/cl/419077 changed most GCE builders from 4vCPU/16GB to 16vCPU/64GB instances, significantly speeding them up, but linux-arm{64}-aws are still on 4vCPU/16GB instances (m6g.xlarge). There is a 16vCPU/64GB instance type available (m6g.4xlarge), which we could potentially switch to. |
These are the post-submit builds, but data is similar for trybots (both arm builders are actually slightly faster in trybots). That said, this data doesn't seem to match your observation "linux-arm-aws and linux-arm64-aws TryBots always seem to take 15 minutes or more." For |
Change https://go.dev/cl/449615 mentions this issue: |
This change replaces linux-arm64-aws with linux-arm64. The new builder runs on GCP and has an increased set of resources. This should provide a performance improvement. The builder will now be used: - To release linux-arm64 - As the linux-arm64 trybot Fixes golang/go#53851 Updates golang/go#54679 Updates golang/go#40561 Change-Id: I9abfe2ed1d1bd8a68bba1e1f924637e829026e6d Reviewed-on: https://go-review.googlesource.com/c/build/+/449615 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Jenny Rakoczy <jenny@golang.org>
I noticed this again on https://farmer.golang.org/try?commit=41ddcbff. |
On https://farmer.golang.org/try?commit=14f8fad9, The next-slowest builder ( |
@golang/release, if speeding this up isn't a priority at the moment, any chance we could demote the |
Here's the event log from the aforementioned 20-minute TryBot run, for reference:
There's no one particular thing that's slow, it just takes 20 minutes to get through everything. The "saved -7 seconds" part in the summary stands out:
The builder is configured to have 1 try-helper, so its execution should be sharded across 2 instances. However, it seems it failed to get the helper due to "ResourceNotReady" and so it ran everything on the one instance (hence there was no time saved above):
If the builder is so slow that having it be a post-submit builder only is more helpful than having it as a trybot, making that change for now seems reasonable to me. It would be replaced by a misc-compile trybot that would provide bare minimum coverage. |
A couple of thoughts about this:
For the time being, the post-submit builder route seems like the right call. |
Change https://go.dev/cl/468535 mentions this issue: |
On several CLs lately, I've seen nearly all of the builders complete within 10 minutes or so, but the
linux-arm-aws
andlinux-arm64-aws
TryBots always seem to take 15 minutes or more.It appears that these builders aren't currently sharded at all:
https://cs.opensource.google/go/x/build/+/master:dashboard/builders.go;l=2417;drc=9ca9dc28e477c63197a65122b31a98146c49d07d
https://cs.opensource.google/go/x/build/+/master:dashboard/builders.go;l=2434;drc=9ca9dc28e477c63197a65122b31a98146c49d07d
Is there something we can do to bring the latency for these builders up to par with the other TryBots? (Perhaps enable sharding?)
(CC @prattmic, who I think has done some recent latency measurements)
The text was updated successfully, but these errors were encountered: