-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: runtime package tests consistently fail due to 45 min timeout on darwin-amd64-longtest, only when running via all.bash #60919
Comments
Some observations.
|
In #59634 our theory was that the builder was hitting swap (and may still be, for some tests). Can we measure the low water mark of available memory on the builder during an |
I noticed we use the same GO_TEST_TIMEOUT_SCALE value of 5 for this -longtest builder as for the other ones, which don't have as much performance overhead. Doubling it from 5 to 10 allowed all.bash to complete successfully twice in a row. (Their I checked the output of the Mailed CL 506515 to unblock bringing back this builder during release testing, where its speed is far from being a bottleneck. That said, improving the builder speed can still be done independently, but this issue was about it reproducibly failing to pass at all, which I think will be resolved. |
Change https://go.dev/cl/504524 mentions this issue: |
Change https://go.dev/cl/506515 mentions this issue: |
This reapplies CL 503758, which was rolled back in CL 503516. Trying again after scaling watchdog timeout to take into account that this is a longtest builder and it's expected to take longer, and after doubling its timeout scale value in the builder configuration. Since it's faster than signing, it's not expected to have any effect on the speed of the release process. We also have the option of using the "skip testing" workflow parameter if needing to temporarily skip it again, something I forgot about when rolling it back earlier. For golang/go#29252. For golang/go#60919. Change-Id: I179479b46a31ec5e95d4e9cd8b1f8fedb765880c Reviewed-on: https://go-review.googlesource.com/c/build/+/504524 Reviewed-by: Heschi Kreinick <heschi@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Running all.bash on the darwin-amd64-longtest builder fairly consistently fails with a 45 minute timeout in the runtime package tests. Testing was done using at a near-tip commit on release-branch.go1.21 (ebbff91). From a sample of about 10 runs, there was one time the runtime package tests completed successfully:
But all other times they failed due to a 45 min timeout. (The 45 minutes comes from cmd/dist's default of 180 seconds for package tests, times 3 because cmd/go test are running, times 5 because the builder has GO_TEST_TIMEOUT_SCALE=5.)
More
Can be reproduced in a gomote with:
Here's a complete all.bash output from one such run:
Issue #59634 for x/tools looks related.
CC @golang/release, @prattmic.
The text was updated successfully, but these errors were encountered: