Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/internal/imports: test frequently times out on dragonfly-amd64 builder #34034

Open
bcmills opened this issue Sep 3, 2019 · 8 comments
Open

x/tools/internal/imports: test frequently times out on dragonfly-amd64 builder #34034

bcmills opened this issue Sep 3, 2019 · 8 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly Testing An issue that has been verified to require only test changes, not just a test failure. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented Sep 3, 2019

The test for golang.org/x/tools/internal/imports seems to be timing out on around 25% of runs on the dragonfly-amd64 builder.

Example logs:

The dragonfly-amd64 builder seems to be a bit underpowered in general (see also #29583 and #25796). Perhaps we should set GO_TEST_TIMEOUT_SCALE=2 in its environment and see if that improves the situation for x/tools?

CC @matloob @ianthehat @tdfbsd
@bcmills bcmills added the Testing An issue that has been verified to require only test changes, not just a test failure. label Sep 3, 2019
@gopherbot gopherbot added this to the Unreleased milestone Sep 3, 2019
@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly labels Sep 3, 2019
@bcmills bcmills mentioned this issue Sep 3, 2019
Open
@bcmills bcmills added the Soon This needs to be done soon. (regressions, serious bugs, outages) label Sep 3, 2019
@timdarbydotnet
Copy link

The builder is on a dedicated machine with a Core I5-6600 cpu and 16GB RAM.

@bcmills
Copy link
Contributor Author

bcmills commented Sep 3, 2019

Hrm. That seems like it should be a good enough machine to run those tests without timing out.

All of the timeouts seem to involve either filesystem operations or go command invocations (which themselves involve many filesystem operations). Is it possible that the builder has high disk contention?

@timdarbydotnet
Copy link

The other processes on this machine are very light duty. The only thing that could cause serious filesystem contention is the nightly hammer cleanup process that typically runs around 10am UTC.

These failures feel "racy" to me, in part because they aren't consistent. DragonFly has gone to great lengths in the past year to improve SMP performance and I'm wondering if maybe they removed one lock too many. I have no clue how to troubleshoot that.

@stamblerre stamblerre added the Tools This label describes issues relating to any tools in the x/tools repository. label Sep 9, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Feb 25, 2020 •
edited

Dragonfly isn't one of the first-class ports according to the checkbox on https://build.golang.org. It seems no one is actively working on this. As a result, I don't think it's appropriate for this issue to have the Soon label, which is meant for regressions, serious bugs, and outages.

This issue still needs to be investigated and resolved if possible, but unless I'm missing something, it doesn't meet the bar for Soon.

@dmitshur dmitshur removed the Soon This needs to be done soon. (regressions, serious bugs, outages) label Feb 25, 2020
@timdarbydotnet
Copy link

cc'ing @tuxillo. Please remove me from the issue as I'm no longer maintaining the DragonFly builder.

@tuxillo
Copy link
Contributor

tuxillo commented Feb 26, 2020

There have been some changes to master (or tip as you call it I think) branch in DragonFly which might help this issue. I am not sure if they'll be merged to the soon to be released 5.8 RELEASE, which reminds me at that point we'll have to ask you guys to change the 5_6 builder name to 5_8 :)

@tuxillo
Copy link
Contributor

tuxillo commented Apr 26, 2020

builder 5_8 is now active in the dashboard

@bcmills bcmills mentioned this issue Mar 24, 2021
Closed
tuxillo added a commit to tuxillo/build that referenced this issue Apr 30, 2021
@tuxillo
dashboard: increase GO_TEST_TIMEOUT_SCALE for dragonfly builders
9b4fb80 
This sets the GO_TEST_TIMEOUT_SCALE=2 for dragonfly builders in order
to try to solve intermittent test failures.

For golang/go#45215.
For golang/go#34034.
Fixes golang/go#45216.
@tuxillo tuxillo mentioned this issue Apr 30, 2021
Closed
@gopherbot
Copy link

Change https://golang.org/cl/315329 mentions this issue: dashboard: increase GO_TEST_TIMEOUT_SCALE for dragonfly builders

gopherbot pushed a commit to golang/build that referenced this issue Apr 30, 2021
@tuxillo @dmitshur
dashboard: increase GO_TEST_TIMEOUT_SCALE for dragonfly builders
249a248 
This sets the GO_TEST_TIMEOUT_SCALE=2 for dragonfly builders in order
to try to solve intermittent test failures.

For golang/go#45215.
For golang/go#34034.
Fixes golang/go#45216.

Change-Id: I81dae533c137ff1903bf6b1a52cf9752e8921b7c
GitHub-Last-Rev: 9b4fb80
GitHub-Pull-Request: #35
Reviewed-on: https://go-review.googlesource.com/c/build/+/315329
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Carlos Amedee <carlos@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Dragonfly Testing An issue that has been verified to require only test changes, not just a test failure. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
6 participants
@tuxillo @dmitshur @bcmills @stamblerre @gopherbot @timdarbydotnet