Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: windows-arm64-10 builder occasionally crashes during sequential all.bash test execution #47965

Open
dmitshur opened this issue Aug 25, 2021 · 1 comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Aug 25, 2021

Breaking this out from #47017 as a narrower issue for tracking and investigation purposes.

There still seems to be some instances where running all.bash sequentially (i.e., without any sharding) on the windows-arm64-10 builder runs into a crash. I've observed one occurrence out of 5 test runs on the go1.17 commit this week:

$ go run . -target=windows-arm64 -version=go1.17beta710 -rev=ec5170397c724a8ae440b2bc529f857c86f0e6b1 -watch

run 1 = "ALL TESTS PASSED"
run 2 = a test known to be flaky (issue #44820) failed: runtime.TestGcLastTime failed: "gc_test.go:126: bad last GC time: got 1629865473098425100, want [1629865473098425100, 1629865473001007600]"
run 3 = "windows-arm64: Error: Buildlet https://farmer.golang.org:443 failed heartbeat after 10.005360314s; marking dead; err=timeout waiting for headers"
run 4 = "ALL TESTS PASSED"
run 5 = "ALL TESTS PASSED"

This problem doesn't reproduce with a physical Windows 10 machine, so this may be a qemu emulation bug. Back when #47017 was filed, this problem happened frequently, but it seems to have become less frequent since then.

CC @golang/release.

@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 25, 2021
@dmitshur dmitshur added this to the Unreleased milestone Aug 25, 2021
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 25, 2021
@gopherbot
Copy link

Change https://golang.org/cl/345129 mentions this issue: cmd/release: start running release tests for windows-arm64

gopherbot pushed a commit to golang/build that referenced this issue Aug 25, 2021
By now, all outstanding issues preventing release tests for
windows-arm64 from being turned on by default are resolved:

- Issues golang/go#46406 and golang/go#46502 are fixed.
- The other remaining blocker golang/go#47017 is determined to be a
  builder-specific issue (not reproducible on physical Windows ARM64
  hardware) that by now happens infrequently enough that automated
  retries should be able to cover for future occurrences.
  Issue golang/go#47965 is opened to track progress on this.
- The builder performance is good and allows release tests to complete
  quickly, not adding any bottlenecks to the release process.

If something changes, we can revisit this, but information available
so far suggests it's a good time to start running release tests for
windows/arm64 by default.

Fixes golang/go#47017.
Updates golang/go#47965.

Change-Id: Ie96164821a2f8e795a22ac2d36b7292587a3e117
Reviewed-on: https://go-review.googlesource.com/c/build/+/345129
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

2 participants