Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: linux-arm64 packet builders are offline #35524

Closed
bradfitz opened this issue Nov 12, 2019 · 1 comment
Closed

x/build: linux-arm64 packet builders are offline #35524

bradfitz opened this issue Nov 12, 2019 · 1 comment
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@bradfitz
Copy link
Contributor

https://farmer.golang.org/ reports:

Packet linux/arm64 machines [docs]:
packet01 missing, not seen for 3h38m18s
packet02 missing, not seen for 3h38m17s
packet03 missing, not seen for 3h38m21s
packet04 missing, not seen for 3h38m22s
packet05 missing, not seen for 3h38m21s
packet06 missing, not seen for 3h38m14s
packet07 missing, not seen for 3h38m20s
packet08 missing, not seen for 3h38m23s
packet09 missing, not seen for 3h38m24s
packet10 missing, not seen for 3h38m15s
packet11 missing, not seen for 3h38m27s
packet12 missing, not seen for 3h38m22s
packet13 missing, not seen for 3h38m17s
packet14 missing, not seen for 3h38m15s
packet15 missing, not seen for 3h38m22s
packet16 missing, not seen for 3h37m26s
packet17 missing, not seen for 3h38m20s
packet18 missing, not seen for 3h38m21s
packet19 missing, not seen for 3h38m17s
packet20 missing, not seen for 3h38m21s
20 machines missing, 100% of capacity

So we lost them around 7pm US/Pacific (Nov 11)

/cc @cagedmantis @dmitshur

@gopherbot gopherbot added this to the Unreleased milestone Nov 12, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 12, 2019
@cagedmantis cagedmantis self-assigned this Nov 12, 2019
@cagedmantis
Copy link
Contributor

cagedmantis commented Nov 12, 2019

The builders are back online. The outage was due to a hardware issue with the network interface around that time. The service provider is monitoring it and will inform us if this will require any preventative maintenance.

There has been some interest expressed within the team to be alerted (via an email or bot alert) when there is an outage with a builder. I will create an issue for this if one doesn't already exist.

Follow Up: Found this issue: #32372

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

3 participants