Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: one ppc64le osuosl builder is configured differently from the rest, causing failures #48614

Closed
pmur opened this issue Sep 24, 2021 · 10 comments
Closed

x/build: one ppc64le osuosl builder is configured differently from the rest, causing failures #48614

pmur opened this issue Sep 24, 2021 · 10 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Release Team
Milestone
Unreleased

Comments

@pmur
Copy link
Contributor

pmur commented Sep 24, 2021

This is related to #48554, it would appear some power9 builders are configured differently.

Sometimes the test to open /dev/rtc0 is either skipped or passes, and sometime it fails. It is not clear if this test was always skipped previously (e.g the builder had no RTC device, or was prohibited from opening it, or has an RTC which supports alarms).

Are all the power9 builders provisioned and configured identically? Likewise, is it possible to tag golang.org/x/sys CI logs with more information about which builder they have run on (I am not sure if I am overlooking this one)?

Thanks!
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Sep 24, 2021
@gopherbot gopherbot added this to the Unreleased milestone Sep 24, 2021
@pmur
Copy link
Contributor Author

pmur commented Sep 24, 2021

@laboger fyi.

@pmur
Copy link
Contributor Author

pmur commented Sep 27, 2021

Looking into this further, it seems like we have one ppc64le power9 builder which isn't configured like the others. From farmers.golang.org I see a builder named go-ppc64le-p9 which doesn't set the HOSTNAME envvar. This matches the behavior of the failures on golang.org/x/sys.

Is this builder supposed to exist?

@laboger
Copy link
Contributor

laboger commented Sep 27, 2021

Also, this odd builder is not mentioned in x/build/env/linux-ppc64le/osuosl/NOTES and has a different distro than the others.

@laboger
Copy link
Contributor

laboger commented Sep 29, 2021

@bcmills We have found that the failures from #48554 were due to this rogue p9 builder that is not set up like the others so fails in a different way. This builder is not documented and not set up like the others. This builder should be stopped/disabled so people don't waste time tracking down bugs on this builder.

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 4, 2021
@mknyszek mknyszek changed the title x/build: ppc64le osuosl builder configuration question x/build: one ppc64le osuosl builder is configured differently from the rest, causing failures Oct 4, 2021
@mknyszek
Copy link
Contributor

mknyszek commented Oct 4, 2021

CC @dmitshur @toothrot

@pmur
Copy link
Contributor Author

pmur commented Oct 4, 2021

I would like to shutdown the go-ppc64le-p9 builder. Does anyone object? Also, what steps are needed to deactivate a builder?

@cagedmantis
Copy link
Contributor

@pmur I'm not seeing go-ppc64le-p9 on farmer.

@laboger
Copy link
Contributor

laboger commented Oct 4, 2021

I 'm not seeing go-ppc64le-p9 on farmer.

We had opened a ticket with OSU and Paul asked them to delete it. It is gone now and it looks like all is working as expected.
The builders now match what is documented in x/build/env/linux-ppc64le/osuosl/NOTES.

@cagedmantis
Copy link
Contributor

Thank you!

@dmitshur dmitshur added this to In Progress in Go Release Team Oct 5, 2021
@dmitshur
Copy link
Contributor

dmitshur commented Oct 5, 2021

Closing since I understand this is all resolved, thanks again!

@dmitshur dmitshur closed this as completed Oct 5, 2021
Go Release Team automation moved this from In Progress to Done Oct 5, 2021
@golang golang locked and limited conversation to collaborators Oct 5, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Release
Archived in project
Go Release Team
Done
Milestone
Unreleased
Development

No branches or pull requests
6 participants
@cagedmantis @mknyszek @dmitshur @gopherbot @laboger @pmur