Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: perf builders are failing with "infra failed" #68517

Closed
cherrymui opened this issue Jul 18, 2024 · 6 comments
Closed

x/build: perf builders are failing with "infra failed" #68517

cherrymui opened this issue Jul 18, 2024 · 6 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@cherrymui
Copy link
Member

cherrymui commented Jul 18, 2024

Both gotip-linux-amd64_c2s16-perf_vs_release and gotip-linux-amd64_c3h88-perf_vs_release are failing with "infra failed"

Error in build protocol: Expected a terminal build status, got STATUS_UNSPECIFIED, while top level status is STARTED.

E.g. https://ci.chromium.org/ui/p/golang/builders/ci/gotip-linux-amd64_c2s16-perf_vs_release/b8742123808151649825/overview

The failure has persisted for 5 commits.

cc @golang/release

@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jul 18, 2024
@gopherbot gopherbot added this to the Unreleased milestone Jul 18, 2024
@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 18, 2024
@cherrymui
Copy link
Member Author

cc @mknyszek

@gabyhelp
Copy link

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@dmitshur
Copy link
Contributor

dmitshur commented Jul 18, 2024

In the last successful build from 20 hours ago, the "7. upload benchmark results" step took 6.9 seconds. In the failing builds since then, it's failing after 7-9 minutes and the "results" log appears very large (e.g., around 1.4 GB).

@mknyszek
Copy link
Contributor

I know the root cause and I have a CL out. I'll mark that CL with this issue. In short, yeah, the log is just huge because CockroachDB spews a ton of data out and a recent x/benchmarks CL dumped that data by default.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 18, 2024
@dmitshur dmitshur moved this to In Progress in Go Release Jul 18, 2024
@mknyszek
Copy link
Contributor

The CL is https://go.dev/cl/599455 if anyone wants to give it a quick review.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/599455 mentions this issue: sweet: collect and dump only hijacked fd 2 for cockroachdb

@github-project-automation github-project-automation bot moved this from In Progress to Done in Go Release Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Development

No branches or pull requests

5 participants