Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: perf builders are failing to build cockroachdb for baseline at tip #68555

Closed
dmitshur opened this issue Jul 23, 2024 · 8 comments
Closed
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) FixPending Issues that have a fix which has not yet been reviewed or submitted. NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Jul 23, 2024

The gotip-linux-amd64_{c2s16,c3h88}-perf_vs_release builders are failing at tip due sweet running into an error building cockroachdb for baseline:

[sweet] error: build cockroachdb for baseline: exit status 1. stderr:
# github.com/cockroachdb/cockroach/pkg/util/goschedstats
../../util/goschedstats/runtime_go1.23.go:32:6: puintptr redeclared in this block
	../../util/goschedstats/runtime_go1.20_21_22.go:24:6: other declaration of puintptr
../../util/goschedstats/runtime_go1.23.go:34:6: muintptr redeclared in this block
	../../util/goschedstats/runtime_go1.20_21_22.go:26:6: other declaration of muintptr
../../util/goschedstats/runtime_go1.23.go:36:6: guintptr redeclared in this block
[...]
[sweet] error: failed to execute benchmarks, see log for details
2024/07/23 02:47:40 Error running sweet: error running sweet run: exit status 1
2024/07/23 02:47:40 FAIL
exit status 1

For example, see https://logs.chromium.org/logs/golang/buildbucket/cr-buildbucket/8741671757414064545/+/u/step/20/log/2.

There was an untested_go_version mechanism added a month ago in cockroachdb/cockroach@e513328 which may be relevant, since gotip represents the early development cycle of Go 1.24 now. Edit: CL 600196 already took that into account yesterday. (CC @aclements.)

CC @mknyszek, @dr2chase.

@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 23, 2024
@dmitshur dmitshur added this to the Unreleased milestone Jul 23, 2024
@gabyhelp
Copy link

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@mknyszek
Copy link
Contributor

This should be fixed by https://go.dev/cl/600196.

@mknyszek mknyszek reopened this Jul 23, 2024
@mknyszek
Copy link
Contributor

mknyszek commented Jul 23, 2024

No, sorry, it's still broken as of that CL, and I see now @dmitshur noticed that above.

@dmitshur dmitshur moved this to In Progress in Go Release Jul 23, 2024
@mknyszek
Copy link
Contributor

Ugh, I can't seem to reproduce the failure locally... This is making me think there's something specifically about the environment of the builder that's causing problems, or perhaps how cmd/bench is invoking sweet.

@dmitshur
Copy link
Contributor Author

dmitshur commented Jul 23, 2024

@mknyszek Could the problem be that the baseline toolchain is something like 1.22 so the constraints in runtime_go1.20_21_22.go match, but because sweet now sets the untested_go_version tag, the constraint in runtime_go1.23.go also matches? This might be off, I looked only very briefly.

@aclements
Copy link
Member

I think you're right. Ideally, we'd fix this in CockroachDB upstream, but alternatively Sweet could be more careful to only set that build tag if it's actually on an unreleased version. I think the alternative is somewhat unstable in the long run, but it would work in the short run.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/600475 mentions this issue: sweet: apply flags to cockroachdb 'go build' based on 'go version'

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/600495 mentions this issue: main.star: enable longtest builders on x/benchmarks for sweet

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. FixPending Issues that have a fix which has not yet been reviewed or submitted. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 23, 2024
gopherbot pushed a commit to golang/build that referenced this issue Jul 23, 2024
It's already annoying to have to ask for longtest builders to run the
end-to-end test for sweet, but the fact that we also forget to run the
Go 1.22 end-to-end test means we get real breakages like
golang/go#68555.

Resolve this once and for all by enabling longtest in presubmit if any
Go file is touched by a CL in the x/benchmarks repository.

Change-Id: Ia76258edf9bc93b88501cd64755beddd39186f06
Reviewed-on: https://go-review.googlesource.com/c/build/+/600495
Auto-Submit: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@github-project-automation github-project-automation bot moved this from In Progress to Done in Go Release Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) FixPending Issues that have a fix which has not yet been reviewed or submitted. NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Development

No branches or pull requests

5 participants