Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/relui: tag golang.org/x repos continuously #48523

Closed
toothrot opened this issue Sep 21, 2021 · 41 comments
Closed

x/build/cmd/relui: tag golang.org/x repos continuously #48523

toothrot opened this issue Sep 21, 2021 · 41 comments
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@toothrot
Copy link
Contributor

Plan: once a month (or whatever time unit)
For each x repo in dependency order (lowest first)
go get update all the deps to their latest tags
Wait for that to be green
Tag it with next version

Can probably shoehorn into existing process, maybe not running automatically at first.
0.1.0, 0.2.0, 0.3.0...

@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Sep 21, 2021
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Sep 21, 2021
@gopherbot gopherbot added this to the Unreleased milestone Sep 21, 2021
@toothrot toothrot added the Soon This needs to be done soon. (regressions, serious bugs, outages) label Oct 5, 2021
@dmitshur dmitshur added this to Planned in Go Release Team Oct 5, 2021
@dmitshur dmitshur changed the title x/build/updatedstd: tag imported versions of repos x/build: tag imported versions of repos Nov 9, 2021
@dmitshur
Copy link
Contributor

dmitshur commented Feb 8, 2022

Wait for that to be green

Automating this process will be blocked on coming up with a machine-understandable criteria of what "green" means.

@gopherbot
Copy link

Change https://go.dev/cl/423036 mentions this issue: internal/workflow: clarify ownership

@gopherbot
Copy link

Change https://go.dev/cl/423037 mentions this issue: internal/workflow: add expansions

gopherbot pushed a commit to golang/build that referenced this issue Aug 11, 2022
For golang/go#48523.

Change-Id: Ie55cdf3c90687d065fb1b81d0a9f97a81d6ffd1f
Reviewed-on: https://go-review.googlesource.com/c/build/+/423036
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
@heschi heschi moved this from Planned to In Progress in Go Release Team Aug 12, 2022
@heschi heschi self-assigned this Aug 12, 2022
@gopherbot
Copy link

Change https://go.dev/cl/425088 mentions this issue: internal/task: start x repo tagging tasks

@gopherbot
Copy link

Change https://go.dev/cl/425089 mentions this issue: internal/task: continue x repo tagging

@gopherbot
Copy link

Change https://go.dev/cl/425090 mentions this issue: internal/relui,internal/task: move useful structs to task

@gopherbot
Copy link

Change https://go.dev/cl/425091 mentions this issue: internal/task: finish initial implementation of x/ repo tagging

gopherbot pushed a commit to golang/build that referenced this issue Aug 25, 2022
One of the upcoming use cases for relui, tagging x/ repos, has a
dependency graph we can't know without doing significant work. This is
in conflict with one of the workflow package's design principles, that
workflows be defined ahead of time. So here's one idea for bending that
rule.

Introduce the concept of an expansion task. An expansion task runs the
same as a normal task, but in addition to its inputs it receives (a copy
of) the workflow definition and can add tasks and outputs to it as
usual. Once the task finishes running, the workflow swaps in the new
definition and continues. Other tasks can run concurrently.

One significant difference is that the expansion task must run every
time the workflow is started or resumed, so they really should be pure
functions with no network access or side effects. To encourage this,
they don't receive a Context.

Introducing this required some refactoring of the existing Run loop; we
need to allow these apparently-unused tasks to run, and deal with new
tasks popping up mid-run. But overall I think it's not too huge a
distortion to the design.

For golang/go#48523.

Change-Id: I92c164883e06474fa951abba5fa18fd78eacb8ce
Reviewed-on: https://go-review.googlesource.com/c/build/+/423037
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/425495 mentions this issue: internal/task: integration test for x repo tagging

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Sep 14, 2022
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
First step: list all our projects/repositories, and select those that
have a go.mod labeling them as golang.org/x as candidates to release as
the next version, or v0.1.0 for untagged repositories.

For golang/go#48523.

Change-Id: Ice92319a0726daf3bf5f94581582d8802640dffc
Reviewed-on: https://go-review.googlesource.com/c/build/+/425088
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
Next step: expand the workflow with tagging tasks, currently all no-ops.

For golang/go#48523.

Change-Id: I48cb8804dc1db6c17b748b54a194e9ef64ef6bce
Reviewed-on: https://go-review.googlesource.com/c/build/+/425089
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
For whatever reason I've decided to write the tagging workflow in the
task package, so I need the test helpers here. I also need the website
structs to find the latest Go release later.

For golang/go#48523.

Change-Id: I92a1430bd27f0aa90c9fccefa12fb057e7e7f864
Reviewed-on: https://go-review.googlesource.com/c/build/+/425090
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/431356 mentions this issue: maintner/maintnerd/maintapi: validate page size in mixed mode

@gopherbot
Copy link

Change https://go.dev/cl/431355 mentions this issue: cmd/coordinator/internal/legacydash: uncap maintner request

gopherbot pushed a commit to golang/build that referenced this issue Sep 16, 2022
We've been seeing the coordinator fail to schedule subrepo runs on
release branches. As far as I can tell, the problem is the request to
maintner: branch = "mixed" means to get at least one commit from each
branch in the repository, but we're asking for 30 commits when we have
50-some branches. So obviously some are going to be left out. That
absence breaks findWork when it tries to map commits to branches.

I don't fully understand which branches happen to be included and which
don't, but this behavior seems clearly wrong. Let maintner decide how
many responses to send back, currently hardcoded at 500.

For golang/go#48523.

Change-Id: I1792e89de96e83ceed6a430a7456b119f633869b
Reviewed-on: https://go-review.googlesource.com/c/build/+/431355
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 20, 2022
This one got away from me a little, sorry.

Add cycle detection and breaking: we need to break some cycles involving
x/tools. We could consider fixing them by creating nested modules but I
don't want to deal with that now.

Implement UpdateGoMod: fetch the latest Go release, download it to a
buildlet with network access, and run go get to update to the previously
tagged versions. But only do this if there are dependencies, both to
save time and because go get throws an error if you try to get nothing.

Implement Tag: if there hasn't been a change since the last tag, reuse
that one. Otherwise, tag it with the selected new verison.

For golang/go#48523.

Change-Id: I29c05eb639fd748af1e0e356596adc8c3f15ccf0
Reviewed-on: https://go-review.googlesource.com/c/build/+/425091
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/mod that referenced this issue Oct 19, 2022
The tagging process is blocked with the following error:

    golang.org/x/mod/zip tested by
    	golang.org/x/mod/zip.test imports
    	golang.org/x/tools/txtar loaded from golang.org/x/tools@v0.0.0-20191119224855-298f0cb1881e,
    	but go 1.16 would select v0.1.12

Resolve the error by upgrading tools.

For golang/go#48523.

Change-Id: I37d6673c2880f1170152237a0a4807eb54b95c15
Reviewed-on: https://go-review.googlesource.com/c/mod/+/443936
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Oct 19, 2022
go.mod generation is inherently racy with people submitting other CLs,
so I wasn't worried about getting the base commit outside of the update
process. But I didn't think about the fact that it might take a few
days. I also didn't consider what would happen if the updates broke for
some reason, and now they have:

    golang.org/x/mod/zip tested by
    	golang.org/x/mod/zip.test imports
    	golang.org/x/tools/txtar loaded from golang.org/x/tools@v0.0.0-20191119224855-298f0cb1881e,
    	but go 1.16 would select v0.1.12

CL 443936 will fix that error, but relui needs to use that commit as the
basis for updating go.mod, not the one it chose on Monday.

I'll redesign the workflow in the next CL -- I can't change it while
it's running.

For golang/go#48523.

Change-Id: Ibf9c74e465a2dccbe831fb80f2904481c51d74c0
Reviewed-on: https://go-review.googlesource.com/c/build/+/444115
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/444117 mentions this issue: internal/task: remove initial tagging support

gopherbot pushed a commit to golang/build that referenced this issue Oct 19, 2022
With the first run done this code is now dead. We can create manual tags
for future bootstrapping.

For golang/go#48523.

Change-Id: Iff147d095205f3e3687136d85eaaa08bae207069
Reviewed-on: https://go-review.googlesource.com/c/build/+/444117
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/443857 mentions this issue: internal/task: tidy gopls while tagging tools

@gopherbot
Copy link

Change https://go.dev/cl/444295 mentions this issue: go.mod: mark as requiring -compat 1.13

@gopherbot
Copy link

Change https://go.dev/cl/444156 mentions this issue: cmd/go/internal/modload: update TestQueryImport to pass with tagged versions of x/net

gopherbot pushed a commit that referenced this issue Oct 20, 2022
…ersions of x/net

For #48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/444435 mentions this issue: [release-branch.go1.19] cmd/go/internal/modload: update TestQueryImport to pass with tagged versions of x/net

@gopherbot
Copy link

Change https://go.dev/cl/444436 mentions this issue: [release-branch.go1.18] cmd/go/internal/modload: update TestQueryImport to pass with tagged versions of x/net

gopherbot pushed a commit that referenced this issue Oct 20, 2022
…rt to pass with tagged versions of x/net

For #48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
(cherry picked from commit 3e6ca3a)
Reviewed-on: https://go-review.googlesource.com/c/go/+/444435
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit that referenced this issue Oct 20, 2022
…rt to pass with tagged versions of x/net

For #48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
(cherry picked from commit 3e6ca3a)
Reviewed-on: https://go-review.googlesource.com/c/go/+/444436
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/tools that referenced this issue Oct 20, 2022
When the tagging process runs go mod tidy it needs to pass -compat 1.16.
This magic comment will cause it to do so. See CL 443857.

For golang/go#48523.

Change-Id: I6824a2c78462604b9d4737444b4d16d4c21ecc99
Reviewed-on: https://go-review.googlesource.com/c/tools/+/444295
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Oct 20, 2022
Clean up after the change to read the branch head later. Also fix a bug:
in the version tasks, we want to know exactly what commit we're tagging,
so we pass in the expected commit, and AwaitCL returns that if no CL was
necessary. But here we want to tag the latest branch head if there's no
CL necessary. So we need a different Await function.

For golang/go#48523.

Change-Id: I97f92b22d4f29edeb52d06c311fddaa6596da0c2
Reviewed-on: https://go-review.googlesource.com/c/build/+/444116
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
@dmitshur dmitshur changed the title x/build: tag imported versions of repos x/build/cmd/relui: tag golang.org/x repos continuously Oct 20, 2022
gopherbot pushed a commit to golang/build that referenced this issue Oct 20, 2022
gopls has a replace directive that uses the local tools version, which
means that they have to be updated together.

This is currently a one-off, so rather than trying to figure it out
automatically tools is hardcoded. I'm not proud of it. But at least
there's a test! (?)

For golang/go#48523.

Change-Id: If59e669ba3ede08967b72ba3bb36b2c66e1a03a7
Reviewed-on: https://go-review.googlesource.com/c/build/+/443857
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
@heschi
Copy link
Contributor

heschi commented Oct 20, 2022

I'm gonna call this done. We haven't run it fully end-to-end yet because of the hiccups with x/tools, but it got very close. One possible followup is to have it mail CLs to specified reviewers, since it turns out almost none of our repositories have owners any more, but I'll decide about that next time we run it.

@gopherbot
Copy link

Change https://go.dev/cl/445955 mentions this issue: internal/task: add support for tagging a single repository

gopherbot pushed a commit to golang/build that referenced this issue Oct 31, 2022
To make one-off tagging less error-prone, run a miniature version of the
overall workflow to tag a specific repository.

Also includes a tiny bug fix for expansions with parameters.

For golang/go#48523.

Change-Id: Ic9089966aa28d20c716169b41121e2da21faf54a
Reviewed-on: https://go-review.googlesource.com/c/build/+/445955
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
romaindoumenc pushed a commit to TroutSoftware/go that referenced this issue Nov 3, 2022
…ersions of x/net

For golang#48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/449038 mentions this issue: internal/task: final touches on x/ repo tagging

gopherbot pushed a commit to golang/build that referenced this issue Nov 9, 2022
Tagging has now run to completion and I'm comfortable turning it loose.
Remove the approval requirement and send CLs to specified reviewers so
that it can be run as a scheduled workflow.

For golang/go#48523.

Change-Id: I2411a2680b09f877b8f4e5905ed51ae2e9ef2bdd
Reviewed-on: https://go-review.googlesource.com/c/build/+/449038
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
andrew-d pushed a commit to tailscale/go that referenced this issue Dec 7, 2022
…rt to pass with tagged versions of x/net

For golang#48523.

Change-Id: Ied35d15462cbae1002e1db1e6e119a6c9f8323da
Reviewed-on: https://go-review.googlesource.com/c/go/+/444156
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
(cherry picked from commit 3e6ca3a)
Reviewed-on: https://go-review.googlesource.com/c/go/+/444435
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/501835 mentions this issue: go.mod: delete repeated "indirect"

gopherbot pushed a commit to golang/text that referenced this issue Jun 8, 2023
It got added by 'go mod tidy' in CL 471399
because the '; ' separator was missing.

For golang/go#48523.
For golang/go#58737.

Change-Id: I63642a4f49beb9679f9c5bbb4e2aa8ed0cc73c9e
Reviewed-on: https://go-review.googlesource.com/c/text/+/501835
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Go Release Team
In Progress
Development

No branches or pull requests

5 participants