Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/relui: add ability to schedule workflows #54476

Closed
toothrot opened this issue Aug 16, 2022 · 11 comments
Closed

x/build/cmd/relui: add ability to schedule workflows #54476

toothrot opened this issue Aug 16, 2022 · 11 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release Team
Milestone
Unreleased

Comments

@toothrot
Copy link
Contributor

There are at least several different use-cases where we'd like to be able to schedule workflows in advance. Relui should be able to keep track of these and start workflows as appropriate.

I've written out each use-case below, but a crontab time format is probably sufficient.

Once

Will run the job at the specified date and time.

This is useful for scheduling major and minor releases in advance.

k v
Run at 2022-08-16 10:49

Interval

Runs a job every interval as specified.

This would handle nightly releases, or other automated tasks that we do regularly, like tagging repos or updating dependencies.

k v
Interval Duration Minute, hour, day, week, month
Interval Quantity 1,2,3...
Run at 2022-08-16 10:49

Date-based

Runs a job every nth day of week or month.

This may be useful for updating buildlet images and binaries.

k v
Interval Duration Minute, hour, day, week, month
Run at 2022-08-16 10:49
@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Aug 16, 2022
@toothrot toothrot added this to the Unreleased milestone Aug 16, 2022
@toothrot toothrot self-assigned this Aug 16, 2022
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 16, 2022
@joedian joedian added this to In Progress in Go Release Team Aug 16, 2022
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Aug 16, 2022
@gopherbot
Copy link

Change https://go.dev/cl/424002 mentions this issue: internal/agenda: create periodic task library

@gopherbot
Copy link

Change https://go.dev/cl/429695 mentions this issue: internal/relui: add ability to schedule workflows

@gopherbot
Copy link

Change https://go.dev/cl/432397 mentions this issue: internal/relui: create cron-based workflow schedules

@gopherbot
Copy link

Change https://go.dev/cl/432404 mentions this issue: internal/relui: show schedules on workflow lists

@gopherbot
Copy link

Change https://go.dev/cl/432405 mentions this issue: internal/relui: add delete schedules button

gopherbot pushed a commit to golang/build that referenced this issue Sep 22, 2022
@toothrot @gopherbot
internal/relui: add ability to schedule workflows
6a129b0 
This adds a new field on workflow creation to schedule the creation of a
workflow. Only immediate running and a single future-date schedule are
supported, but the vast majority of the implementation for duration
(every N minutes) and cron-based schedules is included.

Schedules are resumed from the database on start-up.

Future CLs will include support for the remaining schedule types, a UI
for viewing and deleting schedules, archiving defunct schedules, and
notifications for scheduled workflow failures.

For golang/go#54476

Change-Id: Ica6a9df85087657aeabd85dae7973487d648dd59
Reviewed-on: https://go-review.googlesource.com/c/build/+/429695
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Jenny Rakoczy <jenny@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 22, 2022
@toothrot @gopherbot
internal/relui: create cron-based workflow schedules
15ca61f 
Adds functionality to schedule Workflows using a standard cron-spec.
Updates resume to skip RunOnce schedules that are in the past.

For golang/go#54476

Change-Id: I30f5bb45353a88ca9398b8787ee7ea44ae96d7e1
Reviewed-on: https://go-review.googlesource.com/c/build/+/432397
Auto-Submit: Jenny Rakoczy <jenny@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 22, 2022
@toothrot @gopherbot
internal/relui: show schedules on workflow lists
1143a32 
Add a small table to show scheduled workflows, including the last and
next time they will run.

For golang/go#54476

Change-Id: I82a7e34f58ff77eb59df330f52564aff6b6c581f
Reviewed-on: https://go-review.googlesource.com/c/build/+/432404
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Jenny Rakoczy <jenny@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Sep 22, 2022
@toothrot @gopherbot
internal/relui: add delete schedules button
6b41ff9 
Add a button to permanently delete a workflow schedule.

For golang/go#54476

Change-Id: I14d5ba66e7ce609dfea97af9ecd2aa6173d03524
Reviewed-on: https://go-review.googlesource.com/c/build/+/432405
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Jenny Rakoczy <jenny@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/436155 mentions this issue: internal/relui: remove approval step from dry-run

gopherbot pushed a commit to golang/build that referenced this issue Sep 28, 2022
@toothrot @gopherbot
internal/relui: remove approval step from dry-run
9294235 
This removes the approval step from dry-run releases. There's no need to
approve test failures in this scenario, and we should just fail the
workflow.

For golang/go#54476

Change-Id: I468a0bda42b4d5b6a57d7da79c448d76383c648f
Reviewed-on: https://go-review.googlesource.com/c/build/+/436155
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
Auto-Submit: Jenny Rakoczy <jenny@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/444695 mentions this issue: internal/relui: email on schedule failure

gopherbot pushed a commit to golang/build that referenced this issue Oct 27, 2022
@toothrot
internal/relui: email on schedule failure
67d6243 
The user who scheduled a workflow may not be closely watching at the
time of failure. This adds functionality to send an email if a scheduled
workflow cannot progress.

The current implementation logs on stall, but it's trivial to
configure to email as some tasks do.

For golang/go#54476

Change-Id: Id0deefd3c1b07f569585600a583ba4e04f8f7be1
Reviewed-on: https://go-review.googlesource.com/c/build/+/444695
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/451296 mentions this issue: internal/relui: add nightly release workflow

@toothrot
Copy link
Contributor Author

toothrot commented Jan 3, 2023

Remaining work:

  • No workflows currently fail. If a workflow has been scheduled, it should notify when it stalls. The code to do this exists, but has not been tested, due to workflows with approval steps never failing.

@toothrot toothrot removed their assignment Jan 3, 2023
@heschi
Copy link
Contributor

heschi commented May 8, 2023

We saw a workflow failure over the weekend that properly logged at https://cs.opensource.google/go/x/build/+/master:internal/relui/listener.go;l=183;drc=67d624370c7018c64d7dcb4a4ee2ffcf5b029a07. So it should be fine to hook it up to a real email client.

@gopherbot
Copy link

Change https://go.dev/cl/493620 mentions this issue: cmd/relui: send emails on scheduled workflow failures

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Go Release Team
In Progress
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@toothrot @dmitshur @gopherbot @heschi