The failing task (#58227) didn't just fail with an error, but panicked, and that caused all of relui to restart. If relui restarting causes a workflow to get marked as completed, maybe that's unavoidable, or maybe we can do better.

A particular task can avoid taking down the entire workflow by adding a defer recover that converts any unexpected panics into internal errors. (Hopefully this shouldn't be needed very often thanks to problems being caught by tests, but perhaps can be useful sometimes when a new task is being rolled out for the first time.)

I see what happened—the sequence of events here was unfortunate:

1. a "Go 1.20 final" workflow was running and almost finished
2. relui restarted and tried to resume all pending workflows
3. during workflow registration, since a go1.20 tag was already published, a workflow named "Go 1.21 final" was registered, and a workflow named "Go 1.20 final" was not

I guess we should take this possible sequence of events into account in our logic for dynamic release workflow registration. 😅

Oops. A bit janky, but perhaps we should GetTag on the latest tag and drop back to the previous one if it's less than, say, 3 days old?

Making the workflows more idempotent (if possible) would help prevent this from happening in the future.

Change https://go.dev/cl/466145 mentions this issue: internal/relui: don't forget how to do major releases too fast