Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/relui: deal with hung tasks #54134

Closed
heschi opened this issue Jul 29, 2022 · 5 comments
Closed

x/build/cmd/relui: deal with hung tasks #54134

heschi opened this issue Jul 29, 2022 · 5 comments
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@heschi
Copy link
Contributor

heschi commented Jul 29, 2022

I've seen tasks get hung a few times now, and it's really annoying to deal with. Decide on a way to prevent it from happening again.

cc @golang/release

@heschi heschi added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Jul 29, 2022
@heschi heschi added this to the Unreleased milestone Jul 29, 2022
@heschi heschi self-assigned this Jul 29, 2022
@gopherbot
Copy link

Change https://go.dev/cl/420221 mentions this issue: internal/workflow: set a watchdog on tasks by default

@gopherbot
Copy link

Change https://go.dev/cl/420541 mentions this issue: internal/task: log buildlet output in chunks

@gopherbot
Copy link

Change https://go.dev/cl/420540 mentions this issue: internal/task,internal/relui: log every minute while tasks wait

@gopherbot
Copy link

Change https://go.dev/cl/420542 mentions this issue: internal/task/releaselet: add logging to MSI building

@dmitshur dmitshur added this to In Progress in Go Release Team Aug 2, 2022
gopherbot pushed a commit to golang/build that referenced this issue Aug 2, 2022
Building the MSIs can take quite a while. Add some logging to keep the
watchdog happy.

For golang/go#54134.

Change-Id: I6178323f6e34a6ecd409370e9e396f683a6cdb9f
Reviewed-on: https://go-review.googlesource.com/c/build/+/420542
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Aug 4, 2022
In preparation for adding a watchdog timer, use a common helper function
that can reset the watchdog automatically.

For golang/go#54134.

Change-Id: I257a97cfeb540e90b0c005e255c40ee1d054bae0
Reviewed-on: https://go-review.googlesource.com/c/build/+/420540
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Aug 4, 2022
Rather than buffering buildlet output and only showing it in case of
error, stream it to a user-supplied Writer. In relui, buffer 10
seconds of output, or up to 1MB, then dump everything up to the most
recent newline to the logger.

For golang/go#54134.

Change-Id: Ie0eb9af3aa6c3455afd03b358722924e38058063
Reviewed-on: https://go-review.googlesource.com/c/build/+/420541
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Go Release Team automation moved this from In Progress to Done Aug 4, 2022
@gopherbot
Copy link

Change https://go.dev/cl/422597 mentions this issue: internal/task: always log through the TaskContext

gopherbot pushed a commit to golang/build that referenced this issue Aug 10, 2022
Buildlet tasks weren't resetting the watchdog timer because they
bypassed the TaskContext's Printf method. Ideally the Logger would be
unexported to prevent this mistake, but we construct TaskContext's in
tests all over the place. For now, I got rid of all the existing reads
and we can worry about it later if it comes up again.

For golang/go#54134.

Change-Id: I43c38cabd9bff298c72affaa1792b0ac0ad2cb5c
Reviewed-on: https://go-review.googlesource.com/c/build/+/422597
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
@golang golang locked and limited conversation to collaborators Aug 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Development

No branches or pull requests

2 participants