Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/watchflakes: start running it as a service in GCP #65353

Closed
dmitshur opened this issue Jan 29, 2024 · 7 comments
Closed

x/build/cmd/watchflakes: start running it as a service in GCP #65353

dmitshur opened this issue Jan 29, 2024 · 7 comments
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@dmitshur
Copy link
Contributor

CL 557115 adds an initial version of watchflakes that talks with LUCI as its source of builds and test results. As part of its initial development, it's currently being run as a command line tool and iterated upon by @cherrymui, but it has generally proven to be working by now and is ready for its next steps.

This is a tracking issue to deploy it as a service in the x/build GCP project, similarly to how gopherbot and some other services are deployed and run there. This will happen some time after the command-line version is checked in to x/build and proves to not need many more changes.

CC @cherrymui.

@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jan 29, 2024
@dmitshur dmitshur added this to the Unreleased milestone Jan 29, 2024
@dmitshur dmitshur self-assigned this Jan 29, 2024
@gopherbot
Copy link

Change https://go.dev/cl/579318 mentions this issue: cmd/watchflakes: replace a sleep with a ticker

@gopherbot
Copy link

Change https://go.dev/cl/579316 mentions this issue: cmd/watchflakes: fetch GitHub token from Secret Manager

@gopherbot
Copy link

Change https://go.dev/cl/579315 mentions this issue: cmd/watchflakes: add deployment files

@gopherbot
Copy link

Change https://go.dev/cl/579319 mentions this issue: cmd/watchflakes: start posting

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 16, 2024
@gopherbot
Copy link

Change https://go.dev/cl/579320 mentions this issue: cmd/watchflakes: return errors from LUCIClient to caller

@gopherbot
Copy link

Change https://go.dev/cl/579299 mentions this issue: cmd/watchflakes: ignore failed builds without output properties

@gopherbot
Copy link

Change https://go.dev/cl/580016 mentions this issue: cmd/watchflakes: use github.NewClient API

gopherbot pushed a commit to golang/build that referenced this issue Apr 19, 2024
Add a fairly minimal set of files to deploy watchflakes as a service.
Use Cloud Build for building Docker images, so that it's possible to
perform deploys without having Docker as a local dependency.

Also add a README that points to documentation on how to deploy.

For golang/go#65353.

Change-Id: I355a23d02b9a31f99e34b8ab1a1681ec3b9a36a6
Reviewed-on: https://go-review.googlesource.com/c/build/+/579315
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
gopherbot pushed a commit to golang/build that referenced this issue Apr 19, 2024
Unfortunately, the only API that creates a github.Client requires that
credentials for api.github.com are available in $HOME/.netrc. It would
be nicer to be able to pass a custom *http.Client. For now, modify the
global default HTTP client (http.DefaultClient) that the rsc.io/github
package always uses and provide authentication to it directly that way,
without needing to write to a $HOME/.netrc file. This can go away when
the rsc.io/github API provides a cleaner way of doing this.

For golang/go#65353.

Change-Id: Ie2ec3e957f7b2dc48b85ce5475a09d19df71622d
Reviewed-on: https://go-review.googlesource.com/c/build/+/579316
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
gopherbot pushed a commit to golang/build that referenced this issue Apr 19, 2024
Make it run equally often, no matter how long a single iteration takes.

For golang/go#65353.

Change-Id: I6c2814c1b91e4eef65e09d858b984290a3d5585f
Reviewed-on: https://go-review.googlesource.com/c/build/+/579318
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 19, 2024
Make some progress towards it being possible to handle unexpected errors
by logging them, waiting a bit, and trying again, instead of log.Fatalln
whenever anything unexpected happens, now that watchflakes is becoming a
continuously running service.

Also use some higher level APIs for limited concurrency, and various doc
improvements.

For golang/go#65353.

Change-Id: I7af83a8f368ff41d176d80a475698a04ca407325
Reviewed-on: https://go-review.googlesource.com/c/build/+/579320
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Apr 19, 2024
It turns out to be possible¹ for a build to fail early enough that it
doesn't get a chance to produce output properties, and not get marked
as an infrastructure failure (even though it should've been). There's
not much choice but to ignore such builds.

¹ For example, https://ci.chromium.org/b/8752293940218446529. It failed
  very early while running bbagent, before golangbuild was even started.

For golang/go#65353.

Change-Id: I14782f41e7a390380a32272b58cda385c91cb881
Reviewed-on: https://go-review.googlesource.com/c/build/+/579299
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Apr 19, 2024
Thanks to Russ for adding it.

For golang/go#65353.

Change-Id: Ie8aa589ade3e076c6a0c1dcf6776dc85a8a3aec2
Reviewed-on: https://go-review.googlesource.com/c/build/+/580016
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Status: Done
Development

No branches or pull requests

2 participants