Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/relui: re-deploying relui doesn't forget everything that happened #47401

Closed
toothrot opened this issue Jul 26, 2021 · 24 comments
Closed

x/build/cmd/relui: re-deploying relui doesn't forget everything that happened #47401

toothrot opened this issue Jul 26, 2021 · 24 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@toothrot
Copy link
Contributor

cmd/relui needs a persistence layer. Let's say postgres! Or select a viable tool that works well in:

  • development
  • trybots
  • production
  • migrations

A local file on disk may not be enough.
@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Jul 26, 2021
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jul 26, 2021
@gopherbot gopherbot added this to the Unreleased milestone Jul 26, 2021
@toothrot toothrot mentioned this issue Jul 26, 2021
Open
3 tasks
@toothrot toothrot self-assigned this Jul 27, 2021
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/340429 mentions this issue: cmd/relui: use go:embed for templates and content

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/340430 mentions this issue: cmd/relui, cmd/relworker: remove pubsub and datastore

gopherbot pushed a commit to golang/build that referenced this issue Aug 11, 2021
@toothrot
cmd/relui: use go:embed for templates and content
f740ced 
This changes relui to use embed from go1.16 for managing content. This
drastically simplifies the deployment steps, in preparation for
containerizing.

For golang/go#47401

Change-Id: I1eb9f6f63fa490ef73ed454ddecd1fd99db4f960
Reviewed-on: https://go-review.googlesource.com/c/build/+/340429
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Aug 11, 2021
@toothrot
cmd/relui, cmd/relworker: remove pubsub and datastore

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode
6865319 
This will make room for a new workflow definition, and a new datastore.

For golang/go#47401

Change-Id: I4bdc8a9a0ed2c5a3743284d0f267b838dddec8aa
Reviewed-on: https://go-review.googlesource.com/c/build/+/340430
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/347292 mentions this issue: cmd/relui: move implementation to internal/relui

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/347289 mentions this issue: cmd/relui: add Docker and Postgres configuration

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/347291 mentions this issue: cmd/relui: remove original implementation

gopherbot pushed a commit to golang/build that referenced this issue Sep 2, 2021
@toothrot
cmd/relui: add Docker and Postgres configuration
3cef35c 
This adds a Dockerfile for relui, as well as a file for running
integration tests against the database. It also adds a configuration for
a development database, as well as a test environment.

For golang/go#47401

Change-Id: I668c99d028265c865baa7e15dfe627423815af94
Reviewed-on: https://go-review.googlesource.com/c/build/+/347289
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Sep 2, 2021
@toothrot
cmd/relui: remove original implementation
02a8684 
This is in preparation for a move to using internal/workflow.

For golang/go#47401

Change-Id: Ie58224fff4fb73d006f2ce56f17329193140a5e6
Reviewed-on: https://go-review.googlesource.com/c/build/+/347291
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Sep 2, 2021
@toothrot
cmd/relui: move implementation to internal/relui
3734795 
This makes it explicit that the code is not intended to be used outside
this module.

For golang/go#47401

Change-Id: I6c7ca7402b451dbcfac39e504d5fc767811c973d
Reviewed-on: https://go-review.googlesource.com/c/build/+/347292
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/349951 mentions this issue: cmd/relui,internal/relui: add datbase migrations

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/350449 mentions this issue: cmd/relui,internal/relui: create and run workflows

gopherbot pushed a commit to golang/build that referenced this issue Sep 17, 2021
@toothrot
cmd/relui,internal/relui: add database migrations
d026839 
This change adds golang-migrate to manage database migrations for relui.
relui will run all database migrations on start, and return an error for
any failures.

The current developer tooling is managed through the Makefile. For ease
of use, common tasks will be moved to a script in a future CL.

For golang/go#47401

Change-Id: I678c3b21d7e89d7d5a66305f66075f02cc2f3284
Reviewed-on: https://go-review.googlesource.com/c/build/+/349951
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/352569 mentions this issue: cmd/relui,internal/relui: cleanup go:build comments

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/352169 mentions this issue: internal/relui: create workflows package

gopherbot pushed a commit to golang/build that referenced this issue Sep 28, 2021
@toothrot
cmd/relui,internal/relui: create and run workflows
cbb5112 
Create and run workflows, storing state in a postgres database. Generate
database queries using sqlc, which allows the author to have complete
control over the query structure.

The basic structure of creating and running a workflow works, but
there is still much work to do in terms of selecting which workflow to
create, recording logs, maintaining a list of currently running
workflows, and recording workflow-level results.

Task persistence needs to be updated, as the current implementation
cannot be resumed due to losing the type during marshalling.

For golang/go#47401

Change-Id: I9ccddf117d023f70568de655c482820a1152d9cb
Reviewed-on: https://go-review.googlesource.com/c/build/+/350449
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Sep 28, 2021
@toothrot
internal/relui: create workflows from parameters
88204a4 
This change updates the UI and creation endpoint to use the
parameters specified for the workflow.

For golang/go#47401

Change-Id: I46623a17161806c39cad98853e996f0fee5a0305
Reviewed-on: https://go-review.googlesource.com/c/build/+/352169
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Sep 28, 2021
@toothrot
cmd/relui,internal/relui: cleanup go:build comments
4c1ebef 
These are no longer necessary as embed is in all supported verions of
Go.

For golang/go#47401

Change-Id: I3b128a28028d0ca15088d71202984c8b8ef4469c
Reviewed-on: https://go-review.googlesource.com/c/build/+/352569
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/353170 mentions this issue: internal/relui: persist logs to database

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/353229 mentions this issue: cmd/relui: document database migrations

gopherbot pushed a commit to golang/build that referenced this issue Sep 30, 2021
@toothrot
cmd/relui: document database migrations
7959cfa 
Add some missing documentation about creating and running database
migrations.

For golang/go#47401

Change-Id: I42ca1af43b7097c7913cb3c1ae40c70425ae8977
Reviewed-on: https://go-review.googlesource.com/c/build/+/353229
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/353349 mentions this issue: internal/relui: format task results on home

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/353409 mentions this issue: cmd/relui: add migrate-down-up flag

gopherbot pushed a commit to golang/build that referenced this issue Sep 30, 2021
@toothrot
internal/relui: persist logs to database
71603fa 
Save log output from workflow tasks in the database. Render logs in the
UI alongside each task. This will almost certainly be a performance
problem if we have many logs until we implement pagination.

For golang/go#47401

Change-Id: I4ddc3d7845e4559cc636b3b7972e0adefe9fcec4
Reviewed-on: https://go-review.googlesource.com/c/build/+/353170
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Oct 1, 2021
@toothrot
internal/relui: format task results on home
79bf688 
Render the workflow task results as a table using a css grid layout,
rather than a table, to allow for us to expand rows to show logs and
eventually other results without using javascript.

For golang/go#47401

Change-Id: I1ab1bb3a1b87d2e864b769eeb232336af98a306d
Reviewed-on: https://go-review.googlesource.com/c/build/+/353349
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/353553 mentions this issue: cmd/relui,internal/relui: resume workflows

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/353552 mentions this issue: cmd/relui: use migrations for sqlc schema

gopherbot pushed a commit to golang/build that referenced this issue Oct 4, 2021
@toothrot
cmd/relui: use migrations for sqlc schema
1409edd 
Use migrations instead of maintaining a schema.sql. Having a schema dump
can be useful, but we can instead go with our migrations to avoid
confusion.

For golang/go#47401

Change-Id: I426ffd46e17d3aa381d5bc0d517c8529961702f0
Reviewed-on: https://go-review.googlesource.com/c/build/+/353552
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Oct 20, 2021
@toothrot
cmd/relui,internal/relui: resume workflows
19d8baf 
Resume all unfinished workflows on relui startup. Moves most workflow
running logic to a new Worker type.

This also adds a couple new columns to the workflow to record output and
whether a workflow has completed. In order to actually resume workflows,
we'll have to either add functionality to unmark them as "finished"
(restarting manually, which seems reasonable), or not marking a workflow
finished if it terminates with context.Cancelled.

For golang/go#47401

Change-Id: I3e0ed021d7a47fb125f1034df83dc3c6d95887f8
Reviewed-on: https://go-review.googlesource.com/c/build/+/353553
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363535 mentions this issue: cmd/relui: add deployment configuration

gopherbot pushed a commit to golang/build that referenced this issue Nov 12, 2021
@toothrot
cmd/relui: add deployment configuration
a9ac038 
Add deployment configuration for relui using IAM database
authentication.

For golang/go#47401

Change-Id: Ifde113bd3e09d19eb69bffb5a0ce3689e6c6410a
Reviewed-on: https://go-review.googlesource.com/c/build/+/363535
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363695 mentions this issue: cmd/relui: correct database configuration

gopherbot pushed a commit to golang/build that referenced this issue Nov 12, 2021
@toothrot
cmd/relui: correct database configuration
99c01e6 
Add missing IAM flag for cloud_sql_proxy, and use the correct user and
database in relui environment.

For golang/go#47401

Change-Id: I2fa760c00f7000552d87069f259750fa455cce92
Reviewed-on: https://go-review.googlesource.com/c/build/+/363695
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363696 mentions this issue: deploy: add /releases to build.golang.org

gopherbot pushed a commit to golang/build that referenced this issue Nov 12, 2021
@toothrot
deploy: add /releases to build.golang.org
95ddf45 
This mounts relui to the appropriate locaiton.

For golang/go#47401

Change-Id: I247ac925b47206acc2a6b4a109da003645af3a68
Reviewed-on: https://go-review.googlesource.com/c/build/+/363696
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363697 mentions this issue: deploy: fix precedent for build.golang.org/releases

gopherbot pushed a commit to golang/build that referenced this issue Nov 15, 2021
@toothrot
deploy: fix precedent for build.golang.org/releases
0a59650 
Order matters for path matching.

For golang/go#47401

Change-Id: I92f5fb716b1d893313a7d2883f8de63cb8278156
Reviewed-on: https://go-review.googlesource.com/c/build/+/363697
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363975 mentions this issue: cmd/relui,internal/relui: serve from sub-paths

gopherbot pushed a commit to golang/build that referenced this issue Nov 15, 2021
@toothrot
cmd/relui,internal/relui: serve from sub-paths
9076251 
This change allows relui to correctly serve from a path, like
build.golang.org/releases. It adds a base-url flag which is used to
prefix all paths referenced in the application.

For golang/go#47401

Change-Id: Ib8f6fe429591ceabfaf0f419e5258a677b375ff8
Reviewed-on: https://go-review.googlesource.com/c/build/+/363975
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363977 mentions this issue: internal/relui: correct baseURL behavior

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363976 mentions this issue: internal/relui: default to postgres as maintdb

gopherbot pushed a commit to golang/build that referenced this issue Nov 15, 2021
@toothrot
internal/relui: default to postgres as maintdb
4d86312 
In non-development environments, the default will be a database that is
the same as the username, which is not likely to exist.

For golang/go#47401

Change-Id: I2e4664f7917d1a63417cc1a19f800a043dff9d43
Reviewed-on: https://go-review.googlesource.com/c/build/+/363976
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Nov 15, 2021
@toothrot
internal/relui: correct baseURL behavior
ecf31e9 
Nested http.ServeMux should be configured with the appropriate
trailing slash, but we should not use the trailing slash when using
http.StripPrefix. This simplifies ServeHTTP slightly, and corrects
behavior when handling nested mux.

For golang/go#47401

Change-Id: I2e6ab792e210a500025b8b07a28e3db67696397d
Reviewed-on: https://go-review.googlesource.com/c/build/+/363977
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363979 mentions this issue: deploy: add wildcard for releases endpoints

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/363978 mentions this issue: internal/relui/templates: add missing baseLink

gopherbot pushed a commit to golang/build that referenced this issue Nov 16, 2021
@toothrot
internal/relui/templates: add missing baseLink
7318d32 
This fixes creation in a production environment.

For golang/go#47401

Change-Id: I9aa83e77b98c512c6efe75a30ecbb745aad5a1f4
Reviewed-on: https://go-review.googlesource.com/c/build/+/363978
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Nov 16, 2021
@toothrot
deploy: add wildcard for releases endpoints
7b9db79 
Otherwise, the strict match will not match endpoints other than the
home, which redirects.

For golang/go#47401

Change-Id: I1317e6c097374a40be69481a5b36934bb15e5b81
Reviewed-on: https://go-review.googlesource.com/c/build/+/363979
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@heschi heschi moved this to Done in Go Release Sep 27, 2022
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Oct 14, 2022
@golang golang locked and limited conversation to collaborators Oct 14, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@toothrot @dmitshur @gopherbot