Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: Kubernetes-ify all misc build services & bots #18817

Closed
bradfitz opened this issue Jan 27, 2017 · 23 comments
Closed

x/build: Kubernetes-ify all misc build services & bots #18817

bradfitz opened this issue Jan 27, 2017 · 23 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone

Comments

@bradfitz
Copy link
Contributor

The Go build system & project bots are all ran in different ways & places.

This is a tracking bug to unify them to all run consistently on Kubernetes on GKE in our existing build project.

First CL was https://golang.org/cl/35810 to write a Dockerfile for the coordinator.

/cc @kevinburke @quentinmit @rakyll

@bradfitz bradfitz added the Builders x/build issues (builders, bots, dashboards) label Jan 27, 2017
@bradfitz bradfitz added this to the Unreleased milestone Jan 27, 2017
@bradfitz bradfitz self-assigned this Jan 27, 2017
@gopherbot
Copy link

CL https://golang.org/cl/35911 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Jan 30, 2017
…te service

Currently the Gerrit polling & git mirroring & serving & syncing
functionality is in the "watcher" process, which is in the same binary
as the cmd/coordinator (farmer.golang.org), but runs as a separate
process on that machine, and actually run in a separate Docker
container so it has a cache volume and the /usr/bin/git binary, etc. A
flag determines where func main delegates to: the build coordinator,
or the git mirror.

Start cleaning this up, in prep to run on Kubernetes as a separate service.

This copies the env/watcher-world/Dockerfile into a new gitmirror
Dockerfile, and copies the existing cmd/coordinator/watcher_process.go
into a new cmd/gitmirror/gitmirror.go process.

None of this is deployed or working yet. Once it's running and happy,
I'll then switch the coordinator to find & use it on GKE, and then
delete the old code.

Updates golang/go#18817

Change-Id: Ibaa87f244f08ce61662e689b815740c72c997536
Reviewed-on: https://go-review.googlesource.com/35911
Reviewed-by: Jaana Burcu Dogan <jbd@google.com>
@gopherbot
Copy link

CL https://golang.org/cl/36016 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Jan 31, 2017
Updates golang/go#18817

Change-Id: Ifee53384486b0692899b77be2eaa42ca9006ef8e
Reviewed-on: https://go-review.googlesource.com/36016
Reviewed-by: Chris Broadfoot <cbro@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/36473 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Feb 7, 2017
Updates golang/go#18817

Change-Id: Iff09e580d720ab1dfc8de0da2c45340467a335db
Reviewed-on: https://go-review.googlesource.com/36473
Reviewed-by: Kevin Burke <kev@inburke.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/36692 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Feb 9, 2017
And moves Dial methods to *kubernetes.Client, instead of a separate
Dialer type.

Updates golang/go#18817

Change-Id: I045ac48441b9139cb0e01ceb4969b29cf5e72507
Reviewed-on: https://go-review.googlesource.com/36692
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/36801 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Feb 11, 2017
It's now running, but not hooked up the coordinator yet.

Updates golang/go#18817

Change-Id: I5870af1e0bfe5213886f7faeb138127986a7234c
Reviewed-on: https://go-review.googlesource.com/36801
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/36807 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Feb 11, 2017
Updates golang/go#18817

Change-Id: I1142f079f3748fefcf9fd1d5749e58db86afbda6
Reviewed-on: https://go-review.googlesource.com/36807
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@siennathesane
Copy link

What else is left to do for this?

@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 6, 2017

So far gitmirror (internal) and https://pubsubhelper.golang.org/ are on GKE.
So is tip.golang.org.
TODO: beta.golang.org
TODO: finish maintner, move it to GKE
TODO: finish gopherbot, move it to GKE, subscribing to maintner
TODO: move the coordinator from CoreOS to GKE

@siennathesane
Copy link

Where is beta.golang.org?

@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 6, 2017

A GCE VM running Debian. Overkill.

@siennathesane
Copy link

Gotcha, where does the code exist and is there an existing issue for it? I can work on migrating it SoonTM.

@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 6, 2017

I created #19867 with the details for what beta.golang.org needs.

@jessfraz
Copy link
Contributor

jessfraz commented Apr 12, 2017

I can help with converting the builder dockerfiles away from the VM stuff! I just realized this is the goal and I loveeeeee small Dockerfiles!

@gopherbot
Copy link

CL https://golang.org/cl/40570 mentions this issue.

@gopherbot
Copy link

CL https://golang.org/cl/40571 mentions this issue.

@gopherbot
Copy link

CL https://golang.org/cl/40572 mentions this issue.

@gopherbot
Copy link

CL https://golang.org/cl/40574 mentions this issue.

kevinburke pushed a commit to kevinburke/build that referenced this issue Apr 16, 2017
Before:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/gitmirror     1.32GB

After:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/gitmirror     208MB

Updates golang/go#18817

Change-Id: I633fbbde2c5bba52535a8cfefdcb1f0532c45c06
gopherbot pushed a commit to golang/build that referenced this issue Apr 17, 2017
Before:
REPOSITORY                             SIZE
gcr.io/go-dashboard-dev/pubsubhelper   834MB

After:
REPOSITORY                             SIZE
gcr.io/go-dashboard-dev/pubsubhelper   12.3MB

Updates golang/go#18817

Change-Id: Ibf1f33a04043c0defddd7fe6b8d0455092044ac0
Reviewed-on: https://go-review.googlesource.com/40574
Reviewed-by: Kevin Burke <kev@inburke.com>
gopherbot pushed a commit to golang/build that referenced this issue Apr 17, 2017
Before:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/coordinator   895MB

After:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/coordinator   20.5MB

Updates golang/go#18817

Change-Id: I144923fcd85aab5f1aaaa3f012143bac45f5086b
Reviewed-on: https://go-review.googlesource.com/40570
Reviewed-by: Kevin Burke <kev@inburke.com>
gopherbot pushed a commit to golang/build that referenced this issue Apr 17, 2017
Before:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/gitmirror     1.32GB

After:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/gitmirror     208MB

Updates golang/go#18817

Change-Id: I633fbbde2c5bba52535a8cfefdcb1f0532c45c06
Reviewed-on: https://go-review.googlesource.com/40571
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Kevin Burke <kev@inburke.com>
gopherbot pushed a commit to golang/build that referenced this issue Apr 20, 2017
Before:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/maintnerd     786MB

After:
REPOSITORY                            SIZE
gcr.io/go-dashboard-dev/maintnerd     8.85MB

Updates golang/go#18817
Updates golang/go#19866

Change-Id: Ibfdadbaf043e44b26d5807c1da45746e934a7dce
Reviewed-on: https://go-review.googlesource.com/40572
Reviewed-by: Kevin Burke <kev@inburke.com>
@gopherbot
Copy link

CL https://golang.org/cl/42251 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 1, 2017
Updates golang/go#18817

Change-Id: I8947eaf02d9b4ea49db00879bef8d1c6cb54184a
Reviewed-on: https://go-review.googlesource.com/42251
Reviewed-by: Quentin Smith <quentin@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/42231 mentions this issue.

kevinburke pushed a commit to kevinburke/build that referenced this issue May 2, 2017
Updates golang/go#18817

Change-Id: I8947eaf02d9b4ea49db00879bef8d1c6cb54184a
Reviewed-on: https://go-review.googlesource.com/42251
Reviewed-by: Quentin Smith <quentin@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue May 2, 2017
- buildongce is taught how to create the go cluster
- config files for Docker services and pods
- Dockerfile updated to fix caching

Updates golang/go#18817

Change-Id: If25d41723996b92069117f4d75488202b63310ef
Reviewed-on: https://go-review.googlesource.com/42231
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/42492 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 3, 2017
…GCE VM

coordinator only runs on GKE now.

Updates golang/go#18817

Change-Id: I1866c767224fecabf17789cf95794f6761636ce3
Reviewed-on: https://go-review.googlesource.com/42492
Reviewed-by: Quentin Smith <quentin@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/42550 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 4, 2017
Updates golang/go#18817

Change-Id: I8d919d409ff3d59145b4410db35a5653c9eeed28
Reviewed-on: https://go-review.googlesource.com/42550
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/42850 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue May 5, 2017
- Make URLs point to correct external IP
- Disable windows-amd64-2008 builder type that doesn't exist in
  the staging farm

Updates golang/go#18817

Change-Id: Id64a63694f90e70c4fd78f9d1433ed5031822111
Reviewed-on: https://go-review.googlesource.com/42850
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor Author

With the coordinator now on GKE, I think we're all done.

@golang golang locked and limited conversation to collaborators May 12, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Development

No branches or pull requests

4 participants