Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: reverse builder config uniformity #21260

Closed
s-mang opened this issue Aug 1, 2017 · 16 comments
Closed

x/build: reverse builder config uniformity #21260

s-mang opened this issue Aug 1, 2017 · 16 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone
Unreleased

Comments

@s-mang
Copy link
Contributor

s-mang commented Aug 1, 2017

Onboarding overhead is very high ATM for builder project.

I'd like to discuss a config uniformity overhaul for the reverse buildlets.
We have been migrating various reverse buildlet Dockerfiles / configurations over to various different stages of "new" piecemeal b/c lack of headcount. eg. uses X version of rundockerbuildlet, uses rundockerbuildlet, are builds hermetic?, uses --reverse (legacy) vs. --reverse-type (latest) flag to cmd/buildlet/buildlet.go

I'd like us document the ideal/latest blessed reverse buildlet config, and push to bring all reverse buildlet configurations up to this config scheme.
I'd also like to discuss the potential for a plan to keep reverse buildlet configurations at latest config scheme in the future.
@gopherbot gopherbot added this to the Unreleased milestone Aug 1, 2017
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 1, 2017
@s-mang s-mang self-assigned this Aug 1, 2017
@bradfitz
Copy link
Contributor

bradfitz commented Aug 2, 2017

I will start referencing bugs from here.

Here is why the Solaris/illumos builders are not ideal: #15581

@gopherbot
Copy link

Change https://golang.org/cl/52633 mentions this issue: cmd/coordinator: clarify in logs which reverse clients are using legacy config

@gopherbot
Copy link

Change https://golang.org/cl/52650 mentions this issue: cmd/buildlet/stage0: stop using deprecated --reverse flag to buildlet

@bradfitz
Copy link
Contributor

bradfitz commented Aug 2, 2017

I deployed CL 52633 to see which hosts are using the old --reverse= argument to buildlet instead of --reverse-type= and found:

      1 host type host-darwin-10_8 (mapped from legacy modes ["darwin-amd64-10_8"])
      1 host type host-linux-mipsle (mapped from legacy modes ["linux-mipsle"])
      1 host type host-linux-mips (mapped from legacy modes ["linux-mips"])
      1 host type host-linux-s390x (mapped from legacy modes ["linux-s390x-ibm"])
      2 host type host-darwin-10_12 (mapped from legacy modes ["darwin-amd64-10_12"])
      2 host type host-solaris-amd64 (mapped from legacy modes ["solaris-amd64-smartosbuildlet"])
      3 host type host-darwin-10_10 (mapped from legacy modes ["darwin-amd64-10_10"])
      3 host type host-linux-arm5spacemonkey (mapped from legacy modes ["linux-arm-arm5spacemonkey"])
      5 host type host-linux-ppc64le-osu (mapped from legacy modes ["linux-ppc64le-buildlet"])
      5 host type host-linux-ppc64-osu (mapped from legacy modes ["linux-ppc64-buildlet"])
     14 host type host-darwin-10_11 (mapped from legacy modes ["darwin-amd64-10_11"])
     50 host type host-linux-arm-scaleway (mapped from legacy modes ["host-linux-arm-scaleway"])

Updating the Scaleway images can happen in https://go-review.googlesource.com/c/52293

A bunch of the others are done in https://go-review.googlesource.com/c/52650 (once ppc64 and arm5-spacemonkey are updated)

Updating Mac images is super hard and tedious and SLOW and error prone. I think we'd be better of keeping the mapping in the coordinator and buildlet for a bit. We can hide it from flag listing if helpful to minimize doc clutter.

MIPS seems to be updateable easily enough.

I also want to figure out a plan for #21191 (speaking of cleanup).

gopherbot pushed a commit to golang/build that referenced this issue Aug 2, 2017
@bradfitz
cmd/coordinator: clarify in logs which reverse clients are using lega…
31150ba 
…cy config

We want to kill off the --reverse vs --reverse-host code. This will
point out any clients still using the old way. There might not be any.

Updates golang/go#21260

Change-Id: Ic4b283b02891655ee43f2b7db76088df3f12065a
Reviewed-on: https://go-review.googlesource.com/52633
Reviewed-by: Sarah Adams <shadams@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Aug 2, 2017
@bradfitz
cmd/buildlet/stage0: stop using deprecated --reverse flag to buildlet
d29668b 
Updates golang/go#21260

Change-Id: Ic4d96ef3984017944c5a9af087ea7010e5193f32
Reviewed-on: https://go-review.googlesource.com/52650
Reviewed-by: Jessica Frazelle <jessfraz@google.com>
@bradfitz
Copy link
Contributor

Latest update on which hosts are still using the legacy ways:

  • host type host-darwin-10_10 (mapped from legacy modes ["darwin-amd64-10_10"])
  • host type host-darwin-10_11 (mapped from legacy modes ["darwin-amd64-10_11"])
  • host type host-darwin-10_12 (mapped from legacy modes ["darwin-amd64-10_12"])
  • host type host-linux-arm-scaleway (mapped from legacy modes ["host-linux-arm-scaleway"])
  • host type host-linux-ppc64-osu (mapped from legacy modes ["linux-ppc64-buildlet"])
  • host type host-linux-s390x (mapped from legacy modes ["linux-s390x-ibm"])
  • host type host-solaris-amd64 (mapped from legacy modes ["solaris-amd64-smartosbuildlet"])

Basically no change since Aug 1 2017, except that we no longer OS X 10.8 and the MIPS builders disappeared.

The one fix seems to be that host-linux-ppc64le-osu and host-linux-ppc64-osu are no longer doing the legacy registration.

Also, macOS 10.14 is at least using the new way. And the Mac VMware cluster is easier to work with now since it was upgraded, so we could fix those old versions at least.

@bradfitz
Copy link
Contributor

Two of those builders are gone now, so remaining is:

  • host type host-darwin-10_11 (mapped from legacy modes ["darwin-amd64-10_11"])
  • host type host-darwin-10_12 (mapped from legacy modes ["darwin-amd64-10_12"])
  • host type host-linux-arm-scaleway (mapped from legacy modes ["host-linux-arm-scaleway"])
  • host type host-linux-ppc64-osu (mapped from legacy modes ["linux-ppc64-buildlet"])
  • host type host-linux-s390x (mapped from legacy modes ["linux-s390x-ibm"])

@bradfitz bradfitz mentioned this issue Oct 10, 2019
Closed
@gopherbot
Copy link

Change https://golang.org/cl/200757 mentions this issue: env/linux-ppc64le/osuosl: update NOTES

gopherbot pushed a commit to golang/build that referenced this issue Oct 11, 2019
@bradfitz
env/linux-ppc64le/osuosl: update NOTES
85c334c 
One of the little ppc64le VMs was lost in the upgrade, and we got one
new beefy one.

Updates golang/go#34830
Updates golang/go#21189
Updates golang/go#21260

Change-Id: I4488d106c5c146c6e55d2d3528913f5507b9d57b
Reviewed-on: https://go-review.googlesource.com/c/build/+/200757
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/200820 mentions this issue: env/linux-ppc64{,le}/osuosl: update NOTES

gopherbot pushed a commit to golang/build that referenced this issue Oct 15, 2019
@bradfitz
env/linux-ppc64{,le}/osuosl: update NOTES
9c8585c 
Add details of 3 new big VMs: POWER8 (le), POWER9 (le), and POWER8 (be).

Still not configured, but we have them provisioned.

Updates golang/go#34830
Updates golang/go#21189
Updates golang/go#21260

Change-Id: If648e9966485062cda8131caf2ac5f87ab839fc5
Reviewed-on: https://go-review.googlesource.com/c/build/+/200820
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/201637 mentions this issue: all: update linux-ppc64le builders

gopherbot pushed a commit to golang/build that referenced this issue Oct 17, 2019
@bradfitz
all: update linux-ppc64le builders
c53d4ba 
Fixes golang/go#21189
Updates golang/go#21260
Updates golang/go#34830 (ppc64 big endian remains)

Change-Id: I7ff38edb519bf4f34cd05bb795b6a41f72b10640
Reviewed-on: https://go-review.googlesource.com/c/build/+/201637
Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/203886 mentions this issue: env/linux-ppc64/osuosl: upgrade linux/ppc64 hardware & OS, use Docker

gopherbot pushed a commit to golang/build that referenced this issue Nov 4, 2019
@bradfitz
env/linux-ppc64/osuosl: upgrade linux/ppc64 hardware & OS, use Docker
b1a7aa2 
Collaboration with @tiborvass at Docker who got Docker running on
big-endian PPC64. Go for ppc64 doesn't support cgo or external
linking, so runc doesn't work, but a new OCI-compliant runc
implementation written in C (https://github.com/containers/crun) means
we can run Docker after all. See NOTES & build-*.sh

Then add a Dockerfile & associated cleanup in buildlet & stage0 to use
rundockerbuildlet.

Once done, might help with golang/go#35188, golang/go#32613, etc.

Fixes golang/go#34830
Updates golang/go#21260

Change-Id: I43d7afa1d58bbdfa16e3c57670bc41f1d1932d80
Reviewed-on: https://go-review.googlesource.com/c/build/+/203886
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor

bradfitz commented Nov 4, 2019

Down to:

  • host type host-darwin-10_11 (mapped from legacy modes ["darwin-amd64-10_11"])
  • host type host-darwin-10_12 (mapped from legacy modes ["darwin-amd64-10_12"])
  • host type host-linux-arm-scaleway (mapped from legacy modes ["host-linux-arm-scaleway"])
  • host type host-linux-s390x (mapped from legacy modes ["linux-s390x-ibm"])

@bradfitz bradfitz mentioned this issue Nov 5, 2019
Closed
@bradfitz
Copy link
Contributor

bradfitz commented Nov 5, 2019

Just double checked, and yes:

$ kubectl logs $(k get -o name po -l app=coordinator) | grep "mapped from legacy" | perl -ne 'm/host type (\S+)/ and print "$1\n"' | sort | uniq -c
     29 host-darwin-10_11
    215 host-darwin-10_12
    327 host-linux-arm-scaleway 
    214 host-linux-s390x

@bradfitz
Copy link
Contributor

bradfitz commented Nov 5, 2019

And I just fixed the darwin ones.

@gopherbot
Copy link

Change https://golang.org/cl/205357 mentions this issue: cmd/buildlet, cmd/makemac: stop using legacy buildlet --reverse flag for Macs

gopherbot pushed a commit to golang/build that referenced this issue Nov 5, 2019
@bradfitz
cmd/buildlet, cmd/makemac: stop using legacy buildlet --reverse flag …
74c30b5 
…for Macs

Updates golang/go#21260

Change-Id: I4e2c61117cfebffd20ab4acd3ee736172c95d153
Reviewed-on: https://go-review.googlesource.com/c/build/+/205357
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/205603 mentions this issue: all: upgrade scaleway linux-arm builders, stop using deprecated --reverse flag

@bradfitz bradfitz assigned bradfitz and unassigned s-mang Nov 6, 2019
@gopherbot
Copy link

Change https://golang.org/cl/205608 mentions this issue: cmd/buildlet, cmd/coordinator: delete old --reverse mode

gopherbot pushed a commit to golang/build that referenced this issue Nov 6, 2019
@bradfitz
all: upgrade scaleway linux-arm builders, stop using deprecated --rev…
d489097 
…erse flag

Updates golang/go#21260 (no more buildlets using the --reverse flag)
Updates golang/go#33574 (linux-arm kernel+userspace updated)

Change-Id: I7455f6fa3e851f1f9f81d6f1eb487ef7e4bea55b
Reviewed-on: https://go-review.googlesource.com/c/build/+/205603
Reviewed-by: Bryan C. Mills <bcmills@google.com>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@bradfitz @codebien
env/linux-ppc64le/osuosl: update NOTES
0e75cfb 
One of the little ppc64le VMs was lost in the upgrade, and we got one
new beefy one.

Updates golang/go#34830
Updates golang/go#21189
Updates golang/go#21260

Change-Id: I4488d106c5c146c6e55d2d3528913f5507b9d57b
Reviewed-on: https://go-review.googlesource.com/c/build/+/200757
Reviewed-by: Bryan C. Mills <bcmills@google.com>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@bradfitz @codebien
env/linux-ppc64{,le}/osuosl: update NOTES
cde585d 
Add details of 3 new big VMs: POWER8 (le), POWER9 (le), and POWER8 (be).

Still not configured, but we have them provisioned.

Updates golang/go#34830
Updates golang/go#21189
Updates golang/go#21260

Change-Id: If648e9966485062cda8131caf2ac5f87ab839fc5
Reviewed-on: https://go-review.googlesource.com/c/build/+/200820
Reviewed-by: Bryan C. Mills <bcmills@google.com>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@bradfitz @codebien
all: update linux-ppc64le builders
3000e42 
Fixes golang/go#21189
Updates golang/go#21260
Updates golang/go#34830 (ppc64 big endian remains)

Change-Id: I7ff38edb519bf4f34cd05bb795b6a41f72b10640
Reviewed-on: https://go-review.googlesource.com/c/build/+/201637
Reviewed-by: Carlos Eduardo Seo <cseo@linux.vnet.ibm.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@bradfitz @codebien
env/linux-ppc64/osuosl: upgrade linux/ppc64 hardware & OS, use Docker
62fee1d 
Collaboration with @tiborvass at Docker who got Docker running on
big-endian PPC64. Go for ppc64 doesn't support cgo or external
linking, so runc doesn't work, but a new OCI-compliant runc
implementation written in C (https://github.com/containers/crun) means
we can run Docker after all. See NOTES & build-*.sh

Then add a Dockerfile & associated cleanup in buildlet & stage0 to use
rundockerbuildlet.

Once done, might help with golang/go#35188, golang/go#32613, etc.

Fixes golang/go#34830
Updates golang/go#21260

Change-Id: I43d7afa1d58bbdfa16e3c57670bc41f1d1932d80
Reviewed-on: https://go-review.googlesource.com/c/build/+/203886
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@bradfitz @codebien
cmd/buildlet, cmd/makemac: stop using legacy buildlet --reverse flag …
b521a2a 
…for Macs

Updates golang/go#21260

Change-Id: I4e2c61117cfebffd20ab4acd3ee736172c95d153
Reviewed-on: https://go-review.googlesource.com/c/build/+/205357
Reviewed-by: Bryan C. Mills <bcmills@google.com>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@bradfitz @codebien
all: upgrade scaleway linux-arm builders, stop using deprecated --rev…
41d0125 
…erse flag

Updates golang/go#21260 (no more buildlets using the --reverse flag)
Updates golang/go#33574 (linux-arm kernel+userspace updated)

Change-Id: I7455f6fa3e851f1f9f81d6f1eb487ef7e4bea55b
Reviewed-on: https://go-review.googlesource.com/c/build/+/205603
Reviewed-by: Bryan C. Mills <bcmills@google.com>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@bradfitz @codebien
cmd/buildlet, cmd/coordinator: delete old --reverse mode
0a823b5 
All the buildlets have been updated to use --reverse-type with a host
type instead of a builder type.

Fixes golang/go#21260

Change-Id: I1264261f099c3686fe01455949486f523b94c6de
Reviewed-on: https://go-review.googlesource.com/c/build/+/205608
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@golang golang locked and limited conversation to collaborators Nov 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @s-mang @gopherbot