Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: add NetBSD 9.0 builder #37244

Closed
bradfitz opened this issue Feb 15, 2020 · 18 comments
Closed

x/build: add NetBSD 9.0 builder #37244

bradfitz opened this issue Feb 15, 2020 · 18 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. new-builder OS-NetBSD
Milestone

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Feb 15, 2020

NetBSD 9.0 is out:
https://netbsd.org/releases/formal-9/NetBSD-9.0.html

Tracking bug for adding a builder.

/cc @bsiegert

@bradfitz bradfitz added OS-NetBSD Builders x/build issues (builders, bots, dashboards) new-builder labels Feb 15, 2020
@gopherbot gopherbot added this to the Unreleased milestone Feb 15, 2020
@bsiegert
Copy link
Contributor

bsiegert commented Feb 16, 2020 via email

@toothrot
Copy link
Contributor

I think we should replace the 8.0 builder with 9.0. NetBSD is not a first-class port, so I don't think we should expand our coverage to older versions unless there is some compelling reason.

@toothrot toothrot added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 18, 2020
@bradfitz
Copy link
Contributor Author

NetBSD is not a first-class port, so I don't think we should expand our coverage to older versions unless there is some compelling reason.

Works for me.

@gopherbot
Copy link

Change https://golang.org/cl/226977 mentions this issue: dashboard, env/netbsd-{amd64,386}: update to NetBSD 9.0

@dmitshur
Copy link
Contributor

dmitshur commented Apr 2, 2020

(Our current netbsd-amd64-8_0 builder is running NetBSD 8.0RC1 with an image built 2 years ago in CL 109035. CL 185217 updated the env script to NetBSD 8.1, but a new image was not built. The netbsd-386-8_0 builder is disabled due to #31726.)

I agree with @toothrot that there isn't a need for 8.0 once 9.0 is working. However, I think it would be easier to add a new 9.0 builder, letting it run for some time to make sure it works better than 8.0, then remove the 8.0 builder.

We should also increase the disk size while building a new image. Its current one is very small (4 GB) and causes issues like #30839, and inability to install large dependencies needed for some tests. /cc @randall77

@dmitshur
Copy link
Contributor

dmitshur commented Apr 2, 2020

I've investigated what it would take to get a 9.0 image built in CL 226977. The image was built and boots up, buildlet runs, but it seems something went wrong with installing CA certificates on it. It fails when attempting to fetch an HTTPS URL:

golang.org/x/build $ go run ./cmd/debugnewvm -run-build=netbsd-amd64-8_0 -override-image=netbsd-amd64-9-0-2019q4
2020/04/02 12:09:00 Creating debug-temp-1585829340 (with VM image "netbsd-amd64-9-0-2019q4")
2020/04/02 12:09:02 instance requested
2020/04/02 12:09:50 instance created
2020/04/02 12:09:50 serial output error: googleapi: got HTTP response code 404 with body: Not Found
2020/04/02 12:09:50 got instance info; running in https://www.googleapis.com/compute/v1/projects/symbolic-datum-552/zones/us-central1-b
2020/04/02 12:09:50 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:09:55 client buildlet probe error: Get "http://10.240.0.2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/04/02 12:09:56 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:10:01 client buildlet probe error: Get "http://10.240.0.2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/04/02 12:10:02 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:10:07 client buildlet probe error: Get "http://10.240.0.2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/04/02 12:10:08 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:10:08 client buildlet probe error: Get "http://10.240.0.2": dial tcp 10.240.0.2:80: connect: connection refused
2020/04/02 12:10:09 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:10:09 buildlet probe: 200 OK
2020/04/02 12:10:09 WorkDir: /tmp/workdir, <nil>
2020/04/02 12:10:09 Pushing 'go1.4' Go bootstrap dir ...
2020/04/02 12:10:09 Putting Go bootstrap: 500 Internal Server Error; body: fetching URL https://storage.googleapis.com/go-builder-data/gobootstrap-netbsd-amd64-2da6b33.tar.gz: Get https://storage.googleapis.com/go-builder-data/gobootstrap-netbsd-amd64-2da6b33.tar.gz: x509: certificate signed by unknown authority
exit status 1

In contrast, using the current netbsd-amd64-8-0-2018q1 image results in:

golang.org/x/build $ go run ./cmd/debugnewvm -run-build=netbsd-amd64-8_0
...
2020/04/02 12:14:31 About to hit http://10.240.0.5 to see if buildlet is up yet...
2020/04/02 12:14:31 buildlet probe: 200 OK
2020/04/02 12:14:31 WorkDir: /tmp/workdir, <nil>
2020/04/02 12:14:31 Pushing 'go1.4' Go bootstrap dir ...
2020/04/02 12:14:34 Pushing 'go' dir...
2020/04/02 12:14:42 Running src/all.bash ...
...

@bsiegert Do you have ideas what would cause that? Are you able to help with investigating this? Thanks.

@iamleot
Copy link

iamleot commented Apr 2, 2020 via email

@dmitshur
Copy link
Contributor

dmitshur commented Apr 2, 2020

@iamleot Note that the mozilla-rootcerts package is already being installed, and that was sufficient in NetBSD 8.0. See here and here. It seems likely something about the way it's being installed has changed and requires adjustment in the script.

But thanks for the suggestion, I can explore how the mozilla-rootcerts-openssl package differs.

@bsiegert
Copy link
Contributor

bsiegert commented Apr 2, 2020

You might need to add mozilla-rootcerts-openssl in addition to mozilla-rootcerts. I admit that I don't know why the latter alone is not enough.

@iamleot
Copy link

iamleot commented Apr 2, 2020 via email

@dmitshur
Copy link
Contributor

dmitshur commented Apr 2, 2020

@iamleot Thanks for providing context. However, note that the mozilla-rootcerts install script has been invoked manually on this line:

/usr/pkg/sbin/mozilla-rootcerts install

If adding the mozilla-rootcerts-openssl package fixes this problem, it's likely because that invocation above has stopped working. It seems more brittle to invoke that command via an absolute path than to install a package, so I'd be okay with removing the manual mozilla-rootcerts install invocation if it turns out to obsoleted by adding the mozilla-rootcerts-openssl package.

@dmitshur
Copy link
Contributor

dmitshur commented Apr 2, 2020

Adding the mozilla-rootcerts-openssl package has resolved the problem. It continued to work after I removed these two commands:

touch /etc/openssl/openssl.cnf
/usr/pkg/sbin/mozilla-rootcerts install

I'll go with that approach in the CL. Thanks for the suggestion.

@randall77
Copy link
Contributor

FYI, I can allocate a netbsd 9.0 builder, but I can't do anything with it.

$ gomote create netbsd-amd64-9_0
# still creating netbsd-amd64-9_0 after 5s; 0 requests ahead of you
# still creating netbsd-amd64-9_0 after 11s; 0 requests ahead of you
# still creating netbsd-amd64-9_0 after 15s; 0 requests ahead of you
# still creating netbsd-amd64-9_0 after 20s; 0 requests ahead of you
# still creating netbsd-amd64-9_0 after 26s; 0 requests ahead of you
user-khr-netbsd-amd64-9_0-0
$ gomote ssh user-khr-netbsd-amd64-9_0-0
2020/04/07 08:56:42 Builder type "netbsd-amd64-9_0" not known to this gomote binary. Update your gomote binary. TODO: teach gomote to fetch build configs from the server (Issue 30929)
$ go get -u golang.org/x/build/cmd/gomote
$ gomote ssh user-khr-netbsd-amd64-9_0-0
2020/04/07 09:03:07 Builder type "netbsd-amd64-9_0" not known to this gomote binary. Update your gomote binary. TODO: teach gomote to fetch build configs from the server (Issue 30929)

None of the other commands I tried (ls, push) work either.

@randall77
Copy link
Contributor

In fact, I can't even destroy the one I just created.

@dmitshur
Copy link
Contributor

dmitshur commented Apr 7, 2020

@randall77 The NetBSD 9.0 builder is still in development, so you'll need to re-build gomote with CL 226977 applied to be able to use it at this time.

I'll update this thread when the new builder is operational.

gopherbot pushed a commit to golang/build that referenced this issue Apr 7, 2020
Update NetBSD amd64 and 386 builders to the latest NetBSD 9.0 release,
using the latest versions of anita (2.3) and pkgsrc packages (2019Q4).

Also increase the image disk size from 4 GB to 16 GB. The motivation
is to help tests that require more disk space and large dependencies.
Additional adjustments to the image building script will need to be
made to make use of increased disk space for the /tmp volume, as it
is currently being mounted in RAM with size equal to 25% of memory.

Remove the manual invocation of the mozilla-rootcerts install command.
It no longer works as is in NetBSD 9.0. The mozilla-rootcerts-openssl
package is added instead, which successfully performs that same task.
Hopefully this will be a more robust approach for the future.

Continue to use an explicit URL for the NetBSD release download, and
remove unused find_latest_release.

The plan is to roll out these builders first (without trybots), gain
more confidence, enable trybots, and then remove NetBSD 8.0 builders.

For golang/go#37244.
For golang/go#30839.
For golang/go#31858.
For golang/go#31726.
For golang/go#38283.

Change-Id: Id41278eaebe64cfbcf7b1ab9bfd64e550ee0bbf1
Reviewed-on: https://go-review.googlesource.com/c/build/+/226977
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur dmitshur self-assigned this Apr 7, 2020
@gopherbot
Copy link

Change https://golang.org/cl/227538 mentions this issue: dashboard, env/netbsd-{amd64,386}: mount /tmp on disk instead of RAM

gopherbot pushed a commit to golang/build that referenced this issue Apr 8, 2020
The /tmp partition was being mounted as tmpfs -s=ram%25 by default.
On the n1-highcpu-4 machine type with 3.6 GB RAM, this resulted in
there being less than 1 GB of available disk space, which is not
sufficient to build Go and run its tests.

CL 226977 has increased the image disk size from 4 GB to 16 GB, so
using the disk for /tmp resolves issues with insufficient disk space.

I also considered using an alternative machine type like n1-standard-4
with 15 GB RAM instead, but didn't pursue that option for now because
it was not clear if it could be made to work for the 32-bit builder.

Remove the replacement of wd0 with sd0 in /etc/fstab. It is no longer
required because the NetBSD 9.0 installer defaults to installing on GPT
rather than MBR as was done in earlier versions.

For golang/go#37244.
For golang/go#30839.
For golang/go#31726.

Change-Id: Icbd0f909ab7768d775c96a6b606b5279a600f1bd
Reviewed-on: https://go-review.googlesource.com/c/build/+/227538
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Run-TryBot: Benny Siegert <bsiegert@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/227545 mentions this issue: dashboard: unset known issue for NetBSD 9.0 builders, use for trybots

gopherbot pushed a commit to golang/build that referenced this issue Apr 8, 2020
The NetBSD 9.0 builders are operational as of CL 227538. They've been
running on a few commits in all repositories without issues.

The netbsd/386 builder is passing, no longer hanging in the runtime
tests as the NetBSD 8.0 version did, so keep it enabled.

If anything unexpected comes up, we should investigate.

For golang/go#37244.
For golang/go#31726.

Change-Id: Ie3167fa3eabb7f707487561ca2cd4505572df741
Reviewed-on: https://go-review.googlesource.com/c/build/+/227545
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/227738 mentions this issue: dashboard: remove netbsd-{amd64,386}-8_0 builders

@golang golang locked and limited conversation to collaborators Apr 9, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. new-builder OS-NetBSD
Projects
None yet
Development

No branches or pull requests

7 participants