Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: update NetBSD builders to NetBSD 8-beta #20852

Closed
bradfitz opened this issue Jun 29, 2017 · 77 comments
Closed

x/build: update NetBSD builders to NetBSD 8-beta #20852

bradfitz opened this issue Jun 29, 2017 · 77 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted OS-NetBSD
Milestone

Comments

@bradfitz
Copy link
Contributor

In https://go-review.googlesource.com/c/47036/, Christos Zoulas mentions that there are known bugs in the NetBSD 7.1 kqueue code and we should switch our builders to NetBSD 8-beta.

Also mentioned in #20836 (comment)

This is a tracking bug to update our NetBSD builders past 7.1

I've updated https://golang.org/wiki/MinimumRequirements too.

/cc @krytarowski @bsiegert @zoulasc @adams-sarah @andybons

@gopherbot gopherbot added this to the Unreleased milestone Jun 29, 2017
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jun 29, 2017
@bradfitz bradfitz modified the milestones: Go1.10, Unreleased Jun 29, 2017
@zoulasc
Copy link
Contributor

zoulasc commented Jul 9, 2017

This should also be fixed in the 7.1 branch now, builds available from:
https://releng.netbsd.org/builds/netbsd-7-1/201707090640Z/

@bsiegert
Copy link
Contributor

I would be more comfortable with a netbsd-8 system than with 7.1. http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=52034 is still pending, and it means that significant disk activity is likely to crash the VM, or make it hang.

@gopherbot
Copy link

CL https://golang.org/cl/48091 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Jul 14, 2017
There have been a number of fixes for issues found with the Go test
suite in kqueue, signal handling and vio drivers.

Updates golang/go#20852

Change-Id: I95861220990c05f3e89a4a40893e17758d29ebe0
Reviewed-on: https://go-review.googlesource.com/48091
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

CL https://golang.org/cl/48552 mentions this issue.

@bradfitz
Copy link
Contributor Author

@bsiegert, this doesn't boot on GCE as far as I can tell:

$ gomote create netbsd-amd64-8branch
Error running create: failed to create buildlet: 500 Internal Server Error: buildlet didn't come up at http://10.240.0.21 in 5m0s

Please let me know once you've actually run it on GCE.

Until that time, Go will continue to not run NetBSD builders.

@bradfitz
Copy link
Contributor Author

I didn't mean to close this. Reopened.

@zoulasc
Copy link
Contributor

zoulasc commented Jul 15, 2017

Are there any log messages on how -8 fails to boot on GCE, or are there any GCE hosts available for us to try?

@bradfitz
Copy link
Contributor Author

I didn't have time to capture the serial console. I had assumed @bsiegert tested this.

You can create small GCE VM instances yourself for free: https://cloud.google.com/free/

@bsiegert
Copy link
Contributor

The relevant bit from the serial console:

+ curl -H 'Metadata-Flavor: Google' 'http://metadata.google.internal/computeMetadata/v1/instance/attributes/buildle
t-binary-url'
Shared object "libssl.so.10" not found
+ curl -o /buildlet
Shared object "libssl.so.10" not found
+ echo 'giving up'
+ command echo 'giving up'
giving up
+ sleep 10
+ halt -p
Jul 15 19:11:28 netbsd8-builder-20170715 halt: halted by root

The binary packages are for NetBSD-7, and I suppose they are linked against base OpenSSL from NetBSD-7 :( Sorry for not catching this before sending out the CL.

@zoulasc
Copy link
Contributor

zoulasc commented Jul 15, 2017

Yes, it is unfortunate that there are no pre-built packages for -8 yet. I will work on that. Do you need anything else from me?

@bsiegert
Copy link
Contributor

I'll try unpacking a /lib and /usr/lib from NetBSD-7 to fix this.

@zoulasc
Copy link
Contributor

zoulasc commented Jul 17, 2017

the better way is to make the symlinks to the missing shared libraries to point to the new ones...

@bsiegert
Copy link
Contributor

@zoulasc That sounds unsafe if there was a major bump in between.

@gopherbot
Copy link

CL https://golang.org/cl/49670 mentions this issue.

@zoulasc
Copy link
Contributor

zoulasc commented Jul 19, 2017

it is not an issue, people were just being conservative with libcrypto. OTOH if you replace libraries with older versions every binary on the system will break. But no worries, I'll have packages built soon.

@gopherbot
Copy link

Change https://golang.org/cl/53010 mentions this issue: x/build: use NetBSD-8 packages for netbsd-amd64 builder

gopherbot pushed a commit to golang/build that referenced this issue Aug 4, 2017
This should fix the latest set of errors that bradfitz@ was seeing.

Updates golang/go#20852

Change-Id: Iad7d6a479c5546395de907731643c1d817fe625d
Reviewed-on: https://go-review.googlesource.com/53010
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor Author

bradfitz commented Aug 4, 2017

@bsiegert, it still doesn't work.

Logs from the new x/build/cmd/debugnewvm tool running debugnewvm --host=host-netbsd-8branch --run-build=netbsd-amd64-8branch:

https://gist.github.com/bradfitz/3a6ba42ed3f9d7a47338e2172cfab9bb

@bsiegert
Copy link
Contributor

bsiegert commented Aug 4, 2017

Thanks for testing, and thanks for making me aware of debugnewvm.

It appears that this time, there has been a regression around setting routes. There is a default route via .0.1 but the host route to .0.1 is missing for some reason.

@bsiegert
Copy link
Contributor

bsiegert commented Sep 5, 2017

It took me a long time to find out, but it is a small bug in dhcpcd, the DHCP client. I will get this fix into NetBSD-8, then try again.

@bsiegert
Copy link
Contributor

bsiegert commented Sep 5, 2017

@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 1, 2017

@bsiegert, I just rebuilt the 386 image. It boots saying NetBSD 8.0_BETA (GENERIC.201711301510Z) which is newer than the 201711280120Z I see above. But still no luck:

virtio0: Virtio SCSI Device (rev. 0x00)
virtio0: no matching child driver; not configured

I'll try again tomorrow.

@krytarowski
Copy link
Contributor

https://releng.netbsd.org/cgi-bin/req-8.cgi just monitor 413. Martin who committed the fix cannot pullup it and we need another person from releng to test and merge the patch.

@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 1, 2017

@krytarowski, thanks for the info & link!

@gopherbot
Copy link

Change https://golang.org/cl/81356 mentions this issue: dashboard, cmd/buildlet, env/netbsd: enable netbsd ssh, re-disable netbsd-386

gopherbot pushed a commit to golang/build that referenced this issue Dec 1, 2017
…tbsd-386

Fixes golang/go#22872
Updates golang/go#20852

Change-Id: I2b3454cb59a434cb12b10bb93cb4dd21d5187281
Reviewed-on: https://go-review.googlesource.com/81356
Reviewed-by: Andrew Bonventre <andybons@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/81735 mentions this issue: dashboard: re-enable NetBSD/amd64 builder

gopherbot pushed a commit to golang/build that referenced this issue Dec 4, 2017
Updates golang/go#22968
Updates golang/go#20852

Change-Id: Ida2bc7213d418735e897d0bae8e5fba9ae18fd6c
Reviewed-on: https://go-review.googlesource.com/81735
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 5, 2017

@bsiegert, the NetBSD/386 VM now boots on GCE, but I can't find any NetBSD mirror with binary packages for i386, which means we have no curl and we can't download the buildlet.

The builder script fails on these lines,

https://github.com/golang/build/blob/master/env/netbsd-amd64/mkvm.py#L63

But doesn't stop building the VM image on error.

localhost# PS1='anita-root-shell-pr''ompt-1512499919.93# '
PS1='anita-root-shell-pr''ompt-1512499919.93# '
anita-root-shell-prompt-1512499919.93# env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add bash curl
env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add bash curl
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//bash*: Not Found
pkg_add: no pkg found for 'bash', sorry.
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//curl*: Not Found
pkg_add: no pkg found for 'curl', sorry.
pkg_add: 2 package additions failed
anita-root-shell-prompt-1512499919.93# echo exit_status=$?=
echo exit_status=$?=
exit_status=1=
anita-root-shell-prompt-1512499919.93# exec /bin/sh
exec /bin/sh
localhost# PS1='anita-root-shell-pr''ompt-1512499922.04# '
PS1='anita-root-shell-pr''ompt-1512499922.04# '
anita-root-shell-prompt-1512499922.04# env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add git-base
env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add git-base
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//git-base*: Not Found
pkg_add: no pkg found for 'git-base', sorry.
pkg_add: 1 package addition failed

Does somebody build i386 packages?

Or do we need to build from source? (That's what I did on my test NetBSD/i386 VM, but it was super tedious and involved checking out all of the pkgsrc cvs repo.... I'd rather not do that all in qemu)

@krytarowski
Copy link
Contributor

We need to rebuild the packages for NetBSD-8/i386.

I've noted execution of "race" tests for FreeBSD, Linux, Windows - is this ThreadSanitizer (TSan) related? If so, we could add a NetBSD/amd64 target.

@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 5, 2017

@krytarowski, who rebuilds packages? When can that be done by?

As for race detector support, that's unrelated to this issue. There is at least #19273 open for more platform support. Or you could file a new bug. But it would likely require upstream TSAN support. It won't happen for Go 1.10.

@krytarowski
Copy link
Contributor

We, have already passed the requests internally to responsible developers.If I recall correctly, Manuel Bouyer was generating them.. however -current branches (like -8) are special cases so they will be generated on demand.

@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 5, 2017

Oh, you already ported TSAN (https://blog.netbsd.org/tnf/entry/one_year_checkpoint_and_thread). Feel free to file a new bug for tracking Go support.

@zoulasc
Copy link
Contributor

zoulasc commented Dec 5, 2017

I am trying to build i386 packages for NetBSD-8.0...

@gopherbot
Copy link

Change https://golang.org/cl/82018 mentions this issue: dashboard: re-disable netbsd/386 builder until binary packages are built

gopherbot pushed a commit to golang/build that referenced this issue Dec 6, 2017
Also update the bootstrap toolchain (verified in VMware VM locally).

Updates golang/go#20852

Change-Id: Iac875419ed8dc51fb9db09d8948067f24c601e17
Reviewed-on: https://go-review.googlesource.com/82018
Reviewed-by: Andrew Bonventre <andybons@golang.org>
@zoulasc
Copy link
Contributor

zoulasc commented Dec 6, 2017

There should be packages now for 8.0 + i386. Please let me know if I missed any dependencies.

@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 6, 2017

@zoulasc, I don't seen mozilla-rootcerts here:

http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q3/All/

The complete set we try to install is:

    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add bash curl" % (arch, release),
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add git-base" % (arch, release),
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add mozilla-rootcerts go14" % (arch, release),
    # Interactive debugging tools for users using gomote ssh:
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add emacs25-nox11 vim screen" % (arch, release),

But emacs, vim, screen, go14, and even git-base should be optional. They're more nice-to-have for debugging. But without mozilla-rootcerts, curl & Go might have issues fetching resources.

@zoulasc
Copy link
Contributor

zoulasc commented Dec 6, 2017

stay tuned :-)

@zoulasc
Copy link
Contributor

zoulasc commented Dec 6, 2017

you should have everything but emacs... which should come later.

@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 6, 2017

@zoulasc, thanks. I'm kicking off a build of the image.

@zoulasc
Copy link
Contributor

zoulasc commented Dec 6, 2017

thank you for all your patience :-) I am building git now.

@gopherbot
Copy link

Change https://golang.org/cl/82019 mentions this issue: dashboard, env/netbsd-386: update and re-enable netbsd/386 builder

gopherbot pushed a commit to golang/build that referenced this issue Dec 6, 2017
The necessary binary packages are now available at:
http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q3/All/

Updates golang/go#20852

Change-Id: I20660a29f9e4044899de75d1b632c34619fca19b
Reviewed-on: https://go-review.googlesource.com/82019
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 6, 2017

Okay, deployed. I have it configured to only use 1 VM at a time so it won't use our self-imposed 500 VM quota all at once to catch up, but it'll catch up slowly on its own. I ran one test build manually and watched it get through the start, so I see networking on GCE now works, and bash & curl work, the buildlet starts (I needed to rebuild it for the sysctl fix), and make.bash starts running. I assume it'll be fine. Famous last words. Off to sleep...

Thanks again, @zoulasc and @bsiegert and @krytarowski!

@krytarowski
Copy link
Contributor

krytarowski commented Dec 6, 2017

Thanks for your effort! The next release should ship with TSan, MSan and ARM CI machines on NetBSD!

@andybons
Copy link
Member

andybons commented Dec 6, 2017

Really impressed with all the collaboration on this effort. Nice work, everyone! 🎉

@bradfitz
Copy link
Contributor Author

bradfitz commented Dec 6, 2017

And looks good this morning. Still some flakes here & there, but much better.

I'm going to close this bug. We can open new ones for specific issues that come up.

@bradfitz bradfitz closed this as completed Dec 6, 2017
@golang golang locked and limited conversation to collaborators Dec 6, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted OS-NetBSD
Projects
None yet
Development

No branches or pull requests

6 participants