Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: netbsd-386 builds hang during make.bash #19339

Closed
bradfitz opened this issue Mar 1, 2017 · 14 comments
Closed

build: netbsd-386 builds hang during make.bash #19339

bradfitz opened this issue Mar 1, 2017 · 14 comments

Comments

@bradfitz
Copy link
Contributor

bradfitz commented Mar 1, 2017

The netbsd-386 builds hang during make.bash somewhere.

I'm disabling them from the builders for now, since the build coordinator doesn't yet deal with that well.

/cc @mdempsky @ianlancetaylor @bsiegert

@bradfitz bradfitz added this to the Go1.9 milestone Mar 1, 2017
@bradfitz
Copy link
Contributor Author

bradfitz commented Mar 1, 2017

(These are gomote accessible if anybody wants to reproduce. I'm not disabling that part. The actual host is a NetBSD 7.0 amd64 image, using GOARCH=386 / GOHOSTARCH=386, which gomote will set for you if you create a netbsd-386-70 image)

@gopherbot
Copy link

CL https://golang.org/cl/37622 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Mar 1, 2017
Updates golang/go#19339

Change-Id: Id9e5247c0ffbddf4bcf0d807840399d3a5dd83c5
Reviewed-on: https://go-review.googlesource.com/37622
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
@bradfitz
Copy link
Contributor Author

bradfitz commented Apr 6, 2017

I updated the NetBSD builders to NetBSD 7.1.

Maybe something was fixed. We'll see.

/cc @bsiegert

@bradfitz
Copy link
Contributor Author

I guess we should document in the Release Notes' Known Issues section (https://tip.golang.org/doc/go1.9#known_issues) about NetBSD not being stable and kick this to Go 1.10.

Sound okay @mdempsky, @bsiegert, @ianlancetaylor?

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9 Jun 28, 2017
@bradfitz
Copy link
Contributor Author

I just attached to the serial console of a netbsd-386-71 builder while running make.bash hoping to catch a NetBSD kernel panic like in #20836 but no luck.

I see nothing to the serial port, but the build hangs at:

##### Building packages and commands for netbsd/386.
...
cmd/trace
cmd/vendor/github.com/google/pprof/internal/proftest
cmd/vet/internal/cfg
cmd/vet/internal/whitelist
cmd/vet
                                   
^C
bradfitz@gdev:~/go$ 
bradfitz@gdev:~/go$ gomote run $MOTE go/bin/go version
go version devel gomote.XXXXX netbsd/386

Note that the machine is still usable after killing the make.bash run.

Trying it again, it hangs in a different place:

##### Building packages and commands for netbsd/386.
...
runtime/race
vendor/golang_org/x/text/secure
vendor/golang_org/x/text/unicode
cmd/compile/internal/test
cmd/vet/internal/whitelist

Looking at the GCE console, I see it's using 99% CPU.

screen shot 2017-06-28 at 12 48 30 pm

@gopherbot
Copy link

CL https://golang.org/cl/47036 mentions this issue.

gopherbot pushed a commit that referenced this issue Jun 28, 2017
Updates #20836
Updates #19339
Updates #19652
Updates #20835
Updates #16511
Updates #10166
Updates #8574

Change-Id: If9a7f560489f1a8d628dafab227925bd8989326e
Reviewed-on: https://go-review.googlesource.com/47036
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@zoulasc
Copy link
Contributor

zoulasc commented Jun 29, 2017

What are the steps to reproduce this?

@bradfitz
Copy link
Contributor Author

@zoulasc, set GOARCH=386, cd into your Go checkout's src/ directory, and then run ./make.bash and watch it spin.

We run NetBSD 7.1 VMs on GCE (KVM, virtio, etc), if that matters.

Our image is built using https://github.com/golang/build/tree/master/env/netbsd-amd64

@bradfitz
Copy link
Contributor Author

Benny, can you finish https://go-review.googlesource.com/c/build/+/78619 ?

Let's get netbsd-386 updated to 8.0+

@bradfitz
Copy link
Contributor Author

Actually I think I figured it out.

@bradfitz
Copy link
Contributor Author

The NetBSD i386 kernel never gets networking setup on GCE. See #20852 (comment)

But trying the amd64 kernel with the 32-bit NetBSD binaries with NetBSD 8-BETA, I see the same issue that we saw earlier with NetBSD 7.1: make.bash hangs and just spins CPU.

make.bash normally takes under a minute. After over 15 minutes, I saw:

$ gomote run user-bradfitz-netbsd-amd64-8branch-0 /bin/ps -a
 PID TTY   STAT     TIME COMMAND
   2 tty00 Is+   0:00.01 sh /etc/rc autoboot
   8 tty00 I+    0:00.00 sh /etc/rc autoboot
   9 tty00 I+    0:00.00 sh /etc/rc autoboot
  10 tty00 I+    0:00.03 sh /etc/rc autoboot
  11 tty00 S+    0:00.14 sh /etc/rc autoboot
  16 tty00 I+    0:00.00 sh /etc/rc autoboot
  18 tty00 S+    0:00.11 sh /etc/rc autoboot
 371 tty00 I+    0:00.00 sh /etc/rc autoboot
 503 tty00 I+    0:00.00 sh /etc/rc autoboot
 505 tty00 Il+   0:06.14 /buildlet
 590 tty00 Il+   0:01.57 ./cmd/dist/dist bootstrap -a
 931 tty00 Ol+  25:54.12 /tmp/workdir/go/pkg/tool/netbsd_386/go_bootstrap insta
2494 tty00 S+    0:00.00 sleep 3
2495 tty00 O+    0:00.00 /bin/ps -a

The go_bootstrap process was spinning or something. I failed to figure out how to ktrace it. But it was consuming CPU.

I saw no disk activity. Or at least no new files being created/appended.

Related: #22914

@gopherbot
Copy link

Change https://golang.org/cl/80435 mentions this issue: all: start to revive netbsd-386 support

@gopherbot
Copy link

Change https://golang.org/cl/80515 mentions this issue: runtime: fix sysctl calling convention on netbsd/386

gopherbot pushed a commit that referenced this issue Nov 29, 2017
Thanks to coypoop for noticing at:

  #22914 (comment)

FreeBSD/386 and NetBSD/386 diverged between Go 1.4 and Go 1.5 when
Russ sent https://golang.org/cl/135830043 (git rev 25f6b02)
to change the calling convention of the C compilers to match Go.
But netbsd wasn't updated.

Tested on a NetBSD/386 VM, since the builders aren't back up yet (due
to this bug)

Fixes #22914
Updates #19339
Updates #20852
Updates #16511

Change-Id: Id76ebe8f29bcc85e39b1c11090639d906cd6cf04
Reviewed-on: https://go-review.googlesource.com/80515
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Nov 29, 2017
It's still busted, but this will let other people debug.

Updates golang/go#22914
Updates golang/go#20852
Updates golang/go#19339

Change-Id: Iedc21417e56418dab6abd433574ffef012ae43f3
Reviewed-on: https://go-review.googlesource.com/80435
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz
Copy link
Contributor Author

I'm now getting successful all.bash runs on netbsd/386 running on a 64-bit host.

I think this was fixed by 70ee9b4

Closing.

I'll reopen new bugs if netbsd/386 binaries don't run on 32-bit NetBSD kernels, once tomorrow's NetBSD 32-bit builds are completed with the virtioscsi config fix.

@golang golang locked and limited conversation to collaborators Nov 29, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants