Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: optimize FreeBSD image boot time #22894

Closed
bradfitz opened this issue Nov 28, 2017 · 11 comments
Closed

x/build: optimize FreeBSD image boot time #22894

bradfitz opened this issue Nov 28, 2017 · 11 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted OS-FreeBSD Performance
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

I noticed while testing my images for #19303 and #22854 that our FreeBSD images have a 10 second boot prompt countdown. (I watched it count down with cute art on "serial" over the remote serial API)

Let's remove that 10 second countdown and get FreeBSD booting faster. 10 seconds is a notable percentage of the time our VMs live.

/cc @paulzhol
@gopherbot gopherbot added this to the Unreleased milestone Nov 28, 2017
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 28, 2017
@paulzhol
Copy link
Member

The images need to have /boot/loader.conf appended with autoboot_delay=0 and maybe beastie_disable=YES lines. I'll can send a CL tonight.

@bradfitz
Copy link
Contributor Author

Thanks!

@paulzhol paulzhol self-assigned this Nov 28, 2017
@gopherbot
Copy link

Change https://golang.org/cl/80335 mentions this issue: env/freebsd-amd64: disable boot prompt and logo

@bradfitz
Copy link
Contributor Author

While running make.bash with this latest code on 11.1, I noticed a kernel crash:

===> Creating users
Creating user 'git_daemon' with uid '964'.
Extracting git-2.14.1:  98%panic: ufs_dirbad: /: bad dir ino 1048795 at offset 0
: mangled entry
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff80aada97 at kdb_backtrace+0x67
#1 0xffffffff80a6bb76 at vpanic+0x186
#2 0xffffffff80a6b9e3 at panic+0x43
#3 0xffffffff80d3e0c9 at ufs_lookup_ino+0xe89
#4 0xffffffff8104b963 at VOP_CACHEDLOOKUP_APV+0x83
#5 0xffffffff80b13786 at vfs_cache_lookup+0xd6
#6 0xffffffff8104b853 at VOP_LOOKUP_APV+0x83
#7 0xffffffff80b1d0c1 at lookup+0x701
#8 0xffffffff80b1c576 at namei+0x486
#9 0xffffffff80b3772c at vn_open_cred+0xcc
#10 0xffffffff80b30e5e at kern_openat+0x1fe
#11 0xffffffff80ee0394 at amd64_syscall+0x6c4
#12 0xffffffff80ec392b at Xfast_syscall+0xfb
Uptime: 7m29s
Automatic reboot in 15 seconds - press a key on the console to abort

And then it rebooted:

cd0: Serial Number QM00003
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: 0MB (185 2048 byte sectors)
GEOM: vtbd0: the secondary GPT header is not in the last LBA.
random: unblocking device.
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
WARNING: / was not properly dismounted
Setting hostuuid: 73dce969-d480-11e7-99d3-b15cb8c9d93f.
Setting hostid: 0x281f1cb5.
Entropy harvesting: interrupts ethernet point_to_point swi.
warning: total configured swap (262144 pages) exceeds maximum recommended amount
(204672 pages).
warning: increase kern.maxswzone or reduce amount of swap.
Starting file system checks:
/dev/gpt/rootfs: UNALLOCATED  I=1765670  OWNER=root MODE=100644
/dev/gpt/rootfs: SIZE=0 MTIME=Nov 28 21:09 2017
/dev/gpt/rootfs: FILE=/var/log/utx.lastlogin

/dev/gpt/rootfs: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY.
Automatic file system check failed; help!
ERROR: ABORTING BOOT (sending SIGTERM to parent)!
Nov 28 21:09:45 init: /bin/sh on /etc/rc terminated abnormally, going to single
user mode
Enter full pathname of shell or RETURN for /bin/sh:

Fun.

Want to file an upstream FreeBSD kernel bug?

@paulzhol
Copy link
Member

yikes! does it reproduce? You're running this inside a GCE linux VM right? (When I tried locally I flipped the --enable-kvm flag to speed things up, and it was fine).

@bradfitz
Copy link
Contributor Author

Yes, I'm in a GCE Debian Jessie VM, running your make.bash unmodified. (so qemu with whatever options you specified)

I've seen it twice now: in both 11.1, and 10.3.

I'm in a GNU Screen session, if that matters, and both times I saw it happened might've been while I was flipping between virtual terminals. (Does screen send some signal or keypess when I do that which is confusing qemu?)

@bradfitz
Copy link
Contributor Author

But I just re-ran ./make.bash 11.1 and it succeeded now.

So it's not 100% of the time.

@paulzhol
Copy link
Member

I'm in a GNU Screen session, if that matters, and both times I saw it happened might've been while I was flipping between virtual terminals. (Does screen send some signal or keypess when I do that which is confusing qemu?)

with screen and qemu is running with -curses under expect there could be something terminal related.

But this

===> Creating users
Creating user 'git_daemon' with uid '964'.

came from a regular pkg install of git. I'll try to reproduce, and get a kernel dump if I can.

@gopherbot
Copy link

Change https://golang.org/cl/80304 mentions this issue: dashboard: rebuild FreeBSD VM to avoid 10 second boot prompt

gopherbot pushed a commit to golang/build that referenced this issue Nov 28, 2017
@bradfitz @andybons
dashboard: rebuild FreeBSD VM to avoid 10 second boot prompt
68f76d8 
FreeBSD VMs boot up 10 seconds faster now, and shut down faster.

Updates golang/go#22894
Updates golang/go#22895

Change-Id: I8eef979fc43d95d3c03104a761e70ec5737a3a97
Reviewed-on: https://go-review.googlesource.com/80304
Reviewed-by: Andrew Bonventre <andybons@golang.org>
@paulzhol
Copy link
Member

@bradfitz does adding -m 1G to the qemu command help? Without specifying it qemu defaults to 128MiB, it might be a bit low.

diff --git a/env/freebsd-amd64/make.bash b/env/freebsd-amd64/make.bash
index 0df60bd..201d3bc 100755
--- a/env/freebsd-amd64/make.bash
+++ b/env/freebsd-amd64/make.bash
@@ -148,7 +148,7 @@ sleep 2
 # TODO(wathiede): set serial output so we can track boot on GCE.
 expect <<EOF
 set timeout 600
-spawn qemu-system-x86_64 -display curses -smp 2 -drive if=virtio,file=disk.raw,cache=none -cdrom config.iso -net nic,model=virtio -net user
+spawn qemu-system-x86_64 -display curses -smp 2 -m 1G -drive if=virtio,file=disk.raw,cache=none -cdrom config.iso -net nic,model=virtio -net user
 
 # Speed-up boot by going in to single user mode.
 expect "Welcome to FreeBSD"

@bradfitz
Copy link
Contributor Author

I can try that next time I update the FreeBSD images, but now I have to update some other operating systems.

@golang golang locked and limited conversation to collaborators Nov 28, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge help wanted OS-FreeBSD Performance
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @paulzhol @gopherbot