x/build/builders: ARM builders #8647
Comments
Saving for reference - https://wiki.ubuntu.com/ARM/RootfsFromScratch/QemuDebootstrap I'll test this out for building an ARM debian rootfs entirely from within docker. |
https://wiki.ubuntu.com/ARM/RootfsFromScratch The Debootstrap page actually suggested other methods from this page, rootstock looks promising. |
https://wiki.ubuntu.com/Core more reference. No problem :) I wish gc had mips support too since I actually have the hardware. |
One final reference for buildroot: http://buildroot.uclibc.org/ http://git.buildroot.net/buildroot/tree/board/qemu/arm-versatile/readme.txt Buildroot isn't a full distro (it uses busybox etc) and everything is built from source. If I mirror the tarballs that our config needs, then it is really versatile as you can easily swap kernel versions and the compiled images are under 20 megs. What building a filesystem looks like in buildroot: make goarmbuilder_defconfig make I'll hopefully get a prototype Dockerfile running tonight (New Zealand time). |
Yeah, this is really trying to be a sanity test and a way for people to get a working arm build to debug things. more notes: debootstrap and cdebootstrap are failing on the most recent ubuntu version and docker. I'll need to investigate those further. Buildroot worked well until I realized they removed support for installing a gcc toolchain in the built filesystem. I need to configure my own way to cross compile gcc to run on arm before I can use it. |
My plan is to generate an arm ubuntu or debian base image, install ssh + ssh keys into the image, run the image with qemu-system-arm in docker, then forward commands and scp files from docker to it using qemu userspace tcp redirection. I'm still trying to find a reliable image building system we can use. My next bet to get a vexpress arm ubuntu image is following this guide: https://wiki.ubuntu.com/Kernel/Dev/QemuARMVexpress. The problem with this is that docker build can't mount loop devices (only run --privileged) and I haven't yet tested qemu user mode emulation inside docker which does something to install itself into the system elf execution. Maybe building the arm image base to the point it can run ssh may need to be a different script (possibly with vagrant vm if docker can't do it well enough.). The output of this script is a kernel + initrd + ext filesystem, which can be saved and the docker file consumes it. |
|
Comment 10 by AndrewChamberss: Can someone try this docker file out: https://github.com/andrewchambers/golang-armbuilder-tests It runs a single core arm7 processor with debian using the aurel32 armhf images. I don't really know how to setup and test the builder, but this docker image at least lets you ssh in so it can be used as a starting point. I worry that the debian apt-get install gcc might stop working if it gets out of date and the aurel32 person hasn't updated the images in a while, it might be worth hosting our own image with everything needed preinstalled. I also created a vagrant file and some scripts following https://wiki.ubuntu.com/Kernel/Dev/QemuARMVexpress which can generate a dual core ubuntu arm image: https://github.com/andrewchambers/arm-vexpress-dualcore-image Unfortunately it seems really unstable, and randomly fails to boot/hangs with kernel errors. |
|
I'm making this bug a bit more general about updating all our ARM builders in the new http://golang.org/s/builderplan world, be it emulated, real hardware, or an ARM hosting service like http://labs.online.net/ ... I imagine we'll do all three. For real hardware I want to use Docker, which means we'll need an ARM distro that does Docker. I hear Arch does it. /cc @minux |
|
Online.net's API: https://doc.cloud.online.net/api/ |
|
Testing Go on the new ARM online.net server (Ubuntu 14.10, GNU/Linux 3.17.0-119 armv7l, Marvell a Cortex A9 ARM, 2Gb RAM, 20Gb LSSD). Go compilation (tip) : Docker doesn't work : @bradfitz, ping me if you want a access to a server (beta service). |
|
If we want maximum speed, i think we can wait a little bit for and arm64 should all have very good virtualization support the only problem is that you can't run 32-bit kernel on arm64 |
|
@sdouche, I have an online.net account and have used it. Docker should work if you use an ARM image. Are you sure your "ubuntu" isn't using an x86 ubuntu image? |
|
@bradfitz ok. And you're right, ubuntu is only for x86. Online provides Docker images. |
|
Now that we have trybots, and trybots require VMs, this bug is now high priority, so we're able to get ARM-based trybots. Test sharding is coming soon, too, so I don't care much about speed. I also found https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Virtual_ARM_Linux_environment which is basically what I did before (and I lost my old notes) but uses the Cortex A9 processor. Maybe it's faster. I would now like to see a Dockerfile which generates an x86 Debian or Ubuntu container which is bootable using docker2boot (like our Linux VMs), which uses qemu-system-arm to then boot an ARM system (can also be Debian, or whatever) that then downloads the buildlet from the GCE metadata service and runs it. So the coordinator will also be interacting with an emulated buildlet. I hope that's fast enough. Or since we have to do OS X differently anyway, maybe we focus on OS X first (which will connect to the coordinator to register idle VMs) and then have ARM do the same thing as OS X, but use a bunch of real ARM machines (I have some idle ones), or ARM machines on online.net, and then we make the coordinator just send VM create/destroy requests to online.net as needed. But if somebody wants to work on the GCE qemu-system-arm thing, I would love it. Even if you get it to the point of working mostly in Docker, I can do the docker2boot + buildlet-downloading parts. Just make sure to make the ARM filesystem have curl installed already, and configure the ARM system to run a shell script at boot, and I can take it from there. /cc @adg |
|
Also, @davecheney, maybe you could provide input or help on this stuff. You also have an interest in seeing ARM tested more. |
|
Thanks @bradfitz. If I have been following correctly, integrating OS X builders will work by If this is right then adding a pool of ARM capable machines should be My personal opinion is qemu-arm is more trouble than it's worth, compared Freebsd/arm is something I'd like to revive, I hope before the close of On Fri, Feb 13, 2015 at 11:06 AM, Brad Fitzpatrick <notifications@github.com
|
|
As I'll probably be the one managing the Google-run ARM hardware (I had a corp lab setup for the android builder), I would like reliable hardware. Does any exist? |
|
The new raspberry Pi 2, although not yet proven should be a good first port Others disagree, but my 3 year old pandaboard has always been very reliable Outside my price range are blade chassis type systems intended for VPS On Fri, Feb 13, 2015 at 10:35 PM, David Crawshaw notifications@github.com
|
|
What's the craic on this? I've been working on something similar. I have a little script which builds an arm image based on ubuntu-core (not snappy), and an amd64 Dockerfile which runs the arm image on qemu on boot. It can also run the official Docker binaries, so I'm happy to massage the script to produce an image to run on real hardware. |
|
Here's a starting point: https://go-review.googlesource.com/4812 I've updated it to run the buildlet's stage0 on boot. (Also, I now know what the craic is. I had this tab open since yesterday and forgot to refresh before posting.) |
|
If we wait a little longer, perhaps we can use some arm64 servers
to host 32-bit ARM builders with KVM.
Samsung chromebooks are pretty solid ARM machines if you remove
the battery. I'm not sure if it could run Android though, but it definitely
makes some good 32-bit ARM builders.
(I remembered Russ shared this
http://www.systemcall.org/blog/2014/06/trashing-chromebooks/)
For Android machine, buy some big RAM ones, and do the whole build
in tmpfs (don't forget to also set TMPDIR to point to tmpfs). I'm not sure
if the batter is going to be a problem, but it's better to remove them if
possible.
|
|
Re-opening, because the builders all failed, like: http://build.golang.org/log/c165b6945c12d6c8f1f73aff54b99da6bf0e3651 |
|
Salman, any ideas? Which GCE instance size did you use? |
|
5g is killed by the OOM killer. We need to increase the memory for the VM.
2G should be more than enough.
|
|
But Salman said it worked for him.
The VM host or the qemu VM in the VM? I think the latter. But I also thought QEMU only let you have 1GB of memory available for the emulated ARM machine. I see in Salman's CL:
... which is only 512 MB. But https://wiki.linaro.org/PeterMaydell/QemuVersatileExpress shows using that board with 1024 at least. Can we go higher than 1024 even? http://fedoraproject.org/wiki/Architectures/ARM/Versatile_Express suggests 1GB might be the limit. Salman, why did you make it 512 MB? And why did you have more success with the builds than me? Maybe you were only building Go 1.4.2 and the builders are running tip, which takes more memory? |
|
It does look like an OOM that I thought I no longer got since I upped it to On 27 February 2015 at 21:58, Brad Fitzpatrick notifications@github.com
سلمان |
|
And yes, qemu refuses anything more than 1GB. I successfully ran 1.4 make.bash with only 256MB, so I also suspect this has to do with with tip's 5g. (It's now in go, right?) |
|
I recently heard that on openbsd/386, 8g needs about 800MB memory to run.
so perhaps 1GB should be OK.
Whether 1GB is the limit depends on the physical memory layout and kernel
(device tree for linux).
In the past, I successfully modified qemu and netbsd/arm kernel to have
a larger memory than is possible originally to run the netbsd-arm-qemu
builder.
|
|
Ok, the pprof tests are now failing on qemu. Any ideas? |
|
I think we should set $IN_QEMU=1 inside the qemu-system-arm VM
to skip the test.
The pprof tests can't be tested reliably inside emulated VMs.
|
|
Yes, let's do that. |
|
Does the qemu-arm builder have gdb (with python binding) installed?
If not, we should add gdb to the debootstrap command line.
|
|
It's already got the gdb package, which includes the python bindings afaik. On 1 March 2015 at 00:53, Minux Ma notifications@github.com wrote:
سلمان |
|
Ah! I see, this is actually #9605. (I'm a bit slow, sorry!) |
This starts the process of making the coordinator about to use the buildlet on things other than GCE. Update golang/go#8647 (ARM, reverse proxy or online.net) Update golang/go#10267 (windows 2003 on AWS) Update golang/go#9495 (OS X VMWare VMs on racked Mac Minis) Change-Id: I5df79ea67e0ececba8b880e81bd93d4c80897455 Reviewed-on: https://go-review.googlesource.com/8198 Reviewed-by: David Crawshaw <crawshaw@golang.org>
|
Status update: we're now running 20 machines on Scaleway (previously online.net) using the new "reverse buildlet pool" in the coordinator. They take 30 minutes to run all.bash, or 60 minutes for GOARM=5. We still don't have sharding. Once we have sharding, qemu might be a good option again (especially if it can emulate GOARM=5 better than Scaleway, with more alignment strictness), but real hardware is nice too. |
|
CL https://golang.org/cl/10052 mentions this issue. |
…caleway This is what I used to start the ARM scaleway servers. I forgot to mail it out. I'd like to submit this before I tweak it further. Updates golang/go#8647 Change-Id: Icf789de4e3acae8084fd75151ef5ef02d8073b73 Reviewed-on: https://go-review.googlesource.com/10052 Reviewed-by: Andrew Gerrand <adg@golang.org>
|
Closing this. We can open up new bugs for ARM64. |
The text was updated successfully, but these errors were encountered: