Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: fix broken Docker on arm64 packet builders #24810

Closed
FiloSottile opened this issue Apr 11, 2018 · 3 comments
Closed

x/build: fix broken Docker on arm64 packet builders #24810

FiloSottile opened this issue Apr 11, 2018 · 3 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone
Unreleased

Comments

@FiloSottile
Copy link
Contributor

Some of the arm64 packet builders stopped working.

Over time, the docker daemon lost track of more and more exited containers.

root@go-builder:~# docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
e9da62e0e524        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 seconds ago       Up 2 seconds                            packet07
be4cad637e87        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   About an hour ago   Up About an hour                        packet09
4198b1b3aaf5        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   About an hour ago   Up About an hour                        packet15
48bada399702        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   About an hour ago   Up About an hour                        packet19
5ca10cbf91ad        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   About an hour ago   Up About an hour                        packet20
8c78fd256b38        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   2 hours ago         Up 2 hours                              packet16
4b89ff505063        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 hours ago         Up 3 hours                              packet01
b7918b0fc69e        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 hours ago         Up 3 hours                              packet03
3ea996a7951c        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 hours ago         Up 3 hours                              packet11
51d2bb94fca2        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 hours ago         Up 3 hours                              packet13
7e3530aa3598        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 hours ago         Up 4 hours                              packet02
bed226aba206        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 weeks ago         Up 3 weeks                              packet04
4e4eba80cfb6        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 weeks ago         Up 4 weeks                              packet12
aae8adfd568c        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   5 weeks ago         Up 5 weeks                              packet17
a56f27c8b515        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   3 months ago        Up 3 months                             packet14
39f54b77d230        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 months ago        Up 4 months                             packet08
e64dd9975d1c        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 months ago        Up 4 months                             packet06
595355800eba        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   4 months ago        Up 4 months                             packet05
69559e6aa207        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   5 months ago        Up 5 months                             packet10
39b1689d4c2a        gobuilder-arm64-packet:1   "/usr/local/bin/stage"   5 months ago        Up 5 months                             packet18

root@go-builder:~# docker logs packet04
2018/03/21 03:40:36 network is up.
2018/03/21 03:40:36 Downloading https://storage.googleapis.com/go-builder-data/buildlet.linux-arm64 to ./buildlet.exe ...
2018/03/21 03:40:37 Downloaded ./buildlet.exe (7302642 bytes)
2018/03/21 03:40:37 buildlet starting.
2018/03/21 03:40:37 Dialing coordinator farmer.golang.org:443 ...
2018/03/21 03:40:37 Doing TLS handshake with coordinator (verifying hostname "farmer.golang.org")...
2018/03/21 03:40:38 Registering reverse mode with coordinator...
2018/03/21 03:40:38 Connected to coordinator; reverse dialing active
2018/03/21 08:37:19 buildlet reverse mode exiting.

buildlet reverse mode exiting is followed by os.Exit.

Trying to exec suggests Docker lost track of them.

root@go-builder:~# docker exec -it packet04 bash
rpc error: code = 2 desc = containerd: container not found

There is no spot-on Moby issue that matches this, but some similar ones over the last year. Upgrading and restarting Docker assuming it's a known issue. Will reopen if it keeps happening.
@gopherbot gopherbot added this to the Unreleased milestone Apr 11, 2018
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Apr 11, 2018
@bradfitz
Copy link
Contributor

What's the current version of Docker and what are we upgrading to?

@FiloSottile
Copy link
Contributor Author

Before:

root@go-builder:~# docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Tue Jan 31 23:35:02 2017
 OS/Arch:      linux/arm64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   78d1802
 Built:        Tue Jan 31 23:35:02 2017
 OS/Arch:      linux/arm64

The following packages will be upgraded:
  apparmor apt apt-utils base-files bsdutils cloud-guest-utils cloud-init distro-info-data dnsmasq-base docker.io dpkg initramfs-tools initramfs-tools-bin
  initramfs-tools-core iproute2 isc-dhcp-client isc-dhcp-common klibc-utils kmod libapparmor-perl libapparmor1 libapt-inst2.0 libapt-pkg5.0
  libaudit-common libaudit1 libavahi-client3 libavahi-common-data libavahi-common3 libblkid1 libcryptsetup4 libfdisk1 libgcrypt20 libgnutls-openssl27
  libgnutls30 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libklibc libkmod2 libmount1 libpam-modules libpam-modules-bin libpam-runtime libpam0g
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib librados2 librbd1 libseccomp2 libsmartcols1 libsystemd0 libudev1 libuuid1 logrotate mdadm
  mount ntp ntpdate open-iscsi python python-apt python-apt-common python-minimal python2.7 python2.7-minimal python3-apt runc ssh-import-id sysstat
  systemd systemd-sysv ubuntu-fan udev unattended-upgrades util-linux uuid-runtime xdg-user-dirs

After systemctl restart docker:

root@go-builder:~# docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.6.2
 Git commit:   092cba3
 Built:        Thu Nov  2 06:39:28 2017
 OS/Arch:      linux/arm64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.6.2
 Git commit:   092cba3
 Built:        Thu Nov  2 06:39:28 2017
 OS/Arch:      linux/arm64
 Experimental: false

@FiloSottile
Copy link
Contributor Author

Builders seem happy. Closing and hoping not to reopen.

@golang golang locked and limited conversation to collaborators Apr 11, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @FiloSottile @gopherbot