Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/coordinator: kube buildlets failing #14112

Closed
bradfitz opened this issue Jan 27, 2016 · 13 comments
Closed

x/build/cmd/coordinator: kube buildlets failing #14112

bradfitz opened this issue Jan 27, 2016 · 13 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

Hey @evandbrown, ...

The kube buildlets are unhappy, the farmer says:

Jan 27 07:02:31 farmer.c.symbolic-datum-552.internal dockerd[756]: time="2016-01-27T07:02:31.318909079Z" level=info msg="GET /v1.20/containers/json?all=1&filters=%7B%22status%22%3A%5B%22exited%22%5D%7D"
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 26397f1383ff02770a62160891e9de349aa1fb92  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube d9fdbf48207e24c1d0d771250f2cc811ef97adff  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 313fd1cb1343e46b563b6b8acfef7e58604b5b8f  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 4ec2fd3e6ac4f869d39348bf48016687b731d910  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 9202e9e1b81d1c6d07b516eae2d1da28951c7a76  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 09940b92a08203aa3d2baa90fc29b80ccfcb32c5  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 08594ac7c7e123d4aa46f60690da0a7e4034f4e9  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 73d590b4bd3e4dbf48613a09db938bf2fe03a1aa  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube eb3b1830b01fca572fcdd9c95aeb8090e6c09497  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube c736280e2250a9a873423f6c4923b6f4c1caf528  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 843745240430e6b6c0a94b31ca3ddc2cd77b3b06  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 1d901f55bd9d6ee12bc4692cd60a6a2311366799  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 7037c15e19582cb192cb1edfa7e2c1bf46746ee5  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube d3e61747da7f27c5dd67a514ff533be54df95b85  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube aff6aa0a21baa332bb94c34795520008c8db9198  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 2c12b81739ec2cb85073e125748fcbf5d2febb2c  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 1aab7b9626b8bc735d7340c1eb47d8d46560290f  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube bdc3db0b0c4044ed396238fb7e01574988a14aad  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube c2b40809c136c341716287a30f1a619b2921f075  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 7b773946c09e075ed50c49e76e08f61c16616ee4  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube b4c9d01d81d90a78a4330d3e3567bc5533e8c659  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 038b8139433054c70f17494f4cd96dd4f3ddd696  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 980364b7a2425657a5c66dcad4e52f6cd3723a77  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube e6347659410454cfabc4f4bc04fe37e538f472e8  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 733bf6ef67013b8410c51a72697c6fbff53ad30d  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 830143fa3dd344a72d5c00643983ab62abb88a72  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 0408ca7de12d72025d40c4d28fd8d9fb142b3c87  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 295ec4f6f8f1eb39ddb97933298351cd57b2b088  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 071fcd1ae919775149d43a01379dc8874cbe3d73  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-386-kube 99fa8c38393419fab1452ba5b157787b98f4497e  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube d9fdbf48207e24c1d0d771250f2cc811ef97adff  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 26397f1383ff02770a62160891e9de349aa1fb92  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 4ec2fd3e6ac4f869d39348bf48016687b731d910  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 313fd1cb1343e46b563b6b8acfef7e58604b5b8f  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 09940b92a08203aa3d2baa90fc29b80ccfcb32c5  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 9202e9e1b81d1c6d07b516eae2d1da28951c7a76  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube eb3b1830b01fca572fcdd9c95aeb8090e6c09497  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 08594ac7c7e123d4aa46f60690da0a7e4034f4e9  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube c736280e2250a9a873423f6c4923b6f4c1caf528  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 73d590b4bd3e4dbf48613a09db938bf2fe03a1aa  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 843745240430e6b6c0a94b31ca3ddc2cd77b3b06  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 1d901f55bd9d6ee12bc4692cd60a6a2311366799  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 7037c15e19582cb192cb1edfa7e2c1bf46746ee5  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube d3e61747da7f27c5dd67a514ff533be54df95b85  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 2c12b81739ec2cb85073e125748fcbf5d2febb2c  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube aff6aa0a21baa332bb94c34795520008c8db9198  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 1aab7b9626b8bc735d7340c1eb47d8d46560290f  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube bdc3db0b0c4044ed396238fb7e01574988a14aad  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube c2b40809c136c341716287a30f1a619b2921f075  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 7b773946c09e075ed50c49e76e08f61c16616ee4  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 038b8139433054c70f17494f4cd96dd4f3ddd696  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube b4c9d01d81d90a78a4330d3e3567bc5533e8c659  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 733bf6ef67013b8410c51a72697c6fbff53ad30d  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 980364b7a2425657a5c66dcad4e52f6cd3723a77  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 830143fa3dd344a72d5c00643983ab62abb88a72  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 295ec4f6f8f1eb39ddb97933298351cd57b2b088  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube e6347659410454cfabc4f4bc04fe37e538f472e8  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 99fa8c38393419fab1452ba5b157787b98f4497e  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 0408ca7de12d72025d40c4d28fd8d9fb142b3c87  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
Jan 27 07:02:34 farmer.c.symbolic-datum-552.internal coordinator[843]: 2016/01/27 07:02:34 {nacl-amd64p32-kube 071fcd1ae919775149d43a01379dc8874cbe3d73  } failed: failed to get a buildlet: x509 client key pair could not be generated: crypto/tls: failed to find any PEM data in certificate input
@bradfitz bradfitz added the Builders x/build issues (builders, bots, dashboards) label Jan 27, 2016
@bradfitz bradfitz added this to the Unreleased milestone Jan 27, 2016
@evandbrown
Copy link

Looking into it now. Wondering if the Container Engine API changed how it returns the certs we use to build the http client that creates pods...

@bradfitz
Copy link
Contributor Author

Actually, I might've forgotten to upload the latest coordinator binary before restarting.

I have two other CLs for you, though....

f847554 cmd/coordinator/buildongce: use the right IP address
e2424cb cmd/coordinator/buildongce: set Scopes for DefaultClient

@bradfitz
Copy link
Contributor Author

Yes, sorry, false alarm. It's now running with the right IP address: http://farmer.golang.org/ and kube builds are starting I think.

@bradfitz
Copy link
Contributor Author

Actually, it is failing.

Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-386-kube 1d901f55bd9d6ee12bc4692cd60a6a2311366799  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube b4c9d01d81d90a78a4330d3e3567bc5533e8c659  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube 830143fa3dd344a72d5c00643983ab62abb88a72  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-386-kube 0408ca7de12d72025d40c4d28fd8d9fb142b3c87  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube 2c12b81739ec2cb85073e125748fcbf5d2febb2c  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube 295ec4f6f8f1eb39ddb97933298351cd57b2b088  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube 08594ac7c7e123d4aa46f60690da0a7e4034f4e9  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube bdc3db0b0c4044ed396238fb7e01574988a14aad  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube 071fcd1ae919775149d43a01379dc8874cbe3d73  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube 733bf6ef67013b8410c51a72697c6fbff53ad30d  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube 09940b92a08203aa3d2baa90fc29b80ccfcb32c5  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-386-kube 295ec4f6f8f1eb39ddb97933298351cd57b2b088  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-386-kube e6347659410454cfabc4f4bc04fe37e538f472e8  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-386-kube aff6aa0a21baa332bb94c34795520008c8db9198  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Timed out waiting for pod to leave pending state. Pod will be deleted.
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-386-kube 2c12b81739ec2cb85073e125748fcbf5d2febb2c  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 {nacl-amd64p32-kube 4ec2fd3e6ac4f869d39348bf48016687b731d910  } failed: failed to get a buildlet: pod could not be created: context deadline exceeded
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-313fd1cb-rn020aeb" for nacl-386-kube at 313fd1cb1343e46b563b6b8acfef7e58604b5b8f
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-d9fdbf48-rn324292" for nacl-386-kube at d9fdbf48207e24c1d0d771250f2cc811ef97adff
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-9202e9e1-rnb3f712" for nacl-386-kube at 9202e9e1b81d1c6d07b516eae2d1da28951c7a76
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-4ec2fd3e-rn6c25e3" for nacl-386-kube at 4ec2fd3e6ac4f869d39348bf48016687b731d910
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-eb3b1830-rn036836" for nacl-386-kube at eb3b1830b01fca572fcdd9c95aeb8090e6c09497
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-09940b92-rnef0928" for nacl-386-kube at 09940b92a08203aa3d2baa90fc29b80ccfcb32c5
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-c736280e-rne94642" for nacl-386-kube at c736280e2250a9a873423f6c4923b6f4c1caf528
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-73d590b4-rna30096" for nacl-386-kube at 73d590b4bd3e4dbf48613a09db938bf2fe03a1aa
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-84374524-rn7aa055" for nacl-386-kube at 843745240430e6b6c0a94b31ca3ddc2cd77b3b06
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-1d901f55-rn1eaa5d" for nacl-386-kube at 1d901f55bd9d6ee12bc4692cd60a6a2311366799
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-7037c15e-rnf1fe9c" for nacl-386-kube at 7037c15e19582cb192cb1edfa7e2c1bf46746ee5
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-d3e61747-rn83caa3" for nacl-386-kube at d3e61747da7f27c5dd67a514ff533be54df95b85
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-2c12b817-rnbdeac8" for nacl-386-kube at 2c12b81739ec2cb85073e125748fcbf5d2febb2c
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-aff6aa0a-rncd3809" for nacl-386-kube at aff6aa0a21baa332bb94c34795520008c8db9198
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-1aab7b96-rn342fc1" for nacl-386-kube at 1aab7b9626b8bc735d7340c1eb47d8d46560290f
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-bdc3db0b-rn04f8ee" for nacl-386-kube at bdc3db0b0c4044ed396238fb7e01574988a14aad
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-c2b40809-rn81247e" for nacl-386-kube at c2b40809c136c341716287a30f1a619b2921f075
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-7b773946-rnc1d9f9" for nacl-386-kube at 7b773946c09e075ed50c49e76e08f61c16616ee4
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-038b8139-rn342c97" for nacl-386-kube at 038b8139433054c70f17494f4cd96dd4f3ddd696
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-733bf6ef-rnbb629f" for nacl-386-kube at 733bf6ef67013b8410c51a72697c6fbff53ad30d
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-b4c9d01d-rnc3c795" for nacl-386-kube at b4c9d01d81d90a78a4330d3e3567bc5533e8c659
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-980364b7-rnda6c7a" for nacl-386-kube at 980364b7a2425657a5c66dcad4e52f6cd3723a77
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-e6347659-rn6550ed" for nacl-386-kube at e6347659410454cfabc4f4bc04fe37e538f472e8
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-830143fa-rn01e9b9" for nacl-386-kube at 830143fa3dd344a72d5c00643983ab62abb88a72
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-0408ca7d-rn31cf86" for nacl-386-kube at 0408ca7de12d72025d40c4d28fd8d9fb142b3c87
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-295ec4f6-rne9c96a" for nacl-386-kube at 295ec4f6f8f1eb39ddb97933298351cd57b2b088
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-071fcd1a-rnea8375" for nacl-386-kube at 071fcd1ae919775149d43a01379dc8874cbe3d73
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-386-kube-99fa8c38-rn3cc428" for nacl-386-kube at 99fa8c38393419fab1452ba5b157787b98f4497e
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-d9fdbf48-rnd1d72d" for nacl-amd64p32-kube at d9fdbf48207e24c1d0d771250f2cc811ef97adff
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-4ec2fd3e-rnd1fe57" for nacl-amd64p32-kube at 4ec2fd3e6ac4f869d39348bf48016687b731d910
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-26397f13-rnd835a9" for nacl-amd64p32-kube at 26397f1383ff02770a62160891e9de349aa1fb92
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-313fd1cb-rnfb060e" for nacl-amd64p32-kube at 313fd1cb1343e46b563b6b8acfef7e58604b5b8f
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-09940b92-rne7beb0" for nacl-amd64p32-kube at 09940b92a08203aa3d2baa90fc29b80ccfcb32c5
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-9202e9e1-rnaac09f" for nacl-amd64p32-kube at 9202e9e1b81d1c6d07b516eae2d1da28951c7a76
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-eb3b1830-rn21a155" for nacl-amd64p32-kube at eb3b1830b01fca572fcdd9c95aeb8090e6c09497
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-08594ac7-rn612505" for nacl-amd64p32-kube at 08594ac7c7e123d4aa46f60690da0a7e4034f4e9
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-c736280e-rneda8ab" for nacl-amd64p32-kube at c736280e2250a9a873423f6c4923b6f4c1caf528
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-73d590b4-rn9315cf" for nacl-amd64p32-kube at 73d590b4bd3e4dbf48613a09db938bf2fe03a1aa
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-d3e61747-rnbbd5ef" for nacl-amd64p32-kube at d3e61747da7f27c5dd67a514ff533be54df95b85
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-84374524-rnde88f1" for nacl-amd64p32-kube at 843745240430e6b6c0a94b31ca3ddc2cd77b3b06
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-aff6aa0a-rn5ee29a" for nacl-amd64p32-kube at aff6aa0a21baa332bb94c34795520008c8db9198
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-7037c15e-rn2f9185" for nacl-amd64p32-kube at 7037c15e19582cb192cb1edfa7e2c1bf46746ee5
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-bdc3db0b-rn9a61e3" for nacl-amd64p32-kube at bdc3db0b0c4044ed396238fb7e01574988a14aad
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-2c12b817-rn9ec340" for nacl-amd64p32-kube at 2c12b81739ec2cb85073e125748fcbf5d2febb2c
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-7b773946-rn7d375e" for nacl-amd64p32-kube at 7b773946c09e075ed50c49e76e08f61c16616ee4
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-1aab7b96-rnd6065d" for nacl-amd64p32-kube at 1aab7b9626b8bc735d7340c1eb47d8d46560290f
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-b4c9d01d-rn2b1f69" for nacl-amd64p32-kube at b4c9d01d81d90a78a4330d3e3567bc5533e8c659
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-733bf6ef-rn1a3d93" for nacl-amd64p32-kube at 733bf6ef67013b8410c51a72697c6fbff53ad30d
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-c2b40809-rnc44c86" for nacl-amd64p32-kube at c2b40809c136c341716287a30f1a619b2921f075
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-830143fa-rn635f31" for nacl-amd64p32-kube at 830143fa3dd344a72d5c00643983ab62abb88a72
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-980364b7-rnbde116" for nacl-amd64p32-kube at 980364b7a2425657a5c66dcad4e52f6cd3723a77
Jan 27 07:17:05 farmer.c.symbolic-datum-552.internal coordinator[2118]: 2016/01/27 07:17:05 Creating Kubernetes pod "buildlet-nacl-amd64p32-kube-295ec4f6-rn164c19" for nacl-amd64p32-kube at 295ec4f6f8f1eb39ddb97933298351cd57b2b088

@bradfitz bradfitz reopened this Jan 27, 2016
@bradfitz
Copy link
Contributor Author

More info from farmer:

Recently completed

nacl-386-kube rev 071fcd1a, failed; build log; (nil *buildlet.Client), 20.580758591s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-071fcd1a-rn936ed2
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-071fcd1a-rn936ed2: pod could not be created: context deadline exceeded
nacl-386-kube rev 99fa8c38, failed; build log; (nil *buildlet.Client), 20.642004105s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-99fa8c38-rnd1a81f
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-99fa8c38-rnd1a81f: pod could not be created: context deadline exceeded
nacl-386-kube rev 295ec4f6, failed; build log; (nil *buildlet.Client), 20.69877906s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-295ec4f6-rn6448b5
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-295ec4f6-rn6448b5: pod could not be created: context deadline exceeded
nacl-386-kube rev 733bf6ef, failed; build log; (nil *buildlet.Client), 20.690208359s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-733bf6ef-rn149fcb
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-733bf6ef-rn149fcb: pod could not be created: context deadline exceeded
nacl-386-kube rev 038b8139, failed; build log; (nil *buildlet.Client), 20.678239844s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-038b8139-rn77f60f
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-038b8139-rn77f60f: pod could not be created: context deadline exceeded
nacl-386-kube rev b4c9d01d, failed; build log; (nil *buildlet.Client), 20.683436967s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-b4c9d01d-rn561764
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-b4c9d01d-rn561764: pod could not be created: context deadline exceeded
nacl-386-kube rev 7b773946, failed; build log; (nil *buildlet.Client), 20.675252562s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-7b773946-rn84d793
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-7b773946-rn84d793: pod could not be created: context deadline exceeded
nacl-386-kube rev bdc3db0b, failed; build log; (nil *buildlet.Client), 20.677991588s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-bdc3db0b-rn18e783
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-bdc3db0b-rn18e783: pod could not be created: context deadline exceeded
nacl-386-kube rev 2c12b817, failed; build log; (nil *buildlet.Client), 20.586275176s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-2c12b817-rnc51eae
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-2c12b817-rnc51eae: pod could not be created: context deadline exceeded
nacl-386-kube rev aff6aa0a, failed; build log; (nil *buildlet.Client), 20.706595846s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-aff6aa0a-rn2f4690
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-aff6aa0a-rn2f4690: pod could not be created: context deadline exceeded
nacl-386-kube rev 7037c15e, failed; build log; (nil *buildlet.Client), 20.694223471s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-7037c15e-rn7b65d2
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-7037c15e-rn7b65d2: pod could not be created: context deadline exceeded
nacl-386-kube rev d3e61747, failed; build log; (nil *buildlet.Client), 20.683434252s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-d3e61747-rnf08c06
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-d3e61747-rnf08c06: pod could not be created: context deadline exceeded
nacl-386-kube rev 84374524, failed; build log; (nil *buildlet.Client), 20.571862556s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-84374524-rn837a97
 +120.5s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-84374524-rn837a97: pod could not be created: context deadline exceeded
nacl-386-kube rev 1d901f55, failed; build log; (nil *buildlet.Client), 20.669141752s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-1d901f55-rn17bad9
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-1d901f55-rn17bad9: pod could not be created: context deadline exceeded
nacl-386-kube rev 73d590b4, failed; build log; (nil *buildlet.Client), 20.693600723s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-73d590b4-rn964cab
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-73d590b4-rn964cab: pod could not be created: context deadline exceeded
nacl-386-kube rev eb3b1830, failed; build log; (nil *buildlet.Client), 20.599035175s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-eb3b1830-rn344c59
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-eb3b1830-rn344c59: pod could not be created: context deadline exceeded
nacl-386-kube rev 9202e9e1, failed; build log; (nil *buildlet.Client), 20.675331416s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-9202e9e1-rn69e55c
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-9202e9e1-rn69e55c: pod could not be created: context deadline exceeded
nacl-386-kube rev 4ec2fd3e, failed; build log; (nil *buildlet.Client), 20.67648059s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-4ec2fd3e-rnf5309e
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-4ec2fd3e-rnf5309e: pod could not be created: context deadline exceeded
nacl-386-kube rev 313fd1cb, failed; build log; (nil *buildlet.Client), 20.70667972s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-313fd1cb-rne65546
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-313fd1cb-rne65546: pod could not be created: context deadline exceeded
nacl-386-kube rev d9fdbf48, failed; build log; (nil *buildlet.Client), 20.663000933s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-386-kube-d9fdbf48-rn4686d6
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-386-kube-d9fdbf48-rn4686d6: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev e6347659, failed; build log; (nil *buildlet.Client), 20.572046073s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-e6347659-rn4ec3d0
 +120.5s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-e6347659-rn4ec3d0: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev 830143fa, failed; build log; (nil *buildlet.Client), 20.569240824s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-830143fa-rne7494a
 +120.5s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-830143fa-rne7494a: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev 1aab7b96, failed; build log; (nil *buildlet.Client), 20.573676927s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-1aab7b96-rncd0ceb
 +120.5s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-1aab7b96-rncd0ceb: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev 7037c15e, failed; build log; (nil *buildlet.Client), 20.706551761s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-7037c15e-rn1b2bfb
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-7037c15e-rn1b2bfb: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev eb3b1830, failed; build log; (nil *buildlet.Client), 20.690176819s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-eb3b1830-rnd97032
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-eb3b1830-rnd97032: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev 09940b92, failed; build log; (nil *buildlet.Client), 20.570622112s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-09940b92-rn66f635
 +120.5s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-09940b92-rn66f635: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev 9202e9e1, failed; build log; (nil *buildlet.Client), 20.671254596s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-9202e9e1-rn57e33a
 +120.4s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-9202e9e1-rn57e33a: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev 4ec2fd3e, failed; build log; (nil *buildlet.Client), 20.573981914s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-4ec2fd3e-rn00f8ff
 +120.5s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-4ec2fd3e-rn00f8ff: pod could not be created: context deadline exceeded
nacl-amd64p32-kube rev 26397f13, failed; build log; (nil *buildlet.Client), 20.694358569s ago
         2016-01-27T07:19:19Z get_buildlet 
   +0.0s 2016-01-27T07:19:19Z creating_kube_pod buildlet-nacl-amd64p32-kube-26397f13-rn5b36c3
 +120.3s 2016-01-27T07:21:19Z kube_buildlet_create_failure buildlet-nacl-amd64p32-kube-26397f13-rn5b36c3: pod could not be created: context deadline exceeded

And:

Kubernetes pool capacity: 120/8 CPUs; 120000000Ki/30950376Ki Memory
buildlet-nacl-386-kube-08594ac7-rn2823b4, 6m6.269111817s
buildlet-nacl-amd64p32-kube-1d901f55-rnaae567, 6m5.507266581s
buildlet-nacl-386-kube-26397f13-rnd8fb8c, 6m4.532655661s

120/8 CPUs??

@bradfitz
Copy link
Contributor Author

Disabled Kubernetes for now.

@bradfitz bradfitz mentioned this issue Jan 27, 2016
Closed
@evandbrown
Copy link

You did systemctl restart gobuild.service to deploy the new coordinator?

@bradfitz
Copy link
Contributor Author

Yup, as always:

$ git sync (etc)
$ make upload
$ ./prod-ssh.sh
$ sudo systemctl restart gobuild.service

@bradfitz
Copy link
Contributor Author

Gotta sleep. Early meeting. We can tackle this tomorrow.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18990 mentions this issue.

@gopherbot
Copy link
Contributor

CL https://golang.org/cl/18984 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Jan 27, 2016
@evandbrown
kubernetes: improve error message if pod create fails
cfb4f34 
Updates golang/go#14112

Change-Id: I53e42647e5a8b22cfb70b78fc4571ff78dd23289
Reviewed-on: https://go-review.googlesource.com/18984
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Feb 1, 2016
@bradfitz
dashboard: disable Kubernetes builders
1b6633d 
Updates golang/go#14112

Change-Id: Ie45029e27d8d8ce421135cc0e38f7019965bfa34
Reviewed-on: https://go-review.googlesource.com/18990
Reviewed-by: Andrew Gerrand <adg@golang.org>
@evandbrown
Copy link

Builds have stabilized over the weekend after a few changes. OK to close this, @bradfitz?

@bradfitz
Copy link
Contributor Author

Yup!

@golang golang locked and limited conversation to collaborators Apr 12, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@bradfitz @evandbrown @gopherbot