-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: add Amazon EC2 ARM instances for builders #36841
Comments
See also #36170, for discussion of some of our requirements on ARM builders. We still have work to do there, but it might be worth taking care of it all at once. |
@toothrot, if we did this bug, that bug would be not applicable. We'd run each buildlet in its own VM (like we do on GCE, except using AWS), not using Docker (like Packet & Scaleway & PPC{,64}, etc). We now have quota from AWS to run this for free, so it's just a matter of writing an AWSBuildletPool implementation. |
I understand that, but I wanted to make sure some of the other requirements were still being paid attention to, like creating a highcpu builder. |
Change https://golang.org/cl/226843 mentions this issue: |
This is the first in a series of patches which include an effort to refactor how the cmd/coordinator is structured. In order to move buildlet pools into their own package, it would facilitate the move to have spanlog in it's own internal package. This CL will be followed by one which removes cmd/coordinator/spanlog and changes the path of the package for all the callers. Updates golang/go#36841 Change-Id: Ic310a38f807a3099815e0bdb988edc42b2ca5c85 Reviewed-on: https://go-review.googlesource.com/c/build/+/226843 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
Change https://golang.org/cl/227023 mentions this issue: |
This removes the spanlog package located inside the coordinator package. The spanlog package has already been copied over to the internal package. All imports in the build repository have been changes to point to the internal spanlog package. Updates golang/go#36841 Change-Id: I84499bcfe782c566b12dabcbfa743cd8ce559f18 Reviewed-on: https://go-review.googlesource.com/c/build/+/227023 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Change https://golang.org/cl/227141 mentions this issue: |
Change https://golang.org/cl/227578 mentions this issue: |
Change https://golang.org/cl/227769 mentions this issue: |
Change https://golang.org/cl/227768 mentions this issue: |
Change https://golang.org/cl/227920 mentions this issue: |
Change https://golang.org/cl/228799 mentions this issue: |
This CL creates the internal/coordinator/pool package intended to contain all buildlet pool implementations. In order to keep this change small and carefully discover where the interactions are between the gce buildlet pool and the rest of the coordinator are, this change only moves the gce buildlet over to the new package. The next steps will be to move the rest of the buildlet pools over to this package. After that we will restructure the implementations themselves in order to increase test coverage and increase the ease of testing. Updates golang/go#36841 Updates golang/go#38337 Change-Id: If82ae1b584bd77c697aa84fadf9011c9e79fa409 Reviewed-on: https://go-review.googlesource.com/c/build/+/227141 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
This is a set in a series of steps which will move everything buildlet pool related into a pool package. Updates golang/go#36841 Change-Id: I8efb1f94c7b929be559004d9f455bca0370c7800 Reviewed-on: https://go-review.googlesource.com/c/build/+/227768 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
This is a set in a series of steps which will move everything buildlet pool related into a pool package. Updates golang/go#36841 Updates golang/go#38337 Change-Id: Ic7a0ccd7838345036df2e72b13084070541cb63c Reviewed-on: https://go-review.googlesource.com/c/build/+/227769 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
This change moves all package level accessors into a struct as suggested by Alex: https://go-review.googlesource.com/c/build/+/227141/6/internal/coordinator/pool/gce.go#227 This will make it easier to move away from global state in future changes. Updates golang/go#36841 Updates golang/go#38337 Change-Id: I005884ccd206fe49651d31ef1d3d336fac9c3d5f Reviewed-on: https://go-review.googlesource.com/c/build/+/227920 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
Change https://golang.org/cl/232077 mentions this issue: |
This adds the ability for linux-arm64 builder images to be created on AWS. Instead of writing a bash script which would create an image with all of the dependencies on it, this experiments with using packer to create the image. The image is configured to install and daemonize rundockerbuildlet. Rundockerbuildlet will download and run the latest stage0 image. Stage0 then downloads the latest buildlet and begins processing jobs. A follow-up change will enable rundockerbuildlet to initialize the buildlet without setting it in reverse mode. Updates golang/go#36841 Change-Id: Iee07c2600e2b91d34f5e6a1e062f763833d79904 Reviewed-on: https://go-review.googlesource.com/c/build/+/228799 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
This change adds an AWS buildlet client which allows us to create EC2 instances on AWS. With this change we have also moved a portion of the gce creation logic into a helper function which allows multiple clients to use it. Metadata for the instances are stored in the user data fields. The creation of a buildlet pool and modifications to rundocker buildlet be made in order to enable this change. Updates golang/go#36841 Change-Id: Ice03e1520513d51a02b9d66542e00012453bf0d9 Reviewed-on: https://go-review.googlesource.com/c/build/+/232077 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
Change https://golang.org/cl/232997 mentions this issue: |
The buildlet client creation will either timeout at the current default timeout period or the timeout set in the context. The existing effective timeout is the five seconds set in the probeBuildlet function. Fixes golang/go#38956 Updates golang/go#36841 Change-Id: I09e96e2c5abcc45ccd535596104c52998ddb0d7a Reviewed-on: https://go-review.googlesource.com/c/build/+/232997 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
Change https://golang.org/cl/233800 mentions this issue: |
Change https://golang.org/cl/233798 mentions this issue: |
Change https://golang.org/cl/233801 mentions this issue: |
Change https://golang.org/cl/233799 mentions this issue: |
Change https://golang.org/cl/246639 mentions this issue: |
FreeBSD 9.3, FreeBSD 10.x, and OpenBSD 6.0 have reached end-of-life and are no longer supported. Go 1.12 was the last release¹ that ran on FreeBSD 10.x. Go 1.10 was the last release² that ran on OpenBSD 6.0. These builders have long since been disabled and won't be needed in the future, so remove them. This work helped detect golang/go#40563. Modify TestHostConfigsAllUsed to report errors instead of warnings when it finds unused host configs. Make a map of known exceptions with rationale or a linked issue for each entry. ¹ https://golang.org/doc/go1.12#ports ² https://golang.org/doc/go1.10#ports Updates golang/go#36841. Updates golang/go#19938. Change-Id: Id7fcdc671bfacf7e70174f4168c9c9cc46ba53d7 Reviewed-on: https://go-review.googlesource.com/c/build/+/246639 Reviewed-by: Alexander Rakoczy <alex@golang.org>
Change https://golang.org/cl/247400 mentions this issue: |
This change modifies the WaitUntilVMExists method and sets it to not be exported. It will never be directly called by anything outside of the EC2Client. Updates golang/go#36841 Change-Id: If6dfea8c4c191036c8fadd9ad8767e85e55cd224 Reviewed-on: https://go-review.googlesource.com/c/build/+/247400 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>
Change https://golang.org/cl/247901 mentions this issue: |
Change https://golang.org/cl/247908 mentions this issue: |
Change https://golang.org/cl/247906 mentions this issue: |
Change https://golang.org/cl/247907 mentions this issue: |
This change stops explicitly selecting a random availability zone upon EC2 vm creation if one is not set. The EC2 api will select an availability zone with capacity for the request when an availability zone is not set. The caller maintains the ability to set an availability zone if needed. Updates golang/go#36841 Change-Id: I87d4e8e66fcfadfedbe584a2c016696dc4d62062 Reviewed-on: https://go-review.googlesource.com/c/build/+/247901 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change adds a ledger to the coordinator pool package. The ledger is used to store a record of EC2 instance types created and their attributes. It also keeps a record of resource usage for EC2 instances in order to ensure that we do not exceed the resource limits for the account. Updates golang/go#36841 Change-Id: I6a0afdb31c8e3a83634e7c1fc8b2b733b7a50c01 Reviewed-on: https://go-review.googlesource.com/c/build/+/247906 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
The EC2 buildlet pool added by this commit will manage the lifecycle of buildlets running on EC2. EC2 VMs will only be created for the ARM64 architecture. As VMs are requested, the pool will ensure that there are enough resources for the VM to be created and keep track of the VMs created. Once a VM is destroyed, the resources will be made available for other VMs. This pool will only keep instances as they are needed. Updates golang/go#36841 Updates golang/go#38337 Change-Id: Ic777485c0b0a69ec13726c58b49e9fdc1df4777e Reviewed-on: https://go-review.googlesource.com/c/build/+/247907 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Change https://golang.org/cl/249118 mentions this issue: |
Change https://golang.org/cl/249121 mentions this issue: |
Before destroying an untracked EC2 instance, verify that it is a buildlet. Non buildlets will be destroyed manually as they are most likely instances created while an operator is debugging a problem. For golang/go#36841 Change-Id: Id0f65192ce4e72f9ecc495dd3e94750724091b51 Reviewed-on: https://go-review.googlesource.com/c/build/+/249121 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Add the host-linux-arm64-aws builder. The HostConfig IsVM method has been modified to support the concept that an EC2 VM can be created which runs the buildlet binary in a container. This configuration would have both a VM image and a container image set in the host configuration. For golang/go#36841 Change-Id: I092b48b1dda24d24bcd0b10dff0b2126f5f43017 Reviewed-on: https://go-review.googlesource.com/c/build/+/249118 Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Change https://golang.org/cl/252317 mentions this issue: |
When rundockerbuildlet is running on an EC2 instance, the configuration should be pulled in from the metadata. This is part of the effort to move some configuration for builders over to the coordinator. For golang/go#36841 Change-Id: Icbf0e73301faadd06c87f02aa1172bd0e6f87624 Reviewed-on: https://go-review.googlesource.com/c/build/+/252317 Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
Change https://golang.org/cl/255358 mentions this issue: |
Change https://golang.org/cl/255359 mentions this issue: |
There is no need to set availability zones for EC2 resources at this time. The resources currently in use are available to all zones within a single region. For golang/go#36841 Change-Id: Ifcb2c3404fba2489741c31fe63ad4e7747639ad9 Reviewed-on: https://go-review.googlesource.com/c/build/+/255359 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> TryBot-Result: Go Bot <gobot@golang.org>
This adds a polling function that is tested and can be reused by callers in the x/build repository. This was suggested in the review of CL 247907. The polling function replaces the pollers used in the EC2 buildlet pool. For golang/go#36841 Change-Id: I120ceb83e2740f0bdc5ee2423e0edd3ad727bf4b Reviewed-on: https://go-review.googlesource.com/c/build/+/255358 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Alexander Rakoczy <alex@golang.org>
Change https://golang.org/cl/266697 mentions this issue: |
This change updates which source ami is used as a base for the creation of the linux-arm64-aws builder. Packer will now search for the latest version of the arm64 Debian image to use as the source ami. For golang/go#36841 Change-Id: I036dbaf7ee65d4e7028cd6a0cdf6710bf898fdd2 Reviewed-on: https://go-review.googlesource.com/c/build/+/266697 Trust: Carlos Amedee <carlos@golang.org> Run-TryBot: Carlos Amedee <carlos@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
As @bradfitz noted, we should explore adding ephemeral AWS ARM instances for both 32-bit and 64-bit ARM builds.
@toothrot @dmitshur
The text was updated successfully, but these errors were encountered: