Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: android builds stuck in waiting_for_machine #48969

Closed
findleyr opened this issue Oct 14, 2021 · 7 comments
Closed

build: android builds stuck in waiting_for_machine #48969

findleyr opened this issue Oct 14, 2021 · 7 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@findleyr
Copy link
Member

findleyr commented Oct 14, 2021

It looks like both android builders are stuck [EDIT: the two emulators for x/tools: amd64-emu and 386-emu]. Not seeing any recent builds at https://build.golang.org. Example:

  builder: android-amd64-emu
      rev: 9e4dc6f37f75840d1193accae697a0e9283f5384
 buildlet: (nil *buildlet.Client)
  started: 2021-10-14 04:26:40.152387307 +0000 UTC m=+35583.719633009
   status: still running

Events:
  2021-10-14T04:26:40Z checking_for_snapshot 
  2021-10-14T04:26:40Z finish_checking_for_snapshot after 22.8ms
  2021-10-14T04:26:40Z get_buildlet 
 +34396.2s (now)

Build log:


(buildlet still starting; no live streaming. reload manually to see status)

CC @golang/release

@heschi
Copy link
Contributor

heschi commented Oct 14, 2021

Fallout from https://golang.org/cl/354757. cc @toothrot

2021-10-14 10:14:27.656 EDT2021/10/14 14:14:27 Failed to create VM for host-android-amd64-emu at us-central1-a: Failed to create instance: googleapi: Error 400: Setting minimum CPU platform is not supported for the selected machine type e2-standard-4., badRequest

https://cloud.google.com/compute/docs/general-purpose-machines#e2_machine_types says: "E2 VMs do not support GPUs, local SSDs, sole-tenant nodes, or nested virtualization."

Rolling back for now.

@toothrot toothrot self-assigned this Oct 15, 2021
@toothrot
Copy link
Contributor

toothrot commented Oct 15, 2021

N2 should support nested virtualization, which we need on the android builders for hardware acceleration.

The error @heschi posted is also a problem for a different builder, host-openbsd-amd64-64, which is the only one which sets MinCpuPlatform. I don't think we need to set that anymore, as #29223 is probably no longer an issue on newer VMs.

@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 15, 2021
@toothrot toothrot added this to the Backlog milestone Oct 15, 2021
@toothrot toothrot added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 15, 2021
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 15, 2021
@toothrot
Copy link
Contributor

toothrot commented Oct 15, 2021

ok, I am confused. sorry. It looks like we set that when we ask for nested virtualization, and possibly also Intel Haswell.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/356071 mentions this issue: dashboard: use n2 instances where necessary

gopherbot pushed a commit to golang/build that referenced this issue Oct 18, 2021
e2 and n2d instances do not support specifying a MinCPUVersion or
NestedVirt.

For golang/go#48969

Change-Id: I0c4b359b29cd24302b869748014f848a9b7b5e39
Reviewed-on: https://go-review.googlesource.com/c/build/+/356071
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/356629 mentions this issue: buildlet: update MinCPUPlatform for NestedVirt hosts

gopherbot pushed a commit to golang/build that referenced this issue Oct 18, 2021
The n2 machine types that we now use for instances are either Ice Lake
or Cascade Lake. The error message we received when creating a VM was
specifically to use Cascade Lake, not the older Haswell.

For golang/go#48969

Change-Id: I1c05679c2b512ca3bb7dbb0d2efa6f587305f6aa
Reviewed-on: https://go-review.googlesource.com/c/build/+/356629
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/356630 mentions this issue: buildlet: correct MinCPUPlatform name

gopherbot pushed a commit to golang/build that referenced this issue Oct 18, 2021
Also, specify a valid CPU Platform for n2 VMs for OpenBSD.

See: https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform#availablezones

For golang/go#48969

Change-Id: Id503227ad0fbb844e9eaa923b8b3ac7da591ee76
Reviewed-on: https://go-review.googlesource.com/c/build/+/356630
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@toothrot
Copy link
Contributor

This should be resolved now, along with some quota increases.

@heschi heschi moved this to Done in Go Release Sep 27, 2022
@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
Archived in project
Development

No branches or pull requests

4 participants