Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: add a Windows ARM builder #47019

Open
toothrot opened this issue Jul 1, 2021 · 23 comments
Open

x/build: add a Windows ARM builder #47019

toothrot opened this issue Jul 1, 2021 · 23 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder OS-Windows
Milestone

Comments

@toothrot
Copy link
Contributor

toothrot commented Jul 1, 2021

Our current builder (generously contributed by @zx2c4) is ready to be turned down. We need additional hardware to keep supporting this platform, and can re-use the infrastructure from #42604.

@toothrot toothrot added the NeedsFix The path to resolution is known, but the work has not been done. label Jul 1, 2021
@toothrot toothrot added this to the Unreleased milestone Jul 1, 2021
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jul 1, 2021
@heschi heschi added this to In Progress in Go Release Team Jul 13, 2021
@toothrot toothrot self-assigned this Jul 20, 2021
@gopherbot
Copy link

Change https://golang.org/cl/344990 mentions this issue: dashboard: add windows-arm-10 builder

gopherbot pushed a commit to golang/build that referenced this issue Aug 25, 2021
This adds a windows-arm builder using host-windows-arm64-mini hosts. The
new BuildConfig has a KnownIssue until we can verify that the builders
are building correctly.

For golang/go#47019

Change-Id: I0505d4dc367dbb687290facbddc718eb13567025
Reviewed-on: https://go-review.googlesource.com/c/build/+/344990
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/345130 mentions this issue: dashboard: fix windows-arm-10 build failure

gopherbot pushed a commit to golang/build that referenced this issue Aug 25, 2021
The build is failing with 0xc000007b, which is an architecture mismatch.
Instead, just set GOARM=7, which is aligned with the windows-arm-zx2c4
builder we are replacing.

For golang/go#47019

Change-Id: I1f2aa18f6cdeb0dca4b234a164b915f2148f3936
Reviewed-on: https://go-review.googlesource.com/c/build/+/345130
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@toothrot
Copy link
Contributor Author

We're running into a 0xc000007b failure when attempting to run arm32 binaries in our arm64 windows installation. It could be either the version of windows we have installed, or a strange interaction with QEMU and HVM.

-e GOARM=7 -e GOARCH=arm -e GOHOSTARCH=arm:

go run ./cmd/gomote run -debug -e GOARM=7 -e GOARCH=arm -e GOHOSTARCH=arm  user-rakoczy-windows-arm-10-0 src/make.bat
:: Running C:\workdir\src\make.bat with args ["C:\\workdir\\src\\make.bat"] and env ["GOBUILDEXIT=1" "=C:=C:\\golang" "ALLUSERSPROFILE=C:\\ProgramData" "APPDATA=C:\\Users\\gopher\\AppData\\Roaming" "CommonProgramFiles=C:\\Program Files\\Common Files" "CommonProgramFiles(Arm)=C:\\Program Files (Arm)\\Common Files" "CommonProgramFiles(x86)=C:\\Program Files (x86)\\Common Files" "CommonProgramW6432=C:\\Program Files\\Common Files" "COMPUTERNAME=DESKTOP-0285N7P" "ComSpec=C:\\WINDOWS\\system32\\cmd.exe" "DriverData=C:\\Windows\\System32\\Drivers\\DriverData" "FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer" "FPS_BROWSER_USER_PROFILE_STRING=Default" "GOROOT_BOOTSTRAP=C:\\workdir\\go1.4" "GO_BUILDER_ENV=host-windows-arm64-mini" "HOMEDRIVE=C:" "HOMEPATH=\\Users\\gopher" "LOCALAPPDATA=C:\\Users\\gopher\\AppData\\Local" "LOGONSERVER=\\\\DESKTOP-0285N7P" "NUMBER_OF_PROCESSORS=8" "OneDrive=C:\\Users\\gopher\\OneDrive" "OS=Windows_NT" "PATH=C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\gopher\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\godep\\gcc64\\bin" "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC" "PROCESSOR_ARCHITECTURE=ARM64" "PROCESSOR_IDENTIFIER=ARMv8 (64-bit) Family 8 Model 51 Revision   0, QEMU" "PROCESSOR_LEVEL=81" "PROCESSOR_REVISION=0000" "ProgramData=C:\\ProgramData" "ProgramFiles=C:\\Program Files" "ProgramFiles(Arm)=C:\\Program Files (Arm)" "ProgramFiles(x86)=C:\\Program Files (x86)" "ProgramW6432=C:\\Program Files" "PROMPT=$P$G" "PSModulePath=C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules" "PUBLIC=C:\\Users\\Public" "SESSIONNAME=Console" "SystemDrive=C:" "SystemRoot=C:\\WINDOWS" "TEMP=C:\\Users\\gopher\\AppData\\Local\\Temp" "TMP=C:\\Users\\gopher\\AppData\\Local\\Temp" "USERDOMAIN=DESKTOP-0285N7P" "USERDOMAIN_ROAMINGPROFILE=DESKTOP-0285N7P" "USERNAME=gopher" "USERPROFILE=C:\\Users\\gopher" "windir=C:\\WINDOWS" "GO_STAGE0_NET_DELAY=100ms" "GO_STAGE0_DL_DELAY=700ms" "WORKDIR=C:\\workdir" "GO_BUILDER_NAME=windows-arm-10" "GOARCH=arm" "GOARM=7" "GOHOSTARCH=arm" "GO_DISABLE_OUTBOUND_NETWORK=false"] in dir C:\workdir\src

Building Go cmd/dist using C:\workdir\go1.4
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
go tool dist: FAILED: C:\workdir\pkg\tool\windows_arm\go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 0xc000007b
Error running run: exit status 1
exit status 1

-e GOARM=7 -e GOARCH=arm:

go run ./cmd/gomote run -debug -e GOARM=7 -e GOARCH=arm -e "PATH=C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\gopher\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\godep\\llvm-aarch64\\bin"  user-rakoczy-windows-arm-10-0 src/all.bat
:: Running C:\workdir\src\all.bat with args ["C:\\workdir\\src\\all.bat"] and env ["GOBUILDEXIT=1" "=C:=C:\\golang" "ALLUSERSPROFILE=C:\\ProgramData" "APPDATA=C:\\Users\\gopher\\AppData\\Roaming" "CommonProgramFiles=C:\\Program Files\\Common Files" "CommonProgramFiles(Arm)=C:\\Program Files (Arm)\\Common Files" "CommonProgramFiles(x86)=C:\\Program Files (x86)\\Common Files" "CommonProgramW6432=C:\\Program Files\\Common Files" "COMPUTERNAME=DESKTOP-0285N7P" "ComSpec=C:\\WINDOWS\\system32\\cmd.exe" "DriverData=C:\\Windows\\System32\\Drivers\\DriverData" "FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer" "FPS_BROWSER_USER_PROFILE_STRING=Default" "GOROOT_BOOTSTRAP=C:\\workdir\\go1.4" "GO_BUILDER_ENV=host-windows-arm64-mini" "HOMEDRIVE=C:" "HOMEPATH=\\Users\\gopher" "LOCALAPPDATA=C:\\Users\\gopher\\AppData\\Local" "LOGONSERVER=\\\\DESKTOP-0285N7P" "NUMBER_OF_PROCESSORS=8" "OneDrive=C:\\Users\\gopher\\OneDrive" "OS=Windows_NT" "PATH=C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\gopher\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\godep\\gcc64\\bin" "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC" "PROCESSOR_ARCHITECTURE=ARM64" "PROCESSOR_IDENTIFIER=ARMv8 (64-bit) Family 8 Model 51 Revision   0, QEMU" "PROCESSOR_LEVEL=81" "PROCESSOR_REVISION=0000" "ProgramData=C:\\ProgramData" "ProgramFiles=C:\\Program Files" "ProgramFiles(Arm)=C:\\Program Files (Arm)" "ProgramFiles(x86)=C:\\Program Files (x86)" "ProgramW6432=C:\\Program Files" "PROMPT=$P$G" "PSModulePath=C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules" "PUBLIC=C:\\Users\\Public" "SESSIONNAME=Console" "SystemDrive=C:" "SystemRoot=C:\\WINDOWS" "TEMP=C:\\Users\\gopher\\AppData\\Local\\Temp" "TMP=C:\\Users\\gopher\\AppData\\Local\\Temp" "USERDOMAIN=DESKTOP-0285N7P" "USERDOMAIN_ROAMINGPROFILE=DESKTOP-0285N7P" "USERNAME=gopher" "USERPROFILE=C:\\Users\\gopher" "windir=C:\\WINDOWS" "GO_STAGE0_NET_DELAY=100ms" "GO_STAGE0_DL_DELAY=700ms" "WORKDIR=C:\\workdir" "GO_BUILDER_NAME=windows-arm-10" "GOARCH=arm" "GOARM=7" "PATH=C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\gopher\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\godep\\llvm-aarch64\\bin" "GO_DISABLE_OUTBOUND_NETWORK=false"] in dir C:\workdir\src

Building Go cmd/dist using C:\workdir\go1.4
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, windows/arm64.
Building packages and commands for target, windows/arm.


##### Testing packages.
exit status 0xc000007b
FAIL	archive/tar	0.042s
exit status 0xc000007b
FAIL	archive/zip	0.047s
exit status 0xc000007b
FAIL	bufio	0.064s
exit status 0xc000007b
FAIL	bytes	0.064s
exit status 0xc000007b
FAIL	compress/bzip2	0.068s
exit status 0xc000007b
FAIL	compress/flate	0.075s
exit status 0xc000007b

@toothrot toothrot moved this from In Progress to Planned in Go Release Team Aug 26, 2021
@toothrot toothrot removed their assignment Aug 26, 2021
@gopherbot
Copy link

Change https://golang.org/cl/345330 mentions this issue: dashboard: remove windows-arm-10 builder

gopherbot pushed a commit to golang/build that referenced this issue Aug 26, 2021
Binaries are failing to run with an 0xc000007b error. The cause is
unknown. See the issue for details.

For golang/go#47019

Change-Id: Ie0145a8cbe97745b632e420466fe13f6ff35fe17
Reviewed-on: https://go-review.googlesource.com/c/build/+/345330
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
@bcmills
Copy link
Contributor

bcmills commented Nov 10, 2021

Do we have any actual ARM hardware that we could copy the image to in order to determine whether the problem is with the WIndows version or the emulated host?

@zx2c4
Copy link
Contributor

zx2c4 commented Nov 10, 2021

We're running into a 0xc000007b failure when attempting to run arm32 binaries in our arm64 windows installation. It could be either the version of windows we have installed,

The latest insider build has arm32 issues, fyi, so be sure to run ordinary stable.

Things seem fine on real hardware.

@toothrot
Copy link
Contributor Author

Unfortunately, I couldn't get the stable version to run on the VM instances we're using. As I mentioned on #49266, I'll try updating the builders in my free time this weekend.

I also couldn't reproduce on real hardware, but this seems related to the specific environment we're running on.

Do you know if Windows ARM32 will continue to be supported?

@zx2c4
Copy link
Contributor

zx2c4 commented Nov 10, 2021

The idea is to run the ARM64 build of Windows and test ARM32 inside of it. Microsoft still supports ARM32 for IoT. If you think there's a good argument for sunsetting ARM32 in Go, though, then I guess that'd be a proposal to make, and then we could get some definition on it that way.

@toothrot
Copy link
Contributor Author

We will be giving this another shot, as we're working on upgrading our windows builders this week. The holiday will slow things down a bit, but we're actually working on it.

As a bonus, @heschi is doing it, which means updates won't be blocked solely on my availability.

@zx2c4
Copy link
Contributor

zx2c4 commented Nov 25, 2021

In case it helps,

PS C:\go-src\src> $env:GOARCH='arm'
PS C:\go-src\src> $env:GOHOSTARCH='arm'
PS C:\go-src\src> $env:GOARM='7'
PS C:\go-src\src> $env:GOOS='windows'
PS C:\go-src\src> $env:GOROOT_BOOTSTRAP='C:\go-binary-arm64-release'
PS C:\go-src\src> .\all.bat

You'll need to download llvm and such for cgo, but the above should hopefully get you past the, "it doesn't work at all!" phase, as the above does in fact work, and you can see ARM32 processes in the details view of task manager.

Screenshot_20211125-005204

@zx2c4
Copy link
Contributor

zx2c4 commented Nov 25, 2021

@dmitshur Can we make this issue a goal for before 1.18 beta1? That way we're certain to have the beta1 period to fix bugs that might have slipped in since we went builderless a long while back, and therefore don't regress for 1.18.0.

@heschi heschi self-assigned this Dec 6, 2021
@heschi
Copy link
Contributor

heschi commented Dec 6, 2021

I'm actively working in this area, but it's a very complicated situation and I'm not confident it'll be done before beta.

@zx2c4
Copy link
Contributor

zx2c4 commented Dec 6, 2021

Anything I can do to help? What sorts of complications are you facing?

@heschi
Copy link
Contributor

heschi commented Dec 9, 2021

The goal is to do this as part of #48946, but that's proving difficult.

If you were going to work some sort of miracle, getting Windows 11 working on an AWS ARM machine would be my preference.

@zx2c4
Copy link
Contributor

zx2c4 commented Dec 9, 2021

You might have luck running it under QEMU inside of Linux, on an AWS ARM instance. And then capping the number of CPUs to 4 or 8 or something smallish.

@heschi
Copy link
Contributor

heschi commented Dec 9, 2021

Yeah, we've had some minimal success with that, but with anything more than 4 cores it greenscreens. Even with 4 I think we've seen some instability. See https://cs.opensource.google/go/x/build/+/master:env/windows-arm64/aws/. I just finished verifying that the situation isn't any better with a recent Windows 11 and qemu.

@zx2c4
Copy link
Contributor

zx2c4 commented Dec 9, 2021

Huh, I wonder if that instability still holds on the Windows 11 stable channel images.

@heschi
Copy link
Contributor

heschi commented Dec 10, 2021

I was trying with 22504; given that we've seen the same failures with a 6-month build and a fresh one, I don't think there's much hope for stable to be better :(

@gopherbot
Copy link

Change https://golang.org/cl/377696 mentions this issue: all: add Windows 11 builder

gopherbot pushed a commit to golang/build that referenced this issue Jan 12, 2022
Add a new host type, host-windows11-arm64-mini. It should support both
ARM64 and ARM builds but for starters let's focus on ARM64.

For golang/go#48946, golang/go#47019.

Change-Id: I7ff07a97661fb6621237b9801ff0d0e338c6d4f4
Reviewed-on: https://go-review.googlesource.com/c/build/+/377696
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Alex Rakoczy <alex@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@heschi
Copy link
Contributor

heschi commented Feb 8, 2022

@zx2c4 I finally, finally got to work on this and immediately faceplanted. Running anything with GOARCH=arm gives me This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.

We're running the latest version of Windows 11, not IOT. I can't find a clear answer to whether that's supposed to support ARM32 binaries. Do you know what's going on here?

@gopherbot
Copy link

Change https://go.dev/cl/391136 mentions this issue: dashboard: update expectations for windows-arm64 builders

gopherbot pushed a commit to golang/build that referenced this issue Mar 10, 2022
Update the expected number of reverse buildlets to describe
the current state more accurately.

Update the windows-arm64 alias to point to windows-arm64-11,
since it's more available and used for most recent releases.

For golang/go#51546.
Updates golang/go#47019.

Change-Id: I5ae1a764efd21da4155ab002f20ad974b098b023
Reviewed-on: https://go-review.googlesource.com/c/build/+/391136
Reviewed-by: Carlos Amedee <carlos@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
@heschi
Copy link
Contributor

heschi commented Mar 17, 2022

@zx2c4 any ideas?

@heschi heschi removed their assignment Aug 12, 2022
@gopherbot
Copy link

Change https://go.dev/cl/460295 mentions this issue: all: remove macstadium integration and monitoring

gopherbot pushed a commit to golang/build that referenced this issue Jan 3, 2023
We no longer use this hosting provider.

Removes builders depending on this host type, including Windows ARM
builders.

Fixes golang/go#57562
Updates golang/go#47019

Change-Id: I1b2d3dfb0540cdf1ebc1ce2cfabbcd7d56c30145
Reviewed-on: https://go-review.googlesource.com/c/build/+/460295
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Jenny Rakoczy <jenny@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder OS-Windows
Projects
Status: Planned
Development

No branches or pull requests

6 participants