Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: crash/hang on startup in buildmode=c-shared amd64 DLL when run on ARM64 windows 11. #60996

Open
oddy opened this issue Jun 26, 2023 · 6 comments
Assignees
Labels
arch-arm64 compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Milestone

Comments

@oddy
Copy link

oddy commented Jun 26, 2023

What version of Go are you using (go version)?

$ go version
go version go1.20 windows/amd64

Does this issue reproduce with the latest release?

Yes. Confirmed behavior with go1.21rc2 also.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\bb\AppData\Local\go-build
set GOENV=C:\Users\bb\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\bb\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\bb\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go120
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go120\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=D:\work\go\minimal_cfguard_test\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\bb\AppData\Local\Temp\go-build1785003883=/tmp/go-build -gno-record-gcc-switches
gdb --version: GNU gdb (GDB) 8.1

What did you do?

  • Behaviour exhibits reliably when running as amd64 DLL in either C or Python host EXE on windows ARM machines.
  • Crash/Hang is intermittent but can be reliably triggered in between 5-20 repeated runs of the exe.
  • Trigger condition tracked down to the host exe having windows anti-exploit measure "Control Flow Guard" enabled.

Pre-reqs:

  • Windows 11 ARM machine (vm or baremetal)
  • Go 1.20 or 1.21rc2 installed
  • Visual C Build Tools 2019 (or 2020) installed (for the caller exe)
  • MingGW 8.10 or TDM-GCC latest (for Go DLL compiling)

Steps to reproduce as follows:

mindll.go

package main

import "C"
import "fmt"

//export Fred
func Fred() {
	fmt.Println("go: MINDLL: i am fred")
}

func main() { }

compile it

go mod init
go mod init mindll
go mod tidy
go build -buildmode=c-shared -o mindll.dll .

c_call.c

#include <windows.h>
#include <stdio.h>
typedef void (*fred_func)(void);
int main() {
	HINSTANCE hinstLib = LoadLibrary(TEXT("mindll.dll"));
	fred_func Fred;
	Fred = (fred_func)GetProcAddress(hinstLib, "Fred");
	printf("Calling Fred\n");  fflush(stdout);
	Fred();
	printf("Back from Fred\n");
    return 0;
}

Compile it (no CFGuard)

"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
cl c_call.c -o ccall_nocfg.exe -Wall

ccall_nocfg.bat

ccall_nocfg.exe
ccall_nocfg.bat

Run ccall_nocfg.bat for a while, runs fine, no issues.

Compile caller with CFGuard on

cl c_call.c -o ccall_withcfg.exe -Wall /guard:cf

ccall_withcfg.bat

ccall_withcfg.exe
ccall_withcfg.bat

Run ccall_withcfg.bat, process crashes/hangs quite quickly, anywhere from 5th to 20th run. Then keeps crashing throughout, every 5-20 runs or so.

What did you expect to see?

The C process and Go DLL running reliably 100%

What did you see instead?

The process hanging/crashing after LoadLibrary and/or at first DLL function call (Fred) every 5-20 runs.

Workarounds (& why they are workarounds)

  • "Compile everything for ARM" - Don't have control of the python exe 'mothership'. Viable arm python has only recently appeared with 3.11 but we must support back to py 3.7 for windows 7. Also many python libraries not fully available native for arm windows yet.
  • "Compile caller without Control Flow Guard" - as above + unhappy security auditors
  • "Have the end users disable Control Flow Guard for this process (or system-wide) in windows control panel for ARM windows machines" - very unhappy security auditors. Bad reputational for product.

Notes

  • This behavior does not happen when a minimal C-DLL is substituted for the Go DLL above, regardless of whether or not the test C DLL has control flow guard enabled or disabled.

  • There is only one Go runtime in the process as the 'mothership' is either a python or C exe.

  • I am not freeing or attempting to unload the Go DLL at all.

  • The process briefly hangs then suddenly exits and no errors, stacktraces or anything are printed. Also nothing in the Windows event log, which seems to imply that Go is eating the exception and calling ExitProcess?

  • I see support for CFG itself IN go was deemed not-useful here cmd/compile: add control flow integrity options #35940 however interactions with CFG when GO is a DLL do possibly still need to be addressed.

  • (Speculation) perhaps this issue is similar to that of runtime: TLS slot index over 64 and crash #59213 however this issue is still present in Go 1.21rc2

@cagedmantis cagedmantis changed the title Crash/hang on startup in buildmode=c-shared amd64 DLL when run on ARM windows 11. cmd/compile: crash/hang on startup in buildmode=c-shared amd64 DLL when run on ARM windows 11. Jun 26, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jun 26, 2023
@cagedmantis cagedmantis added this to the Backlog milestone Jun 26, 2023
@qmuntal
Copy link
Contributor

qmuntal commented Jun 26, 2023

Thanks for reporting this issue.

I'm afraid I can't reproduce it (I made sure CFG is enabled). Could it be that you are using an older gcc toolchain? I use mingw-w64 12.2.0:

gcc.exe (MinGW-W64 x86_64-ucrt-posix-seh, built by Brecht Sanders) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

@cagedmantis cagedmantis changed the title cmd/compile: crash/hang on startup in buildmode=c-shared amd64 DLL when run on ARM windows 11. runtime: crash/hang on startup in buildmode=c-shared amd64 DLL when run on ARM windows 11. Jun 26, 2023
@oddy
Copy link
Author

oddy commented Jun 26, 2023

Hi Quim, thanks, i appreciate your time and effort :)

Just double-checking you're trying to repro on an ARM windows machine?

I will try mingw-w64 12.2.0 also and report back.

@oddy
Copy link
Author

oddy commented Jun 27, 2023

Ok i tried with mingGW gcc 12.2.0 also, both the UCRT and MSVCRT versions, behaviour remains unchanged.

bat-file-calling-go-dll

@qmuntal
Copy link
Contributor

qmuntal commented Jun 27, 2023

Just double-checking you're trying to repro on an ARM windows machine?

Huh, I tried to repro on an ARM64 windows machine, not ARM. Can't help here, as I don't have one.

The windows/arm port is not in a good state given that it hasn't had a builder for years. See #57960.

@qmuntal qmuntal added the arch-arm Issues solely affecting the 32-bit arm architecture. label Jun 27, 2023
@oddy
Copy link
Author

oddy commented Jun 27, 2023

Sorry, arm64 is what i meant. Apologies for confusion.

@qmuntal qmuntal changed the title runtime: crash/hang on startup in buildmode=c-shared amd64 DLL when run on ARM windows 11. runtime: crash/hang on startup in buildmode=c-shared amd64 DLL when run on ARM64 windows 11. Jun 27, 2023
@qmuntal qmuntal added arch-arm64 and removed arch-arm Issues solely affecting the 32-bit arm architecture. labels Jun 27, 2023
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 28, 2023
@qmuntal
Copy link
Contributor

qmuntal commented Jun 28, 2023

Sorry, arm64 is what i meant. Apologies for confusion.

Oh, ok. Still can't repro it. Could it be that you are using an older C toolchain? I used Visual C Build Tools 2022:

cl.exe
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30151 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-arm64 compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Projects
Development

No branches or pull requests

4 participants