runtime: unexpected failure and garbage data when c-shared Go DLL on Windows receives CtrlBreak signal concurrently with runtime initialization #46354
Comments
mknyszek
added
help wanted
NeedsInvestigation
|
I have seen this error message followed by garbage in the terminal at a higher rate than reported in the above description with the following steps on a Windows desktop with go1.17beta1:
Potentially even more interesting: If the build fails with the "signal received on a thread not created by Go" message, rerunning the same build seems to crash the same way every subsequent time without typing CTRL-C at all. Running Some example output: |
|
Fascinating! Out of curiosity, does https://golang.org/cl/321789 help at all? I assume not, but it might. |
@ChrisHines I cannot reproduce your problem: I noticed that your output has many Also do try CL 321789 as per @mknyszek #46354 (comment) Thank you. Alex |
|
I was able to reproduce my above scenario with a version of Go built with After checking out So I was able to reproduce similar |
|
That definitely seems like an improvement. I will land my CL. |
|
Change https://golang.org/cl/321789 mentions this issue: |
|
@ChrisHines It's actually kind of weird that the "bad signal" message isn't showing up anymore. That CL just makes it so that |
|
Oh... I wonder if actually exiting harder in That's very odd. Consider the situation where you press Ctrl+C that lands during runtime initialization. Even the default console control handler for Ctrl+C shouldn't land you in the exception handler. I can't find any documentation about how these two might interact. |
That is not enough details for me to try to reproduce your issue. I need to know whatever commands you executed to have your environment setup and built. For example, I use go1.4.3 to bootstap my build of Go, but if I try to build with Perhaps I can still use So just give me your exact instructions, if you want me to reproduce.
I am confused about when you see The 0xc000013a is STATUS_CONTROL_C_EXIT. STATUS_CONTROL_C_EXIT is what default Windows ctrl-break handler returns. See 0xc000013a is also related to CL 243597. Before CL 243597 we would return exit code 2 when ctrl+c goes unhandled. After CL 243597 we do return 0xc000013a in the same situation. I also noticed that 0xc000013a is always happens with compile.exe in your output. Perhaps go.exe have ctrl+c handler installed, but compile.exe does not. Perhaps we can/should fix that.
I agree. If Ctrl+c happens before Go runtime calls SetConsoleCtrlHandler, the signal should endup in default Windows ctrl-c handler (that calls ExitProcess(STATUS_CONTROL_C_EXIT). If signal happens after SetConsoleCtrlHandler returns, the signal should be handled by Go ctrl-c handler. Alex |
|
Here is the sequence I used to build in my last post: The Go repo is cloned at I have installed go1.16.3 with the Windows installer downloaded from https://golang.org/dl/. That builds with CL 321789. I used the same steps to build go1.17beta1 after checking out that branch. I don't know if it matters, but here is a little info about the machine I'm running this on:
My first comment on this issue also has the |
Thanks for instructions. I tried to reproduce your setup. I still cannot see Hopefully I did not muck something up. I tried waiting for a few seconds before hitting ctrl+c. I also watched Tasj Manager to display some compile.exe processes before hitting ctrl+c. like this
Still no luck. Let me know, if you see somthing differenet with my setup. I also tried inserting
I don't see how that is relevant. But here is my
Indeed. Your original comment did have
That is to be expected. CL 321789 fixes memory corruption. But I would like to see 0xc000013a exit code returned before I decided how to treat it. I would like to understand why 0xc000013a exit code is returned. Similarly, I don't understand how memory corruption happens, that you see before CL 321789, and not after. What CL 321789 does it exits immediately when Go runtme tries to run Windows exception handler on a thread without g configured. Normally process with unhandled exception should print stack trace. But printing cannot be done without g. Alex |
|
I don't notice any difference in your setup other than the CPU. I don't know if that matters. I do not know the code paths involved here to provide any suggestions. I just happen to have encountered the behavior, found this issue that seems to have similar symptoms, and reported what I've seen. I am happy to run additional tests if needed, but I don't know what else to say at this point. |
Same. I don't think CPU matters here.
Thanks for trying to help. Perhaps this information will be helpful in the future. Unfortunately I don't see what else we can try here. Let's live it for now. Alex |
|
The error seems slightly different than for Ctrl-Break. The message originates from the go/src/runtime/signal_windows.go Lines 281 to 282 in 051df0d Windows signal implementation is documented as incomplete. Lines 19 to 21 in 051df0d It seems that some luck is required to get to interrupt the runtime itself. |
|
Change https://golang.org/cl/367495 mentions this issue: |
|
Change https://go.dev/cl/409294 mentions this issue: |
gopherbot
added
the
compiler/runtime
In
c-sharedmode, when a Go DLL is loaded, it spawns a new thread that starts runtime initialization. All calls into the DLL block on runtime initialization completing.While investigating #45638, we discovered that the previous iteration of
TestLibraryCtrlHandlerwould often send a CtrlBreak signal to the Go test process before runtime initialization was complete, and this would result in an exception landing in a Go exception handler on a Go thread that wasn't ready to receive it (no G or M).Then, we'd end up in
badsignal2. Previously,badsignal2would not exit after printing the message, and garbage would spew into the stdout (or Windows equivalent) of the process beforebadsignal2would be called again. Still, this garbage spewing was reproducible after this was fixed.As an example, this is what the failure looked like:
Sometimes there would be a lot more garbage (spanning pages and pages of terminal buffer space). Sometimes less.
The failure is rare, maybe once in 30,000 runs. It's still unclear what the root cause is, but we've narrowed it down to the control handler getting installed in the Go runtime. If that is removed, it is not reproducible. It only started showing up in Go 1.17, during which time the way our console control handler worked changed. It's unclear how the exception handler gets invoked, also, because the control handler is generally completely separate from the exception handler on Windows.
Something may be going wrong on a Go-owned thread that hasn't been set up to have an M yet (i.e. between when the first M is properly initialized and when the control handler is installed), however it's still possible to reproduce the issue even if control handler installation is delayed all the way until just before
runtime.maincompletes (signalling that runtime initialization is done).However, it's possible this issue is reproducible under previous Go releases (just that it's more difficult to do so). Notably, it's still possible to reproduce the issue even if one installs a trivial handler written in assembly (much less complex than
callbackasm1) without any of thecompileCallbackmachinery that was added in Go 1.17.The workaround for this issue is to block the application until the Go runtime has fully initialized by calling any Go function in the DLL before proceeding.
CC @alexbrainman @zx2c4
The text was updated successfully, but these errors were encountered: