runtime: unexpected fault address 0xffffffff (windows/386) #12415

AaronO · 2015-08-31T17:07:46Z

Here's my setup:

Go version 1.5
Cross compiling from linux to windows/386
Linking (maybe not relevant):
- Static: libgit2
- Dynamic: -lwinhttp -lws2_32 -lcrypt32 -lole32 -lrpcrt4 -ladvapi32
Running on Windows 10 x64

When running this program on some Windows 10 x64 machines, this fails with the following error below (it does not fail on all machines but fails consistently and repeatedly on those where it does):

> uhub_windows_386.exe
unexpected fault address 0xffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffff pc=0x52a5bb]

goroutine 1 [running, locked to thread]:
runtime.throw(0x86df74, 0x5)
        /usr/local/go-1.5/src/runtime/panic.go:527 +0x7f fp=0x1271fe60 sp=0x1271fe54
runtime.sigpanic()
        /usr/local/go-1.5/src/runtime/signal_windows.go:164 +0x113 fp=0x1271fe78 sp=0x1271fe60
runtime.aeshashbody()
        /usr/local/go-1.5/src/runtime/asm_386.s:999 +0x1bf fp=0x1271fe7c sp=0x1271fe78
runtime.mapassign1(0x7b2f8c, 0x126f7720, 0x92958c, 0x929594)
        /usr/local/go-1.5/src/runtime/hashmap.go:424 +0x8a fp=0x1271fedc sp=0x1271fe7c
time.init()
        /usr/local/go-1.5/src/time/format.go:1151 +0x182 fp=0x1271ff1c sp=0x1271fedc
os.init()
        /usr/local/go-1.5/src/os/types_windows.go:107 +0x48 fp=0x1271ff40 sp=0x1271ff1c
fmt.init()
        /usr/local/go-1.5/src/fmt/scan.go:1190 +0x57 fp=0x1271ff78 sp=0x1271ff40
log.init()
        /usr/local/go-1.5/src/log/log.go:346 +0x48 fp=0x1271ff9c sp=0x1271ff78
main.init()
        /go/src/github.com/GitbookIO/uhub/server.go:69 +0x3e fp=0x1271ffa0 sp=0x1271ff9c
runtime.main()
        /usr/local/go-1.5/src/runtime/proc.go:100 +0x205 fp=0x1271ffc8 sp=0x1271ffa0
runtime.goexit()
        /usr/local/go-1.5/src/runtime/asm_386.s:1662 +0x1 fp=0x1271ffcc sp=0x1271ffc8

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go-1.5/src/runtime/asm_386.s:1662 +0x1

Any ideas of what could be the cause of this ? Are there any known fixes or workarounds ?

I appreciate any insight you may have, thanks !

The text was updated successfully, but these errors were encountered:

minux · 2015-08-31T17:22:52Z

Have you tried to build your binaries natively on windows? Could you reproduce this crash with simpler examples? (by simpler, I mean something that at least doesn't use unsafe or cgo.)

AaronO · 2015-08-31T17:27:52Z

@minux I'm writing a simpler test case now, I'll update here once that's done.

I haven't compiled on windows, for this specific project cross-compiling from linux is much easier (because of cmake dependencies and others). BTW the same binary cross compiled for windows/amd64 (GOOS=windows GOARCH=amd64) doesn't crash.

randall77 · 2015-08-31T17:37:13Z

Possibly an alignment issue with the linker? The failing instruction is reading a 16-byte value that needs to be at 16-byte alignment.

AaronO · 2015-09-01T16:53:24Z

@randall77 I'm cross-compiling using mingw on linux (i686-w64-mingw32).

Do you have a fix or a workaround in mind ?

This has become hard to debug on my side, a colleague of mine gave me an identical copy of his virtual machine (where the binary crashed) but when running the same binaries inside the now imported (but identical) VM it doesn't crash ... (So it's not even repeatable across VMs).

randall77 · 2015-09-01T17:22:31Z

If you can run:

go tool nm uhub_windows_386.exe | grep masks

Look at the alignment of this symbol. If it is not 16-byte aligned, that would be the problem.
It seems unlikely, but worth a try. I don't understand why it would vary with machine, unless there is ASLR or something messing with layouts. Or some machines fault on unaligned reads and some don't?

AaronO · 2015-09-01T23:26:57Z

@randall77 Here's my output:

> go tool uhub_windows_386.exe | grep masks
  92a96c T masks

So indeed, it doesn't seem to be 16-byte aligned and it's amd64 counterpart seems to be correctly aligned:

> go tool nm uhub_windows_amd64.exe | grep masks
  a6ed30 T masks

How can I change my build process to ensure that masks is always 16-byte aligned ?

randall77 · 2015-09-01T23:36:23Z

Well, at least we know what is wrong. It sounds like a genuine bug in the linker and/or our input to the linker. Are you using external linking? It sounds like it. Maybe we need an additional notation in our compiled output to convince the external linker to align sections correctly. ccing people with more linking knowledge.

@ianlancetaylor @rsc

alexbrainman · 2015-09-01T23:41:43Z

gcc would be producing executable. Go linker produces object file. But maybe if we mark .text section with IMAGE_SCN_ALIGN_16BYTES, just maybe gcc will keep appropriate address aligned too. I don't know of a way to align a particular symbol in object file. We'll need some test to reproduce this.

Alex

minux · 2015-09-01T23:42:46Z

I remember seeing problem like this while developing external linking for Windows, i will take a look. Perhaps the real problem is just masked and not fixed.

minux · 2015-09-01T23:56:21Z

Yes, the problem is because the text section is only aligned to 4 bytes, but data and bss are 32-byte aligned. And mask, being defined in assembly, is included in the text section, even though it's declared DATA.

minux · 2015-09-02T00:02:02Z

Please give CL 14166 a try and see if it solves the problem.

AaronO · 2015-09-02T00:07:21Z

@minux Thanks, giving it a try now, I'll report back shortly.

AaronO · 2015-09-02T00:17:25Z

@minux That seems to work 👍

❯ go tool nm ./build/uhub_windows_386.exe | grep masks
  92a980 T masks

I think you've found the root of the problem, so unless there's anything you need on my behalf I think we can close this issue :)

minux · 2015-09-02T00:30:13Z

How you tried the new binary on the old VM where old binary uses to crash? I want to verify that it's indeed the correct fix. Thanks!

AaronO · 2015-09-02T00:41:06Z

It's late here in Europe but I'll try it on my colleague's VM first thing tomorrow morning and I'll report back here.

Thanks for your help !

gopherbot · 2015-09-02T01:00:54Z

CL https://golang.org/cl/14166 mentions this issue.

AaronO · 2015-09-02T17:04:27Z

@minux Just wanted to confirm with you that it does indeed fix the issue.

The new binary built with your patch works while the older one built with go1.5 crashes.

Thanks for your help and work on Go 👍

minux · 2015-09-02T17:15:26Z

Thanks for the confirmation, I will see if this patch could get into Go 1.5.1.

gopherbot · 2015-09-03T07:01:47Z

CL https://golang.org/cl/14207 mentions this issue.

gopherbot · 2015-09-04T06:01:00Z

CL https://golang.org/cl/14279 mentions this issue.

… linking Some symbols, for example, masks requires 16-byte alignment, and they are placed in the text section. Before this change, the text section is only aligned to 4-byte, and it's making masks unaligned. Fixes #12415. Change-Id: I7767778d1b4f7d3e74c2719a02848350782a4160 Reviewed-on: https://go-review.googlesource.com/14166 Run-TryBot: Minux Ma <minux@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> (cherry picked from commit 821e124) Reviewed-on: https://go-review.googlesource.com/14279

We need a runtime check because the original issue is encountered when running cross compiled windows program from linux. It's better to give a meaningful crash message earlier than to segfault later. The added test should not impose any measurable overhead to Go programs. For #12415. Change-Id: Ib4a24ef560c09c0585b351d62eefd157b6b7f04c Reviewed-on: https://go-review.googlesource.com/14207 Reviewed-by: Keith Randall <khr@golang.org> Run-TryBot: Minux Ma <minux@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>

minux added this to the Go1.5.1 milestone Sep 2, 2015

minux closed this as completed in 821e124 Sep 4, 2015

beyang mentioned this issue Oct 27, 2015

runtime: unexpected fault address 0x0 in hashmap.go #13062

Closed

golang locked and limited conversation to collaborators Sep 4, 2016

gopherbot added the FrozenDueToAge label Sep 4, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: unexpected fault address 0xffffffff (windows/386) #12415

runtime: unexpected fault address 0xffffffff (windows/386) #12415

AaronO commented Aug 31, 2015

minux commented Aug 31, 2015 via email

AaronO commented Aug 31, 2015

randall77 commented Aug 31, 2015

AaronO commented Sep 1, 2015

randall77 commented Sep 1, 2015

AaronO commented Sep 1, 2015

randall77 commented Sep 1, 2015

alexbrainman commented Sep 1, 2015

minux commented Sep 1, 2015 via email

minux commented Sep 1, 2015 via email

minux commented Sep 2, 2015 via email

AaronO commented Sep 2, 2015

AaronO commented Sep 2, 2015

minux commented Sep 2, 2015 via email

AaronO commented Sep 2, 2015

gopherbot commented Sep 2, 2015

AaronO commented Sep 2, 2015

minux commented Sep 2, 2015 via email

gopherbot commented Sep 3, 2015

gopherbot commented Sep 4, 2015

runtime: unexpected fault address 0xffffffff (windows/386) #12415

runtime: unexpected fault address 0xffffffff (windows/386) #12415

Comments

AaronO commented Aug 31, 2015

minux commented Aug 31, 2015 via email

AaronO commented Aug 31, 2015

randall77 commented Aug 31, 2015

AaronO commented Sep 1, 2015

randall77 commented Sep 1, 2015

AaronO commented Sep 1, 2015

randall77 commented Sep 1, 2015

alexbrainman commented Sep 1, 2015

minux commented Sep 1, 2015 via email

minux commented Sep 1, 2015 via email

minux commented Sep 2, 2015 via email

AaronO commented Sep 2, 2015

AaronO commented Sep 2, 2015

minux commented Sep 2, 2015 via email

AaronO commented Sep 2, 2015

gopherbot commented Sep 2, 2015

AaronO commented Sep 2, 2015

minux commented Sep 2, 2015 via email

gopherbot commented Sep 3, 2015

gopherbot commented Sep 4, 2015