New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/go: static linking fails on OpenBSD 6.2 #20508
Comments
/cc @ianlancetaylor |
Can you statically link C code on your system? If so, how do you do it? |
Yes, with the
|
Please show the output of |
|
Thanks. That's odd. It looks like it is simply running |
Here's the output of that command:
From that, I can see that
However, trying to run the resulting binary failed with a segfault. |
The What happens if you run |
Here are the results of passing different options to
In the first one, with no extra arguments, we get a dynamically linked PIE. In the second one, with In the third one, with In the fourth one, with In the fifth one, with
With a bit more detail, I think we can see why it doesn't work:
The
As mentioned above, this command does complete successfully:
But then I get a segfault when I try to run the resulting executable:
|
I asked on an OpenBSD mailing list about passing The conclusion seems to be that |
As it happens, the use of -Wl,-nopie is already OpenBSD specific, dating back to https://golang.org/cl/7572049. So this looks like an easy fix. |
CL https://golang.org/cl/45992 mentions this issue. |
OK, so I didn't read closely enough. Apparently using
links successfully, but the resulting program does not run. What do we have to do to get a program that both links and runs? |
The location of the crash suggests that there is something wrong with the support for TLS in a statically linked OpenBSD binary. I'm going to punt this to 1.10. If someone can figure this out soon we can still get the patch into 1.9. |
Sorry for the very slow reply, non-hobby-life interjected. As far as I know, OpenBSD does not support thread-local storage. |
I think it is emulated (though I don't understand the details), lots of things might have changed with the 6.2 openbsd release because it now uses clang by default. |
I have retested this with OpenBSD 6.2. The situation is the same as before. The resulting binary still segfaults at the same place. |
I think someone familiar with OpenBSD is going to have to look into this. My best guess, which is probably wrong, is that we are somehow generating the wrong relocations for TLS references. |
I've had a chance to investigate this a bit more... OpenBSD does not (yet) support thread-local storage. They work around this by using emulated TLS in the runtime library As of OpenBSD 6.2, the most recent release, the base compiler for On OpenBSD (and Android and Windows/Cygwin) Clang enables TLS emulation by default: see Clang.cpp line 3212. This causes LLVM to add a compiler pass that turns references to TLS variables into calls to the relevant emulation functions in the runtime library: see TargetPassConfig.cpp line 640 and TargetLowering.ccp line 3824. Given a C source file that looks like this: __thread int foo;
int get_foo() {
return foo;
} I get the following on a Linux system that supports TLS: $ uname -mrsv
Linux 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64
$ cc -o test.o -c test.c
$ readelf -rW test.o
Relocation section '.rela.text' at offset 0x208 contains 1 entries:
Offset Info Type Symbol's Value Symbol's Name + Addend
0000000000000008 0000000900000017 R_X86_64_TPOFF32 0000000000000000 foo + 0 But on OpenBSD 6.2: $ uname -mrsv
OpenBSD 6.2 GENERIC.MP#134 amd64
$ cc -o test.o -c test.c
$ readelf -rW test.o
Relocation section '.rela.text' at offset 0x140 contains 2 entries:
Offset Info Type Symbol's Value Symbol's Name + Addend
0000000000000007 0000000400000002 R_X86_64_PC32 0000000000000000 __emutls_v.foo + fffffffffffffffc
000000000000000c 0000000300000004 R_X86_64_PLT32 0000000000000000 __emutls_get_address + fffffffffffffffc And GCC, which is still the base compiler for OpenBSD on all architectures except So this all works fine for anything compiled with Clang or GCC. But Go has its own toolchain. I don't know how Go uses TLS. What support, if any, does Go need from the system linker or kernel for TLS? I wondered if commit 9417c02 has anything to do with this. That commit removes some code that previously allocated memory for TLS data on OpenBSD. It was made because OpenBSD 6.0 added support for I tried building the Go toolchain from that commit's parent and testing with that. However, if I try to statically link with that it fails at link time due to multiple definitions of So I guess static linking of Go binaries has never worked on OpenBSD. |
@cac04 Thanks for looking into this. The Go runtime basically uses a single TLS variable, which holds a pointer to the currently running goroutine. When using external linking, which happens by default when using cgo code, the Go linker produces an object file that is passed to the system linker. So we need to arrange for the Go linker to generate the code and relocations that the system linker expects to see. For amd64 this process starts in cmd/compile/internal/amd64/ssa.go. Look for |
A couple of things made me think I must have missed something:
So clearly even if Android and OpenBSD don't support thread-local storage properly, whatever 'properly' means, they support it enough for Go to work without using the Emulated TLS functions. Thanks for the pointer to ELF Handling For Thread-Local Storage describes two different ways of allocating TLS blocks. First:
Second, the Thread Control Block (TCB) contains (at some OS-dependent location) a pointer to a dynamic thread vector that contains pointers to TLS blocks allocated for dynamically-loaded modules (which in this context means modules loaded after program startup, e.g. via There are four access models for TLS variables, which can be split into two types: 'Dynamic' and 'Exec'. The 'Dynamic' models allow access to TLS blocks reached via the dynamic thread vector, while the 'Exec' models only allow access to TLS blocks allocated at program startup. The 'Local Exec' model further restricts access to only those TLS variables defined in the executable itself. The advantage of the 'Local Exec' model is that we know the variables must be in the first TLS block, which is at a fixed (although architecture dependent) offset from the thread pointer. Since the program linker knows the variable's offset within the TLS block, we know the variable's offset from the thread pointer at link-time. So, if we use the 'Local Exec' model then we only need support for TLS relocations in the program linker. The dynamic linker doesn't need to do anything. If this is how Go uses TLS, then Go requires only the following support for TLS:
First, let's check that this really is how Go uses TLS. OpAMD64LoweredGetG loads A comment on runtime/go_tls.h contains this:
And the comment in cmd/internal/obj/x86/obj6.go says:
If we follow that through, the For pure Go programs, the Go linker handles this in cmd/link/internal/ld/data.go by applying an offset of For cgo we will need to use external linking, which means that we will end up on line 384 of cmd/link/internal/amd64/asm.go:
So we do indeed end up with just a This is all good because OpenBSD uses the GNU binutils The only other thing we require is that the static TLS blocks are allocated at program startup. If I understand tib.c and, specifically, line 174 of library.c in OpenBSD's So OpenBSD does not support TLS 'properly' because:
Edited 2017-11-02: Removed incorrect claim that OpenBSD's It's the third point that means Go works fine when dynamically linked but not when statically linked. Before OpenBSD 6.0, The wrapper doesn't help with statically linking anyway, as you just get a symbol clash with the Incidentally, commit 9417c02 appears to have gone a bit too far. It's allowing Initial Exec relocations but they won't work without dynamic linker support. Finally, Android's linker doesn't seem to support TLS relocations either, which is presumably why Clang enables Emulated TLS for Android too. There isn't even any support for the link-time Well, that was fun. Sorry for the wall of text! |
I apologize for spamming this issue with long, rambling comments. I thought it might be interesting for others to follow my chain of reasoning. I believe I have now got to the bottom of this issue. I can replicate the problem without involving Go at all (i.e. with just C and assembly source files). Edited 2017-11-03: Actually, I hadn't got to the bottom of it. Edited to remove an incorrect conclusion. Here are the details: As described in the previous comment, OpenBSD does support 'Local Exec' mode TLS when If we look in the C runtime, line 78 of crt0.c calls This function contains code for setting up a TLS block, see lines 104 and 152, but it is wrapped in: #ifndef PIC When So, when we are dynamically linking, this code is not included and However, TLS does not work with a statically-linked non-PIE executable: Given the following assembly in a file called
I created an object file as follows: $ cc -o test.o -c test.s
$ readelf -rW test.o
Relocation section '.rela.text' at offset 0xc8 contains 1 entries:
Offset Info Type Symbol's Value Symbol's Name + Addend
0000000000000004 0000000300000017 R_X86_64_TPOFF32 0000000000000000 foo + 0 This gives me a With a
I can test linking it dynamically, statically as a PIE, and statically as a non-PIE:
That's the same result I have been getting with my cgo program. If Go supported |
@cac04 Thanks for the investigation. It might not take any work to support |
Thanks for the pointers, I will try that out. Meanwhile, I've realized that I was wrong again. It is true that Since I can recreate this problem without Go, I'll post to an OpenBSD mailing list and see if someone more knowledgeable than me can help. |
Success at last! I received some help on the OpenBSD However, I was given a patch to fix this and it appears to work. With the patch, For the details, see this post to the mailing list: With the patch, my cgo program works correctly. The cgo programs from the Go test suite also seem to work when statically linked (except for those that shouldn't, e.g. issue4029 which calls On the assumption that the patch will make its way into a future OpenBSD release, I think we can call this issue resolved without any changes to Go. There were a couple of things that cropped up while looking into this though:
Now I'll have a go at getting |
The following patch to enable diff --git cmd/dist/test.go cmd/dist/test.go
index bbc2a0f4ad..fd1a79ed52 100644
--- cmd/dist/test.go
+++ cmd/dist/test.go
@@ -879,6 +879,8 @@ func (t *tester) supportedBuildmode(mode string) bool {
return true
case "darwin-amd64":
return true
+ case "openbsd-amd64", "openbsd-386", "openbsd-arm":
+ return true
}
return false
diff --git cmd/go/internal/work/init.go cmd/go/internal/work/init.go
index 0e17286cf6..ae6942859c 100644
--- cmd/go/internal/work/init.go
+++ cmd/go/internal/work/init.go
@@ -136,6 +136,8 @@ func buildModeInit() {
codegenArg = "-shared"
case "darwin/amd64":
codegenArg = "-shared"
+ case "openbsd/amd64", "openbsd/386", "openbsd/arm":
+ codegenArg = "-shared"
default:
base.Fatalf("-buildmode=pie not supported on %s\n", platform)
}
diff --git cmd/link/internal/ld/config.go cmd/link/internal/ld/config.go
index cc95392d77..2f5988efb3 100644
--- cmd/link/internal/ld/config.go
+++ cmd/link/internal/ld/config.go
@@ -45,6 +45,12 @@ func (mode *BuildMode) Set(s string) error {
default:
return badmode()
}
+ case "openbsd":
+ switch objabi.GOARCH {
+ case "amd64", "386", "arm":
+ default:
+ return badmode()
+ }
default:
return badmode()
}
diff --git cmd/link/internal/ld/lib.go cmd/link/internal/ld/lib.go
index cd8b45cd2e..53a3e4b14b 100644
--- cmd/link/internal/ld/lib.go
+++ cmd/link/internal/ld/lib.go
@@ -1093,7 +1093,9 @@ func (ctxt *Link) hostlink() {
argv = append(argv, "-Wl,-no_pie")
}
case objabi.Hopenbsd:
- argv = append(argv, "-Wl,-nopie")
+ if ctxt.BuildMode != BuildModePIE {
+ argv = append(argv, "-nopie")
+ }
case objabi.Hwindows:
if windowsgui {
argv = append(argv, "-mwindows") |
Yes, the issue initially reported here is resolved. There were a couple of other things that came up during investigation of this issue. I didn't know if they should be dealt with as part of this issue, if new issues should be opened for them, or if they were of no interest. Here they are:
There's also a patch above to enable |
Thanks. Please open separate issues for the follow-on items — and describe their impact — so that we can prioritize them independently. If you'd like to merge the patch to enable |
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version
)?What operating system and processor architecture are you using (
go env
)?OpenBSD 6.1 on amd64.
What did you do?
I attempted to follow the instructions at http://blog.hashbangbash.com/2014/04/linking-golang-statically/ for producing a statically linked binary from a
cgo
program.Here is the source:
Which I built like this:
What did you expect to see?
A successful compilation and linking.
What did you see instead?
The text was updated successfully, but these errors were encountered: