Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: static linking fails on OpenBSD 6.2 #20508

Closed
cac04 opened this issue May 27, 2017 · 30 comments
Closed

cmd/go: static linking fails on OpenBSD 6.2 #20508

cac04 opened this issue May 27, 2017 · 30 comments

Comments

@cac04
Copy link

cac04 commented May 27, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

# go version
go version go1.8 openbsd/amd64

What operating system and processor architecture are you using (go env)?

OpenBSD 6.1 on amd64.

# go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="openbsd"
GOOS="openbsd"
GOPATH="/root/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/openbsd_amd64"
GCCGO="gccgo"
CC="cc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="c++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

I attempted to follow the instructions at http://blog.hashbangbash.com/2014/04/linking-golang-statically/ for producing a statically linked binary from a cgo program.

Here is the source:

package main

/*
char* foo(void) { return "hello, world!"; }
*/
import "C"

import "fmt"

func main() {
        fmt.Println(C.GoString(C.foo()))
}

Which I built like this:

go build --ldflags '-extldflags "-static"' ./cgo-example.go

What did you expect to see?

A successful compilation and linking.

What did you see instead?

# go build --ldflags '-extldflags "-static"' ./cgo-example.go
# command-line-arguments
/usr/local/go/pkg/tool/openbsd_amd64/link: running cc failed: exit status 1
/usr/lib/rcrt0.o: In function `__start':
(.text+0x1a): undefined reference to `_DYNAMIC'
/usr/lib/libpthread.a(rthread.o): In function `_rthread_init':
/usr/src/lib/librthread/rthread.c:231: undefined reference to `_DYNAMIC'
/usr/lib/libpthread.a(rthread_fork.o): In function `_dofork':
/usr/src/lib/librthread/rthread_fork.c:75: undefined reference to `_DYNAMIC'
collect2: ld returned 1 exit status
@odeke-em odeke-em changed the title Static linking on OpenBSD 6.1 cmd/go, cmd/cgo: static linking fails on OpenBSD 6.1 May 27, 2017
@odeke-em
Copy link
Member

/cc @ianlancetaylor

@ianlancetaylor ianlancetaylor added this to the Go1.9Maybe milestone May 29, 2017
@ianlancetaylor
Copy link
Contributor

Can you statically link C code on your system? If so, how do you do it?

@cac04
Copy link
Author

cac04 commented Jun 2, 2017

Yes, with the -static argument to gcc, e.g.

gcc -o foo -static foo.c

@ianlancetaylor
Copy link
Contributor

Please show the output of go build -x --ldflags '-extldflags "-static"' ./cgo-example.go (same command, just added -x). Thanks.

@cac04
Copy link
Author

cac04 commented Jun 2, 2017

WORK=/tmp/go-build723594621
mkdir -p $WORK/command-line-arguments/_obj/
mkdir -p $WORK/command-line-arguments/_obj/exe/
cd /home/cac04/tmp
CGO_LDFLAGS="-g" "-O2" /usr/local/go/pkg/tool/openbsd_amd64/cgo -objdir $WORK/command-line-arguments/_obj/ -importpath command-line-arguments -- -I $WORK/command-line-arguments/_obj/ -g -O2 cgo-example.go
cd $WORK
cc -fdebug-prefix-map=a=b -c trivial.c
cc -gno-record-gcc-switches -c trivial.c
cd /home/cac04/tmp
cc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/_cgo_export.o -c $WORK/command-line-arguments/_obj/_cgo_export.c
cc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/cgo-example.cgo2.o -c $WORK/command-line-arguments/_obj/cgo-example.cgo2.c
cc -I . -fPIC -m64 -pthread -fmessage-length=0 -I $WORK/command-line-arguments/_obj/ -g -O2 -o $WORK/command-line-arguments/_obj/_cgo_main.o -c $WORK/command-line-arguments/_obj/_cgo_main.c
cc -I . -fPIC -m64 -pthread -fmessage-length=0 -o $WORK/command-line-arguments/_obj/_cgo_.o $WORK/command-line-arguments/_obj/_cgo_main.o $WORK/command-line-arguments/_obj/_cgo_export.o $WORK/command-line-arguments/_obj/cgo-example.cgo2.o -g -O2
/usr/local/go/pkg/tool/openbsd_amd64/cgo -dynpackage main -dynimport $WORK/command-line-arguments/_obj/_cgo_.o -dynout $WORK/command-line-arguments/_obj/_cgo_import.go
cd $WORK
cc -no-pie -c trivial.c
cd /home/cac04/tmp
cc -I . -fPIC -m64 -pthread -fmessage-length=0 -o $WORK/command-line-arguments/_obj/_all.o $WORK/command-line-arguments/_obj/_cgo_export.o $WORK/command-line-arguments/_obj/cgo-example.cgo2.o -g -O2 -Wl,-r -nostdlib
/usr/local/go/pkg/tool/openbsd_amd64/compile -o $WORK/command-line-arguments.a -trimpath $WORK -p main -buildid 36eeea8ac0b667a0187e81d5b2a6a2dfe754450f -D _/home/cac04/tmp -I $WORK -pack $WORK/command-line-arguments/_obj/_cgo_gotypes.go $WORK/command-line-arguments/_obj/cgo-example.cgo1.go $WORK/command-line-arguments/_obj/_cgo_import.go
pack r $WORK/command-line-arguments.a $WORK/command-line-arguments/_obj/_all.o # internal
cd .
/usr/local/go/pkg/tool/openbsd_amd64/link -o $WORK/command-line-arguments/_obj/exe/a.out -L $WORK -extld=cc -buildmode=exe -buildid=36eeea8ac0b667a0187e81d5b2a6a2dfe754450f --extldflags -static $WORK/command-line-arguments.a
# command-line-arguments
/usr/local/go/pkg/tool/openbsd_amd64/link: running cc failed: exit status 1
/usr/lib/rcrt0.o: In function `__start':
(.text+0x1a): undefined reference to `_DYNAMIC'
/usr/lib/libpthread.a(rthread.o): In function `_rthread_init':
/usr/src/lib/librthread/rthread.c:231: undefined reference to `_DYNAMIC'
/usr/lib/libpthread.a(rthread_fork.o): In function `_dofork':
/usr/src/lib/librthread/rthread_fork.c:75: undefined reference to `_DYNAMIC'
collect2: ld returned 1 exit status

@ianlancetaylor
Copy link
Contributor

Thanks. That's odd. It looks like it is simply running cc -static, which you say works. OK, please try go build --ldflags '-v -extldflags "-v -static"' ./cgo-example.go (this time I added -v in two places). Thanks.

@cac04
Copy link
Author

cac04 commented Jun 3, 2017

Here's the output of that command:

# command-line-arguments
HEADER = -H7 -T0x401000 -D0x0 -R0x1000
searching for runtime.a in $WORK/runtime.a
searching for runtime.a in /usr/local/go/pkg/openbsd_amd64/runtime.a
 0.00 deadcode
 0.02 pclntab=277386 bytes, funcdata total 31525 bytes
 0.02 dodata
 0.04 dwarf
 0.05 symsize = 0
 0.06 reloc
 0.06 asmb
 0.06 codeblk
 0.07 datblk
 0.07 sym
 0.07 symsize = 49512
 0.08 symsize = 49800
 0.08 dwarf
 0.08 headr
 0.18 host link: "cc" "-m64" "-gdwarf-2" "-Wl,-nopie" "-o" "/tmp/go-build509640703/command-line-arguments/_obj/exe/a.out" "-static" "/tmp/go-link-588660080/go.o" "/tmp/go-link-588660080/000000.o" "/tmp/go-link-588660080/000001.o" "-g" "-O2" "-g" "-O2" "-lpthread" "-v" "-static"
/usr/local/go/pkg/tool/openbsd_amd64/link: running cc failed: exit status 1
Reading specs from /usr/lib/gcc-lib/amd64-unknown-openbsd6.1/4.2.1/specs
Target: amd64-unknown-openbsd6.1
Configured with: OpenBSD/amd64 system compiler
Thread model: posix
gcc version 4.2.1 20070719 
 /usr/lib/gcc-lib/amd64-unknown-openbsd6.1/4.2.1/collect2 -e __start -Bstatic -dynamic-linker /usr/libexec/ld.so -o $WORK/command-line-arguments/_obj/exe/a.out /usr/lib/rcrt0.o /usr/lib/crtbegin.o -L/usr/lib/gcc-lib/amd64-unknown-openbsd6.1/4.2.1 -nopie /tmp/go-link-588660080/go.o /tmp/go-link-588660080/000000.o /tmp/go-link-588660080/000001.o -lpthread -lgcc -lc -lgcc /usr/lib/crtend.o
/usr/lib/rcrt0.o: In function `__start':
(.text+0x1a): undefined reference to `_DYNAMIC'
/usr/lib/libpthread.a(rthread.o): In function `_rthread_init':
/usr/src/lib/librthread/rthread.c:231: undefined reference to `_DYNAMIC'
/usr/lib/libpthread.a(rthread_fork.o): In function `_dofork':
/usr/src/lib/librthread/rthread_fork.c:75: undefined reference to `_DYNAMIC'
collect2: ld returned 1 exit status

From that, I can see that go is calling cc with -Wl,-nopie. I added -nopie to the linker flags and the linking stage appeared to work:

go build --ldflags '--extldflags "-static -nopie"' ./cgo-example

However, trying to run the resulting binary failed with a segfault.

@ianlancetaylor
Copy link
Contributor

The --extldflags are passed to cc, so you should use -Wl,-nopie in the --extldflags line. But I don't understand why that would make any difference--the go tool is already passing -Wl,-nopie.

What happens if you run cc -Wl,-nopie -static foo.c -lpthread for some trivial program? Does it work?

@cac04
Copy link
Author

cac04 commented Jun 9, 2017

Here are the results of passing different options to gcc on OpenBSD:

cac04:/home/cac04/tmp$ gcc -o foo foo.c
cac04:/home/cac04/tmp$ ldd foo
foo:
        Start            End              Type Open Ref GrpRef Name
        00001b8c1b800000 00001b8c1ba02000 exe  1    0   0      foo
        00001b8eba746000 00001b8ebaa12000 rlib 0    1   0      /usr/lib/libc.so.89.3
        00001b8e5d200000 00001b8e5d200000 rtld 0    1   0      /usr/libexec/ld.so
cac04:/home/cac04/tmp$
cac04:/home/cac04/tmp$ gcc -static -o foo foo.c
cac04:/home/cac04/tmp$ ldd foo
foo:
        Start            End              Type Open Ref GrpRef Name
        00001c473d25c000 00001c473d47b000 dlib 1    0   0      /home/cac04/tmp/foo
cac04:/home/cac04/tmp$
cac04:/home/cac04/tmp$ gcc -Wl,-nopie -o foo foo.c
cac04:/home/cac04/tmp$ ldd foo
foo:
        Start            End              Type Open Ref GrpRef Name
        0000000000400000 0000000000602000 exe  1    0   0      foo
        0000000292800000 0000000292acc000 rlib 0    1   0      /usr/lib/libc.so.89.3
        0000000228500000 0000000228500000 rtld 0    1   0      /usr/libexec/ld.so
cac04:/home/cac04/tmp$
cac04:/home/cac04/tmp$ gcc -Wl,-nopie -static -o foo foo.c
/usr/lib/rcrt0.o: In function `__start':
(.text+0x1a): undefined reference to `_DYNAMIC'
collect2: ld returned 1 exit status
cac04:/home/cac04/tmp$
cac04:/home/cac04/tmp$ gcc -Wl,-nopie -static -nopie -o foo foo.c
cac04:/home/cac04/tmp$ ldd foo
foo:
not a dynamic executable
cac04:/home/cac04/tmp$

In the first one, with no extra arguments, we get a dynamically linked PIE.

In the second one, with -static, we get a statically linked PIE.

In the third one, with -Wl,-nopie, we get a dynamically linked non-PIE.

In the fourth one, with -Wl,-nopie -static, we get a linker failure.

In the fifth one, with -Wl,-nopie -static -nopie, we get a statically linked non-PIE.

go build seems to be trying to do the fourth one, which doesn't work.

With a bit more detail, I think we can see why it doesn't work:

cac04:/home/cac04/tmp:410$ gcc -static -Wl,-nopie -Wl,-v -o foo foo.c
collect2 version 4.2.1 20070719  (OpenBSD/x86-64 ELF)
/usr/bin/ld -e __start -Bstatic -dynamic-linker /usr/libexec/ld.so -o foo /usr/lib/rcrt0.o /usr/lib/crtbegin.o -L/usr/lib/gcc-lib/amd64-unknown-openbsd6.1/4.2.1 -nopie -v /tmp//ccZ7VS0V.o -lgcc -lc -lgcc /usr/lib/crtend.o
GNU ld version 2.17
/usr/lib/rcrt0.o: In function `__start':
(.text+0x1a): undefined reference to `_DYNAMIC'
collect2: ld returned 1 exit status

The -static argument causes it to link against rcrt0.o, which doesn't work with -Wl,-nopie. We need to add -nopie to get it to link against crt0.o instead:

cac04:/home/cac04/tmp:411$ gcc -static -nopie -Wl,-nopie -Wl,-v -o foo foo.c
collect2 version 4.2.1 20070719  (OpenBSD/x86-64 ELF)
/usr/bin/ld -e __start -Bstatic -dynamic-linker /usr/libexec/ld.so -nopie -o foo /usr/lib/crt0.o /usr/lib/crtbegin.o -L/usr/lib/gcc-lib/amd64-unknown-openbsd6.1/4.2.1 -nopie -v /tmp//cc4exAl2.o -lgcc -lc -lgcc /usr/lib/crtend.o
GNU ld version 2.17

As mentioned above, this command does complete successfully:

go build --ldflags '--extldflags "-static -nopie"' ./cgo-example

But then I get a segfault when I try to run the resulting executable:

Reading symbols from ./cgo-example...done.
Loading Go Runtime support.
(gdb) run
Starting program: /home/cac04/tmp/cgo-example

Program received signal SIGSEGV, Segmentation fault.
runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:141
141             MOVQ    CX, g(BX)
(gdb) bt
#0  runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:141
#1  0x0000000000000001 in ?? ()
#2  0x0000000000000000 in ?? ()

@cac04
Copy link
Author

cac04 commented Jun 15, 2017

I asked on an OpenBSD mailing list about passing -Wl,-nopie or just -nopie to gcc. You can see the thread here: http://marc.info/?t=149745633100003

The conclusion seems to be that go build should be using -nopie rather than -Wl,-nopie when it calls gcc to link.

@ianlancetaylor
Copy link
Contributor

As it happens, the use of -Wl,-nopie is already OpenBSD specific, dating back to https://golang.org/cl/7572049. So this looks like an easy fix.

@gopherbot
Copy link

CL https://golang.org/cl/45992 mentions this issue.

@ianlancetaylor
Copy link
Contributor

OK, so I didn't read closely enough. Apparently using -nopie is not enough. Above it says that

go build --ldflags '--extldflags "-static -nopie"' ./cgo-example

links successfully, but the resulting program does not run.

What do we have to do to get a program that both links and runs?

@ianlancetaylor
Copy link
Contributor

The location of the crash suggests that there is something wrong with the support for TLS in a statically linked OpenBSD binary.

I'm going to punt this to 1.10. If someone can figure this out soon we can still get the patch into 1.9.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.10, Go1.9Maybe Jun 16, 2017
@cac04
Copy link
Author

cac04 commented Oct 7, 2017

Sorry for the very slow reply, non-hobby-life interjected. As far as I know, OpenBSD does not support thread-local storage.

@andrewchambers
Copy link

I think it is emulated (though I don't understand the details), lots of things might have changed with the 6.2 openbsd release because it now uses clang by default.

@cac04
Copy link
Author

cac04 commented Oct 19, 2017

I have retested this with OpenBSD 6.2. The situation is the same as before. The resulting binary still segfaults at the same place.

@ianlancetaylor ianlancetaylor changed the title cmd/go, cmd/cgo: static linking fails on OpenBSD 6.1 cmd/go: static linking fails on OpenBSD 6.1 Oct 19, 2017
@ianlancetaylor
Copy link
Contributor

I think someone familiar with OpenBSD is going to have to look into this. My best guess, which is probably wrong, is that we are somehow generating the wrong relocations for TLS references.

@cac04
Copy link
Author

cac04 commented Oct 30, 2017

I've had a chance to investigate this a bit more...

OpenBSD does not (yet) support thread-local storage. They work around this by using emulated TLS in the runtime library libcompiler_rt.

As of OpenBSD 6.2, the most recent release, the base compiler for i386 and amd64 platforms is Clang/LLVM. It used to be GCC.

On OpenBSD (and Android and Windows/Cygwin) Clang enables TLS emulation by default: see Clang.cpp line 3212.

This causes LLVM to add a compiler pass that turns references to TLS variables into calls to the relevant emulation functions in the runtime library: see TargetPassConfig.cpp line 640 and TargetLowering.ccp line 3824.

Given a C source file that looks like this:

__thread int foo;
int get_foo() {
	return foo;
}

I get the following on a Linux system that supports TLS:

$ uname -mrsv
Linux 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64
$ cc -o test.o -c test.c
$ readelf -rW test.o

Relocation section '.rela.text' at offset 0x208 contains 1 entries:
    Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
0000000000000008  0000000900000017 R_X86_64_TPOFF32       0000000000000000 foo + 0

But on OpenBSD 6.2:

$ uname -mrsv
OpenBSD 6.2 GENERIC.MP#134 amd64
$ cc -o test.o -c test.c
$ readelf -rW test.o

Relocation section '.rela.text' at offset 0x140 contains 2 entries:
    Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
0000000000000007  0000000400000002 R_X86_64_PC32          0000000000000000 __emutls_v.foo + fffffffffffffffc
000000000000000c  0000000300000004 R_X86_64_PLT32         0000000000000000 __emutls_get_address + fffffffffffffffc

And __emutls_get_address is defined in libcompiler_rt: see emutls.c line 183.

GCC, which is still the base compiler for OpenBSD on all architectures except i386 and amd64, had all this before: see tree-emutls.c for the compiler pass and emutls.c for the runtime support. LLVM added compatible support a couple of years ago.

So this all works fine for anything compiled with Clang or GCC. But Go has its own toolchain. I don't know how Go uses TLS. What support, if any, does Go need from the system linker or kernel for TLS?

I wondered if commit 9417c02 has anything to do with this. That commit removes some code that previously allocated memory for TLS data on OpenBSD. It was made because OpenBSD 6.0 added support for PT_TLS sections in ld.so - see tib.c. But that just means that the necessary memory will be allocated by the dynamic linker: I'm not sure that it will work if the binary is statically linked.

I tried building the Go toolchain from that commit's parent and testing with that. However, if I try to statically link with that it fails at link time due to multiple definitions of pthread_create: the wrapper in runtime/cgo/gcc_openbsd_amd64.c clashes with the definition in the system pthread library.

So I guess static linking of Go binaries has never worked on OpenBSD.

@ianlancetaylor
Copy link
Contributor

@cac04 Thanks for looking into this. The Go runtime basically uses a single TLS variable, which holds a pointer to the currently running goroutine. When using external linking, which happens by default when using cgo code, the Go linker produces an object file that is passed to the system linker. So we need to arrange for the Go linker to generate the code and relocations that the system linker expects to see.

For amd64 this process starts in cmd/compile/internal/amd64/ssa.go. Look for OpAMD64LoweredGetG, which is the compiler internal pseudo-instruction that fetches the current goroutine pointer. From what you say above it sounds like that code needs to change to generate a call to an emultls function. But if the emutls function can change any registers, which I assume it can, then simply changing amd64/ssa.go will not be enough; we need to teach the compiler in general about emutls, and teach it to generate function calls for GetG.

@cac04
Copy link
Author

cac04 commented Oct 31, 2017

A couple of things made me think I must have missed something:

  1. Clang also uses Emulated TLS on Android. But Go, including cgo, works fine on Android, doesn't it?
  2. Go, including cgo, seems to work fine on OpenBSD as long as I don't pass -static to the external linker.

So clearly even if Android and OpenBSD don't support thread-local storage properly, whatever 'properly' means, they support it enough for Go to work without using the Emulated TLS functions.

Thanks for the pointer to OpAMD64LoweredGetG: I think I see how this works now. Apologies for teaching grandmother to suck eggs in what follows but this is all new to me so I'm writing out my reasoning in full.

ELF Handling For Thread-Local Storage describes two different ways of allocating TLS blocks. First:

The TLS blocks for the executable itself and all the modules loaded at startup
are located just below the address the thread pointer points to. This allows
compilers to emit code which directly accesses this memory.

Second, the Thread Control Block (TCB) contains (at some OS-dependent location) a pointer to a dynamic thread vector that contains pointers to TLS blocks allocated for dynamically-loaded modules (which in this context means modules loaded after program startup, e.g. via dlopen(3), or lazily loaded modules).

There are four access models for TLS variables, which can be split into two types: 'Dynamic' and 'Exec'. The 'Dynamic' models allow access to TLS blocks reached via the dynamic thread vector, while the 'Exec' models only allow access to TLS blocks allocated at program startup. The 'Local Exec' model further restricts access to only those TLS variables defined in the executable itself.

The advantage of the 'Local Exec' model is that we know the variables must be in the first TLS block, which is at a fixed (although architecture dependent) offset from the thread pointer. Since the program linker knows the variable's offset within the TLS block, we know the variable's offset from the thread pointer at link-time.

So, if we use the 'Local Exec' model then we only need support for TLS relocations in the program linker. The dynamic linker doesn't need to do anything.

If this is how Go uses TLS, then Go requires only the following support for TLS:

  1. Allocation of static TLS blocks at program startup.
  2. Support for R_x_TPOFF32 relocations in the program linker.

First, let's check that this really is how Go uses TLS.

OpAMD64LoweredGetG loads G by loading whatever is pointed to by x86.REG_TLS. If I followed the logic in cmd/internal/obj/x86/asm6.go correctly, x86.REG_TLS will be the %fs segment register on amd64 and %gs on i386. That matches what is specified in ELF Handling For Thread-Local Storage.

A comment on OpAMD64LoweredGetG refers to a comment in cmd/internal/obj/x86/obj6.go, which confirms that we're using the 'local exec model given in "ELF Handling For Thread-Local Storage"'.

runtime/go_tls.h contains this:

#define	get_tls(r)	MOVQ TLS, r
#define	g(r) 0(r)(TLS*1)

And the comment in cmd/internal/obj/x86/obj6.go says:

An offset from the thread-local storage base is written off(reg)(TLS*1).
Semantically it is off(reg), but the (TLS*1) annotation marks this as
indexing from the loaded TLS base. This emits a relocation so that if the
linker needs to adjust the offset, it can.

If we follow that through, the (TLS*1) causes us to end up on line 2840 of cmd/internal/obj/x86/asm6.go where we create a relocation of type objapi.R_TLS_LE.

For pure Go programs, the Go linker handles this in cmd/link/internal/ld/data.go by applying an offset of -1 * ctxt.Arch.PtrSize (computed in cmd/link/internal/ld/sym.go). That fits what we said earlier about the 'Local Exec' model providing access to the first TLS block: ELF Handling For Thread-Local Storage specifies that offsets must be subtracted from the thread pointer.

For cgo we will need to use external linking, which means that we will end up on line 384 of cmd/link/internal/amd64/asm.go:

ctxt.Out.Write64(uint64(elf.R_X86_64_TPOFF32) | uint64(elfsym)<<32)

So we do indeed end up with just a R_x_TPOFF32 relocation, which can be handled by the program linker without needing any support in the dynamic linker.

This is all good because OpenBSD uses the GNU binutils ld as the program linker, which can handle R_x_TPOFF32 relocations, but OpenBSD's dynamic linker does not support any TLS relocations at all.

The only other thing we require is that the static TLS blocks are allocated at program startup.

If I understand tib.c and, specifically, line 174 of library.c in OpenBSD's ld.so correctly, OpenBSD does not provide a dynamic thread vector. But it does allocate static TLS blocks on program start-up.

So OpenBSD does not support TLS 'properly' because:

  1. The dynamic linker does not support TLS relocations
  2. There is no support for a dynamic thread vector of TLS blocks
  3. Static TLS blocks are allocated by ld.so only, not when a statically-linked non-PIE program starts

Edited 2017-11-02: Removed incorrect claim that OpenBSD's pthread_create was lacking support for TLS block allocation. That's wrong: OpenBSD 6.0 added TLS support in both ld.so and pthread_create... which makes sense, as you need to copy the master copy of the TLS blocks whenever a new thread is spawned.

It's the third point that means Go works fine when dynamically linked but not when statically linked.

Before OpenBSD 6.0, ld.so did not support TLS at all. So Go used to provide a wrapper for pthread_create in runtime/cgo/gcc_openbsd_amd64.c that allocated memory for a TLS block. That wrapper was removed in commit 9417c02.

The wrapper doesn't help with statically linking anyway, as you just get a symbol clash with the pthread_create provided by the system.

Incidentally, commit 9417c02 appears to have gone a bit too far. It's allowing Initial Exec relocations but they won't work without dynamic linker support.

Finally, Android's linker doesn't seem to support TLS relocations either, which is presumably why Clang enables Emulated TLS for Android too. There isn't even any support for the link-time R_x_TPOFF32 relocation used for 'Local Exec'. I guess this is why the offset is hard-coded in runtime/cgo/gcc_android_amd64.c.

Well, that was fun. Sorry for the wall of text!

@cac04
Copy link
Author

cac04 commented Nov 2, 2017

I apologize for spamming this issue with long, rambling comments. I thought it might be interesting for others to follow my chain of reasoning.

I believe I have now got to the bottom of this issue. I can replicate the problem without involving Go at all (i.e. with just C and assembly source files).

Edited 2017-11-03: Actually, I hadn't got to the bottom of it. Edited to remove an incorrect conclusion.

Here are the details:

As described in the previous comment, OpenBSD does support 'Local Exec' mode TLS when ld.so is invoked. So for dynamically linked executables, everything is fine. ld.so takes care of allocating and initializing the TLS block.

If we look in the C runtime, line 78 of crt0.c calls _csu_finish. This is defined in libc/dlfcn/init.c.

This function contains code for setting up a TLS block, see lines 104 and 152, but it is wrapped in:

#ifndef PIC

When libc is built as a shared library, it is built with -DPIC so this code is skipped. But when it is built as a static library, PIC is not defined so this code is included.

So, when we are dynamically linking, this code is not included and ld.so takes care of setting up TLS; but when we are statically linking, this code is included and it takes care of setting up TLS. It would appear that TLS should work whichever way we link.

However, TLS does not work with a statically-linked non-PIE executable:

Given the following assembly in a file called test.s:

	.globl	foo
	.section	.tbss,"awT",@nobits
	.align 4
	.type	foo, @object
	.size	foo, 4
foo:
	.zero	4
	.text
	.globl	get_foo
	.type	get_foo, @function
get_foo:
	movl	%fs:foo@tpoff, %eax
	ret

I created an object file as follows:

$ cc -o test.o -c test.s
$ readelf -rW test.o

Relocation section '.rela.text' at offset 0xc8 contains 1 entries:
    Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
0000000000000004  0000000300000017 R_X86_64_TPOFF32       0000000000000000 foo + 0

This gives me a R_x_TPOFF32 relocation, just as would be produced by Go when using external linking (i.e. no emulated TLS functions).

With a foo.c containing a main function as follows:

#include <stdio.h>

extern int get_foo();

int main()
{
	printf("%d\n", get_foo());
	return 0;
}

I can test linking it dynamically, statically as a PIE, and statically as a non-PIE:

$ cc -o foo foo.c test.o
$ ./foo
0
$ cc -static -o foo foo.c test.o
$ ./foo
0
$ cc -static -nopie -o foo foo.c test.o
$ ./foo
Segmentation fault (core dumped)

That's the same result I have been getting with my cgo program.

If Go supported -buildmode=pie on OpenBSD, this problem could be worked around. Until that happens, I don't think there's anything else that Go could do.

@ianlancetaylor
Copy link
Contributor

@cac04 Thanks for the investigation.

It might not take any work to support -buildmode=pie on OpenBSD, since presumably it works the same as on any ELF system. It would be worth tweaking BuildModeInit in cmd/go/internal/work/build.go and (*BuildMode).Set in cmd/link/internal/ld/config.go and (*tester).supportBuildMode in cmd/dist/test.go to permit "pie" on OpenBSD, and see what happens.

@cac04
Copy link
Author

cac04 commented Nov 3, 2017

Thanks for the pointers, I will try that out.

Meanwhile, I've realized that I was wrong again. It is true that -static -nopie breaks TLS on OpenBSD but not for the reason that I thought. I was misled by OpenBSD's fancy anti-ROP mechanism in libc: the .a file that I inspected was not the static libc but rather an archive of shared objects. If I inspect the correct libc.a I can see that it does initialize the static TLS block and this code is called whenever we link statically, regardless of whether or not we build with -nopie.

Since I can recreate this problem without Go, I'll post to an OpenBSD mailing list and see if someone more knowledgeable than me can help.

@cac04
Copy link
Author

cac04 commented Nov 11, 2017

Success at last! I received some help on the OpenBSD tech@ mailing list. In short: the current release of OpenBSD does not initialize TLS blocks when the executable is both statically-linked and not position-independent. It is a limitation of OpenBSD 6.2.

However, I was given a patch to fix this and it appears to work. With the patch, .tbss sections work correctly in statically-linked non-PIE binaries. (There's a bit more work to do to get .tdata sections working but I don't think Go needs them.)

For the details, see this post to the mailing list:
https://marc.info/?l=openbsd-tech&m=150992440513412

With the patch, my cgo program works correctly. The cgo programs from the Go test suite also seem to work when statically linked (except for those that shouldn't, e.g. issue4029 which calls dlopen).

On the assumption that the patch will make its way into a future OpenBSD release, I think we can call this issue resolved without any changes to Go.

There were a couple of things that cropped up while looking into this though:

Now I'll have a go at getting -buildmode=pie to work.

@cac04
Copy link
Author

cac04 commented Nov 23, 2017

The following patch to enable -buildmode=pie on OpenBSD seems to work:

diff --git cmd/dist/test.go cmd/dist/test.go
index bbc2a0f4ad..fd1a79ed52 100644
--- cmd/dist/test.go
+++ cmd/dist/test.go
@@ -879,6 +879,8 @@ func (t *tester) supportedBuildmode(mode string) bool {
 			return true
 		case "darwin-amd64":
 			return true
+		case "openbsd-amd64", "openbsd-386", "openbsd-arm":
+			return true
 		}
 		return false
 
diff --git cmd/go/internal/work/init.go cmd/go/internal/work/init.go
index 0e17286cf6..ae6942859c 100644
--- cmd/go/internal/work/init.go
+++ cmd/go/internal/work/init.go
@@ -136,6 +136,8 @@ func buildModeInit() {
 				codegenArg = "-shared"
 			case "darwin/amd64":
 				codegenArg = "-shared"
+			case "openbsd/amd64", "openbsd/386", "openbsd/arm":
+				codegenArg = "-shared"
 			default:
 				base.Fatalf("-buildmode=pie not supported on %s\n", platform)
 			}
diff --git cmd/link/internal/ld/config.go cmd/link/internal/ld/config.go
index cc95392d77..2f5988efb3 100644
--- cmd/link/internal/ld/config.go
+++ cmd/link/internal/ld/config.go
@@ -45,6 +45,12 @@ func (mode *BuildMode) Set(s string) error {
 			default:
 				return badmode()
 			}
+		case "openbsd":
+			switch objabi.GOARCH {
+			case "amd64", "386", "arm":
+			default:
+				return badmode()
+			}
 		default:
 			return badmode()
 		}
diff --git cmd/link/internal/ld/lib.go cmd/link/internal/ld/lib.go
index cd8b45cd2e..53a3e4b14b 100644
--- cmd/link/internal/ld/lib.go
+++ cmd/link/internal/ld/lib.go
@@ -1093,7 +1093,9 @@ func (ctxt *Link) hostlink() {
 			argv = append(argv, "-Wl,-no_pie")
 		}
 	case objabi.Hopenbsd:
-		argv = append(argv, "-Wl,-nopie")
+		if ctxt.BuildMode != BuildModePIE {
+			argv = append(argv, "-nopie")
+		}
 	case objabi.Hwindows:
 		if windowsgui {
 			argv = append(argv, "-mwindows")

@rsc rsc modified the milestones: Go1.10, Go1.11 Dec 1, 2017
@cac04
Copy link
Author

cac04 commented Dec 2, 2017

The patch mentioned above has been committed to OpenBSD. See here and here.

@bradfitz bradfitz changed the title cmd/go: static linking fails on OpenBSD 6.1 cmd/go: static linking fails on OpenBSD 6.2 Dec 5, 2017
@gopherbot gopherbot modified the milestones: Go1.11, Unplanned May 23, 2018
@bcmills
Copy link
Contributor

bcmills commented Jan 18, 2019

On the assumption that the patch will make its way into a future OpenBSD release, I think we can call this issue resolved without any changes to Go.

The patch mentioned above has been committed to OpenBSD. See here and here.

Does that mean that this issue is resolved?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 18, 2019
@cac04
Copy link
Author

cac04 commented Jan 20, 2019

Does that mean that this issue is resolved?

Yes, the issue initially reported here is resolved.

There were a couple of other things that came up during investigation of this issue. I didn't know if they should be dealt with as part of this issue, if new issues should be opened for them, or if they were of no interest. Here they are:

There's also a patch above to enable -buildmode=pie on OpenBSD.

@bcmills
Copy link
Contributor

bcmills commented Jan 22, 2019

Thanks. Please open separate issues for the follow-on items — and describe their impact — so that we can prioritize them independently.

If you'd like to merge the patch to enable -buildmode=pie please send it through Gerrit or a GitHub PR (per https://golang.org/doc/contribute.html).

@bcmills bcmills closed this as completed Jan 22, 2019
@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 22, 2019
@golang golang locked and limited conversation to collaborators Jan 22, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants