cc @FiloSottile

Thanks for filing bugs for the flaky tests, @mvdan. Hope we get back to a stable dashboard soon.

This is not my kind of TLS ;) it's Thread Local Storage.

Hahaha. Oops. :P

cc @bcmills @ianlancetaylor

The faulting PC, 0xe56e4, is in C code getTLS:

TEXT getTLS(SB) 
  :0                    0xe56d0                 e59f0014                MOVW 0x14(R15), R0      
  :0                    0xe56d4                 e92d4010                PUSH [R4,R14]           
  :0                    0xe56d8                 e08f0000                ADD R0, R15, R0         
  :0                    0xe56dc                 fa001a0a                BLX 0xebf0c             // __tls_get_addr
  :0                    0xe56e0                 e59f3008                MOVW 0x8(R15), R3       
  :0                    0xe56e4                 e7930000                MOVW (R3)(R0), R0       // <--- faulting PC
  :0                    0xe56e8                 e8bd8010                POP [R4,R15]            
  :0                    0xe56ec                 00120938                AND.S.EQ R8>>R9, R2, R0 
  :0                    0xe56f0                 0000001c                AND.EQ R12<<R0, R0, R0  

Here, we just loaded R3=0x1c from the constant pool. The faulting address is 0x1b, which means R0=-1, which means __tls_get_addr returned -1. I haven't looked into what could cause it. Maybe the TLS was not initialized correctly?

2018-04-06T23:03:41-31db81d/linux-arm
2018-04-07T08:29:15-cd682f3/linux-arm
2018-04-09T19:41:38-53c8f61/linux-arm
2018-04-14T13:55:29-77faa65/linux-arm
2018-05-02T17:52:14-e8d417d/linux-arm
2018-05-06T13:53:47-260ae19/linux-arm
2018-05-30T13:40:36-e023d5b/linux-arm

2018-06-17T08:48:39-05f8b44/linux-arm
2018-06-20T18:43:22-ee9c939/linux-arm

https://build.golang.org/log/29aec0ed049250f35242081dc462c98fffe1deb6

https://build.golang.org/log/1b0026514ef9acdc1eb6a7f8cf57fafe8ca50122

https://build.golang.org/log/77b58ebc8c34b46297e53d249f465086e1d9f232

https://build.golang.org/log/29a5fd964dbff7a0be8f4253b7d82ed1a8129290

https://build.golang.org/log/314e99efd7e09337b33c9b7c4ef8af29801b62f3

Wow — the last non-trivial change to this test was for issue #6992 back in 2014!

@cherrymui, do you know who the right person to look into this further would be?

I'll look into this (as soon as I can, maybe not this week though).

First, I spent some time rediscovered what I found last year (#24758 (comment)). I should have read all the comments first...

Then, by attaching strace I found that when the test passes the TLS accesses are made on the main thread, whereas when it fails the TLS accesses are made on a non-main thread. (at least for all the runs I investigated.)

If I change the test so that it always run on a non-main thread, it fails 100%. The same change works fine on Linux/AMD64.

On all the test logs, the failed one is the one with static linking (https://go.googlesource.com/go/+/refs/tags/go1.13/src/cmd/dist/test.go#1072). Here, we build the C code with -fPIC, which will generate __tls_get_addr for TLS access. Then we link with -static. On AMD64, this will make the linker patch __tls_get_addr to static TLS access. This doesn't happen on ARM. From glibc source code (https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/arm/libc-tls.c;h=22ec3b0c194d0315dc87e73126a0f5c71241346e;hb=0f02b6cfc44af73d4d4363c46b3cbb18b8ff9171#l22), it claims that linker patching is not necessary. But it doesn't seem to be the case.

The following C program fails on ARM but works fine on AMD64:

#include <stdio.h>
#include <pthread.h>

static __thread int x;

void* nonmain() {
	printf("nonmain &x=%p\n", &x);
	printf("nonmain x=%d\n", x);
	return 0;
}

int main() {
	pthread_t t;

	printf("main &x=%p\n", &x);
	printf("main x=%d\n", x);
	pthread_create(&t, 0, nonmain, 0);
	pthread_join(t, 0);
	return 0;
}

# cc -fPIC tls.c -pthread -static
# ./a.out 
main &x=0x844dc
main x=0
nonmain &x=0x13
Segmentation fault (core dumped)

So, it seems that -fPIC and -static just don't work together on ARM...

BTW, if would be easier if the builder has gdb installed. Could we install gdb there?

Change https://golang.org/cl/196378 mentions this issue: misc/cgo/testtls: force TLS accesses on a non-main thread

Maybe we could work around this by letting the go command scan ldflags, and if it is linking statically on ARM, force recompiling all the C code in non-PIC mode?

@cherrymui, would you say this is a bug in the C compiler? If so, should we try upgrading the C compiler on the linux-arm builder?

Or, could we have misc/cgo/testtls probe for this bug explicitly (by, say, running the C test program) and skip the test (or suppress the error) if the test program fails?

Yeah, I think this is a bug in the C toolchain, probably either the linker or glibc. The compiler seems ok.

I looked at the source code of glibc and the linker (gold) at tip, and I didn't see anything changed that might fix this. I doubt upgrading the C toolchain would help.

https://build.golang.org/log/b7e22419cbcc25ccb723dc2533ddf4907ab2db50