For TestAtomicStop the test waits for one second, which seems reasonably generous. Any idea why that isn't long enough?

For TestAtomicStop the test waits for one second, which seems reasonably generous. Any idea why that isn't long enough?

Not sure yet, I will look at it more. It fails consistently on ppc64 but not ppc64le.

These two compile errors look similar and only happen on BE.
_testmain.go:12:19: error: incompatible type for field 2 in struct construction
12 | {"TestAddr", net.TestAddr},
| ^
Keeping gotest130557
FAIL: net
_testmain.go:9:24: error: incompatible type for field 2 in struct construction
9 | {"TestMainDeps", load.TestMainDeps},
| ^
Keeping gotest75366
FAIL: cmd/go/internal/load

This failure only happens on ppc64:
make archive/tar/check
--- FAIL: TestSplitUSTARPath (0.00s)
panic: strings: Repeat count causes overflow [recovered]
panic: strings: Repeat count causes overflow

goroutine 224 [running]:
panic
../../../src/libgo/go/runtime/panic.go:588
testing.tRunner..func1
../../../src/libgo/go/testing/testing.go:765
panic
../../../src/libgo/go/runtime/panic.go:535
strings.Repeat
../../../src/libgo/go/strings/strings.go:545
archive..z2ftar.TestSplitUSTARPath
/home/boger/gccgo.work/bld/powerpc64-linux/libgo/gotest5129/test/writer_test.go:949
testing.tRunner
../../../src/libgo/go/testing/testing.go:827
created by testing.T.Run
../../../src/libgo/go/testing/testing.go:878 +968
Keeping gotest5129
FAIL: archive/tar

Change https://golang.org/cl/152160 mentions this issue: cmd/go: add missing gccgo checks for buildmodeInit

There is a runtime failure in archive/tar on ppc64 (not ppc64le) gccgo that started with this revision:

r265710 | ian | 2018-10-31 23:21:16 -0500 (Wed, 31 Oct 2018) | 4 lines

	* go-gcc.cc (Gcc_backend::write_global_definitions): Don't call
	gimplify_function_tree.  Instead call allocate_struct_function if
	necessary.

This commit also causes a failure in net/http
make net/http/check

--- FAIL: TestProxyFromEnvironment (0.00s)
transport_test.go:2563: http_proxy="proxyURL: %v" req="http://example.com": got URL = "proxyurl: %v", want "http://127.0.0.1:8080"
--- FAIL: TestProxyFromEnvironmentLowerCase (0.00s)
transport_test.go:2578: http_proxy="proxyURL: %v" req="http://example.com": got URL = "proxyurl: %v", want "http://127.0.0.1:8080"
2018/12/06 10:15:05 http: TLS handshake error from 127.0.0.1:52354: read tcp 127.0.0.1:39326->127.0.0.1:52354: use of closed network connection
FAIL
FAIL: net/http
make: *** [net/http/check] Error
@ianlancetaylor Do you want a bugzilla for this?

No reason to open an entry in GCC bugzilla. It doesn't make this any likelier to get fixed.

It is bizarre that that change would have any minor effect like breaking a couple of tests. That change should either break everything or nothing. Did you bisect on all changes to GCC, or just changes to gcc/go and libgo?

That said that error looks like a confusion about data initialization. Or generally something very odd. The error message you show is impossible. It suggests that the array proxyFromEnvTests is initialized incorrectly, or modified after startup somehow, or that somehow the error report is printing the wrong string.

The bisection was done on all changes to gcc by the person who monitors the gcc-testresults, and I verified it.
In case I wasn't clear, the failure to archive/tar on ppc64 is displayed above and I think that one is happening on the initialization of the struct because the panic is in strings while doing a repeat. The top of that test looks like this:

func TestSplitUSTARPath(t *testing.T) {
        sr := strings.Repeat

        vectors := []struct {
                input  string // Input path
                prefix string // Expected output prefix
                suffix string // Expected output suffix
                ok     bool   // Split success?
        }{
                {sr("a", nameSize) + "/a", sr("a", nameSize), "a", true},
                {sr("a", prefixSize) + "/a", sr("a", prefixSize), "a", true},
                {sr("/", nameSize+1), sr("/", nameSize-1), "/", true},
                {sr("a", prefixSize) + "/" + sr("b", nameSize),
                        sr("a", prefixSize), sr("b", nameSize), true},
                {sr("a", prefixSize) + "//" + sr("b", nameSize), "", "", false},
                {sr("a/", nameSize), sr("a/", 77) + "a", sr("a/", 22), true},
        }

If I add a print statement to the top of the test before the struct is initialized, it passes.

I can try adding prints to the http test to see if the structure initialization is correct and if that changes they way it runs.

I looked more at os/signal and its intermittent failures. I found that it will fail regularly if I am on a system where a lot of others are doing builds and compiles. I did a profile on just the Atomic test and found that more than 50% is spent in backtrace.qsort. Not sure who is trying to generate a backtrace when running this test but that obviously is much more expensive than golang and causes the this to take much longer on gccgo.

Can we change the limit from 1 to 2 seconds all the time or should I add a check so it is only more for gccgo? I can make the change -- but does it go in golang upstream or gofrontend or both.

The test harness calls the backtrace code a couple of times. I'm surprised that it has that much effect, though. It's pretty bad if the backtrace library causes a program to take more than a second to start up, even if the system is heavily loaded.

If the time is in backtrace_qsort, then I suspect it's from sorting the ELF symbol table. On x86_64 GNU/Linux libgo.so has 142,626 symbols that get sorted (all STT_FUNC or STT_OBJECT symbols that are not SHN_UNDEF). I wonder whether ppc64 has more.

If that is the problem, then I note that in general Go doesn't actually need that information, as it is only used for backtrace_syminfo which Go doesn't normally call. I wonder if we should change libbacktrace to only sort the ELF symbols on demand.

But looking at this more closely, I'm not sure that it could be the backtrace library. The 1 second timeout here is in the loop in atomicStopTestProgram. Nothing in that loop calls into the backtrace library. The backtrace calls would have already happened by the time we enter the loop.

Still, if bumping up the timeout to 2 seconds means that the program always passes then I guess we may as well do that, in both libraries.

We have 3 failures that started with 264546, which was the upgrade to go 1.11. If you plan to upgrade to go 1.12 we could just want and see what happens after that.

Two of those are compile failures that only happens on ppc64 in net and cmd/go/internal/local as I show above.
One is a failure in TestAbort that only happens on ppc64le (not sure if it gets run on ppc64) when running 'go test runtime' under gotools. The failure happens because runtime.abort is not in the panic stacktrace.

=== RUN TestAbort
--- FAIL: TestAbort (0.10s)
crash_test.go:95: testprog Abort exit status: exit status 2
crash_test.go:681: output does not contain "runtime.abort":
SIGABRT: abort
PC=132077898164620 m=0 sigcode=18446744073709551610

    goroutine 1 [running]:
    runtime.sighandler
            /home/boger/gccgo.work/trunk/bld/../src/libgo/go/runtime/signal_sighandler.go:145
    runtime.sigtrampgo
            /home/boger/gccgo.work/trunk/bld/../src/libgo/go/runtime/signal_unix.go:314
    runtime.sigtramp
            /home/boger/gccgo.work/trunk/bld/../src/libgo/runtime/go-signal.c:131

            :0
    gsignal
            :0

......

More detail on the archive/tar test failure on ppc64 shown earlier in this issue.
The failure happens as shown above in TestSplitUSTARPath with the above output.
Interestingly, adding a print statement to the top of the test allows it to pass.

In the case where it works, both calls to strings.Repeat pass the constant value 2 as the length of "a/" which looks correct.

Here is code from the objdump of the test binary from 265708.

    10036dc0:   38 a0 00 02     li      r5,2 <---- length of "a/"
    10036dc4:   3d e2 ff fc     addis   r15,r2,-4
    10036dc8:   7d 49 03 a6     mtctr   r10
    10036dcc:   39 ef 53 b8     addi    r15,r15,21432
/home/boger/gccgo.work/bld/powerpc64-linux/libgo/gotest29555/test/writer_test.go:948
                {sr("a", prefixSize) + "//" + sr("b", nameSize), "", "", false},
    10036dd0:   f8 e1 09 10     std     r7,2320(r1)
    10036dd4:   f9 01 09 18     std     r8,2328(r1)
/home/boger/gccgo.work/bld/powerpc64-linux/libgo/gotest29555/test/writer_test.go:949
                {sr("a/", nameSize), sr("a/", 77) + "a", sr("a/", 22), true},
    10036dd8:   7d e4 7b 78     mr      r4,r15 <--- address of "a/"
    10036ddc:   e8 59 00 08     ld      r2,8(r25)
    10036de0:   4e 80 04 21     bctrl  <---- first call to strings.Repeat with 2 in r5
    10036de4:   e8 41 00 28     ld      r2,40(r1)
    10036de8:   e9 59 00 00     ld      r10,0(r25)
    10036dec:   e8 a1 08 f0     ld      r5,2288(r1)
    10036df0:   e8 c1 08 f8     ld      r6,2296(r1)
    10036df4:   7d e4 7b 78     mr      r4,r15 <---- same address of "a/" as previous call
    10036df8:   7f cb f3 78     mr      r11,r30
    10036dfc:   38 61 08 f0     addi    r3,r1,2288
    10036e00:   7d 49 03 a6     mtctr   r10
    10036e04:   f8 a1 09 60     std     r5,2400(r1)
    10036e08:   f8 c1 09 68     std     r6,2408(r1)
    10036e0c:   38 a0 00 02     li      r5,2   <---- length of "a/"
    10036e10:   38 c0 00 4d     li      r6,77
    10036e14:   e8 59 00 08     ld      r2,8(r25)
    10036e18:   4e 80 04 21     bctrl  <---- second call to strings.Repeat with 2 in r5

Here is the case where it doesn't work. The value passed in r5 to strings.Repeat is not 2 but some strange value, causing the panic when verifying the len and count.

           {sr("a/", nameSize), sr("a/", 77) + "a", sr("a/", 22), true},
10036c7c:   38 a0 00 02     li      r5,2
10036c80:   3f a2 ff fc     addis   r29,r2,-4
10036c84:   7d 69 03 a6     mtctr   r11
10036c88:   3b bd 4e 30     addi    r29,r29,20016
10036c8c:   7f 4b d3 78     mr      r11,r26
10036c90:   7f a4 eb 78     mr      r4,r29
10036c94:   e8 5b 00 08     ld      r2,8(r27)
10036c98:   4e 80 04 21     bctrl  <----- first call to strings.Repeat, passing 2 in r5
10036c9c:   e8 41 00 28     ld      r2,40(r1)
10036ca0:   e9 7b 00 00     ld      r11,0(r27)
10036ca4:   e9 21 05 a8     ld      r9,1448(r1)
10036ca8:   e9 01 05 a0     ld      r8,1440(r1)
10036cac:   38 61 05 a0     addi    r3,r1,1440
10036cb0:   38 c0 00 4d     li      r6,77
10036cb4:   3d 42 ff fc     addis   r10,r2,-4
10036cb8:   7d 69 03 a6     mtctr   r11
10036cbc:   39 4a fd c0     addi    r10,r10,-576 <--- not the same address of "a/" as in previous call
10036cc0:   f9 21 05 d8     std     r9,1496(r1)
10036cc4:   f9 01 05 d0     std     r8,1488(r1)
10036cc8:   7f 4b d3 78     mr      r11,r26
10036ccc:   e8 8a 00 00     ld      r4,0(r10) <---- this is not the same either
10036cd0:   e8 aa 00 08     ld      r5,8(r10)  <---- this is loading a bad value in r5, not 2
10036cd4:   e8 5b 00 08     ld      r2,8(r27)
10036cd8:   4e 80 04 21     bctrl   <------ This call to strings.Repeat gets the panic due to value in r5.

Change https://golang.org/cl/153879 mentions this issue: os/signal: increase deliver time for signal testcase

Change https://golang.org/cl/153826 mentions this issue: os/signal: increase wait time for signal delivery time in testcase

I did some debugging on the failing test in net/http. This one consistently fails at runtime on ppc64 and ppc64le on every system (power7 & power8). Both this one and the failure in archive/tar are related to initializing structures with constant strings.

The failure happens in TestProxyFromEnvironment because the structure for tt from proxyFromEnvTests has been initialized incorrectly. I did various experiments and found that adding print statements will allow it to work. It will fail when processing the 5th entry from the list in proxyFromEnvTest because the string for env is wrong, apparently a constant initialization error -- it is not pointing to the correct string according to gdb at the point it is trying to pass onto the testProxyForRequest function, which can be seen in the error output. (Possibly because the same constant appears twice in the structure and at compile time it is supposed to find the same one but doesn't? Adding print statements will add more string constants and that somehow affects it? My theory.) I tried removing entries from proxyFromEnvTests and found that I had to have the first 5 entries to make it fail. I tried building the test with -O0 so it is not an optimization issue as far as the test is concerned.

Thanks for looking into this. Can you use that information to build a standalone reproduction?

I'll see what I can do.

I was unable to reproduce the problem in net/http in a smaller testcase. Any minor change allows it to pass. So I decided to wait until the Go 1.12 front end update, and I was able to build with that patch yesterday. With the new frontend, net/http no longer fails on ppc64le, and it fails on ppc64 but in a different way. I will verify it behaves the same on multiple systems/distros before posting that one.

The following failure occur in gotools before and after the update to Go 1.12:

                === gotools tests ===

Running cmd/go
cd check-go-dir/src/cmd/go && PATH=/home/boger/gccgo.work/trunk/bld/gotools:/home/boger/golang/elliptic/go/bin:/home/boger/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games GCCGO='/home/boger/gccgo.work/trunk/bld/gotools/check-gccgo' CC='/home/boger/gccgo.work/trunk/bld/gotools/check-gcc' GCCGOTOOLDIR='/home/boger/gccgo.work/trunk/bld/gotools' GO_TESTING_GOTOOLS=yes LD_LIBRARY_PATH=/home/boger/gccgo.work/trunk/bld/powerpc64le-linux/libgo/.libs GOROOT=/home/boger/gccgo.work/trunk/bld/powerpc64le-linux/libgo GOCACHE=/home/boger/gccgo.work/trunk/bld/gotools/gocache-test GOPATH=/home/boger/gccgo.work/trunk/bld/gotools/check-go-dir /home/boger/gccgo.work/trunk/bld/gotools/go test -test.short -test.timeout=480s -test.v
# cmd/go.test
/home/boger/gccgo.work/trunk/bld/gotools/check-go-dir/src/cmd/go/script_test.go:283: error: call lacks nop, can't restore toc; recompile with -fPIC
collect2: error: ld returned 1 exit status
FAIL    cmd/go [build failed]
FAIL: go test cmd/go

This is additional information on the failure in archive/tar that currently only happens on ppc64 but I believe it is the same problem that was previously happening in net/http on ppc64le before the recent gofrontend update. It seems to be random in nature because slight changes to the testcase can make it work, so I have been unable to make it work in a small testcase. (If I remove other tests in the file it will work.) So even though it only occurs on ppc64 I believe it could happen on ppc64le.

I showed the generated code above to see where it goes wrong. Below is the gimple.

It is the middle argument that is wrong based on the objdump above. In the failing case, there is a VIEW_CONVERT_EXPR inserted, but that doesn't appear in the gimple when it works.

In the case where it fails:

      _17 = MEM[(void * *)sr];
      _18 = (struct  (*<T674>) (struct , int)) _17;
      D.24366 = *.LC764;
      GOTMP.5006 = _18 (D.24366, 100); [static-chain: sr] [return slot optimization]
      _19 = MEM[(void * *)sr];
      _20 = (struct  (*<T674>) (struct , int)) _19;
      D.24368 = VIEW_CONVERT_EXPR<struct
      {
        uint8 * __data;
        int __length;
      }>(*.LC149);
      GOTMP.5007 = _20 (D.24368, 77); [static-chain: sr] [return slot optimization]
      _21 = MEM[(void * *)sr];
      _22 = (struct  (*<T674>) (struct , int)) _21;
      D.24370 = *.LC764;
      GOTMP.5008 = _22 (D.24370, 22); [static-chain: sr] [return slot optimization]
      D.24371 = *.LC760;

In the case where it passes:

      D.24358 = *.LC763;
      GOTMP.5004 = _14 (D.24358, 100); [static-chain: sr] [return slot optimization]
      _15 = MEM[(void * *)sr];
      _16 = (struct  (*<T674>) (struct , int)) _15;
      D.24360 = *.LC763;
      GOTMP.5005 = _16 (D.24360, 77); [static-chain: sr] [return slot optimization]
      _17 = MEM[(void * *)sr];
      _18 = (struct  (*<T674>) (struct , int)) _17;
      D.24362 = *.LC763;
      GOTMP.5006 = _18 (D.24362, 22); [static-chain: sr] [return slot optimization]
      D.24363 = *.LC765;
      GOTMP.5007 = D.24363;

In writer_test.go, at the top of TestSplitUSTAR this works:

func TestSplitUSTARPath(t *testing.T) {
        sr := strings.Repeat

        vectors := []struct {
                input  string // Input path
                prefix string // Expected output prefix
                suffix string // Expected output suffix
                ok     bool   // Split success?
        }{
                {sr("a", prefixSize) + "/" + sr("b", nameSize),
                        sr("a", prefixSize), sr("b", nameSize), true},
                {sr("a", prefixSize) + "//" + sr("b", nameSize), "", "", false},
                {sr("a/", nameSize), sr("a/", 77) + "a", sr("a/", 22), true},
        }

This one fails:

func TestSplitUSTARPath(t *testing.T) {
        sr := strings.Repeat

        vectors := []struct {
                input  string // Input path
                prefix string // Expected output prefix
                suffix string // Expected output suffix
                ok     bool   // Split success?
        }{
                {sr("/", nameSize+1), sr("/", nameSize-1), "/", true},
                {sr("a", prefixSize) + "/" + sr("b", nameSize),
                        sr("a", prefixSize), sr("b", nameSize), true},
                {sr("a", prefixSize) + "//" + sr("b", nameSize), "", "", false},
                {sr("a/", nameSize), sr("a/", 77) + "a", sr("a/", 22), true},
        }

gccgo builds are broken on ppc64le, they get an ICE due to a change from yesterday. It is documented here https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89368 @ianlancetaylor

There are a lot of different unrelated problems here. As far as I can tell all the test failures are passing now. If there are still problems, let's open a new issue. Thanks.