Builders on linux/ppc64{,le} are also failing since commit bd83774 (which corresponds to https://golang.org/cl/111682):

https://build.golang.org/log/69ba9e8f2e58f832e1f045a76e932a5b3c1d2eb4
https://build.golang.org/log/7b67595448396001c5101ffa86e2d6cac21946c4

/cc @aclements @ianlancetaylor @heschik

...and the linux/arm64, linux/arm, linux/mips, linux/s390x builders fail as well, though not with a segfault during bootstrap but during various tests:

linux/arm: https://build.golang.org/log/ce0ac926b3379a141c368e72328120ff16e75928
linux/arm64: https://build.golang.org/log/36143a71f0998633c7bd61705d38e301e2450ae9
linux/mips: https://build.golang.org/log/2647b5b5501fbe006587658f098778fcc16d1a02
linux/s390x: https://build.golang.org/log/981714c0a467c108dbdf281a531f72a04d7aed92

CC @aclements @heschik

If I look at the readelf output for go_bootstrap after this change, the section names are garbage. Otherwise the rest looks the same.

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0] ^I^J^K^L<95>^Q       NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] ^J^K^L<95>^Q         PROGBITS         0000000000011000  00001000
       00000000002497c0  0000000000000000  AX       0     0     16
  [ 2]                   PROGBITS         0000000000260000  00251000
       00000000001031c0  0000000000000000   A       0     0     32
  [ 3] ^E^E^E^Fé^Q       STRTAB           0000000000000000  003531c0
       00000000000000fa  0000000000000000           0     0     1
  [ 4]                   PROGBITS         00000000003632c0  003542c0
       00000000000022e4  0000000000000000   A       0     0     32
  [ 5]                   PROGBITS         00000000003655a8  003565a8
       00000000000008b8  0000000000000000   A       0     0     8
  [ 6] ^C^C^C^C^C^C^C^C¨ PROGBITS         0000000000365e60  00356e60
       0000000000000000  0000000000000000   A       0     0     1
  [ 7]                   PROGBITS         0000000000365e60  00356e60
       00000000001921ff  0000000000000000   A       0     0     32
  [ 8]                   PROGBITS         0000000000500000  004f1000
       0000000000011e60  0000000000000000  WA       0     0     32
  [ 9] ^D^D^D^E^G^G<9c>^Q   PROGBITS         0000000000511e60  00502e60
       000000000000df90  0000000000000000  WA       0     0     32
  [10] <9c>^Q               NOBITS           000000000051fe00  00510e00
       0000000000024730  0000000000000000  WA       0     0     32
  [11]                   NOBITS           0000000000544540  00535540
       0000000000002978  0000000000000000  WA       0     0     32
  [12] ^B^B^B^B^C^C^D^D  PROGBITS         0000000000550000  00511000
       00000000000001d3  0000000000000000           0     0     1
  [13] ^KÐ^Q             PROGBITS         00000000005501d3  005111d3
       000000000007a448  0000000000000000           0     0     1
  [14]                   PROGBITS         00000000005ca61b  0058b61b
       000000000004bdc4  0000000000000000           0     0     1
  [15] ^B^B^B^B^B^B^B^C  PROGBITS         00000000006163df  005d73df
       0000000000017d2f  0000000000000000           0     0     1
  [16] ^D^D^F^G^H^J^J^J  PROGBITS         000000000062e10e  005ef10e
       0000000000021482  0000000000000000           0     0     1
  [17]                   PROGBITS         000000000064f590  00610590
       000000000000003b  0000000000000000           0     0     1
  [18] ^G^H^J^PÅ^Q       PROGBITS         000000000064f5cb  006105cb
       0000000000156fe7  0000000000000000           0     0     1
  [19] ^C^D^E^E^G^G^H^I  PROGBITS         00000000007a65b2  007675b2
       0000000000183b8f  0000000000000000           0     0     1
  [20]                   PROGBITS         000000000092a141  008eb141
       000000000008dbb0  0000000000000000           0     0     1
  [21] ^E^E^E^E^E^E^F¯^Q SYMTAB           0000000000000000  00980000
       0000000000034cf8  0000000000000018          22   393     8
  [22] ^Q                STRTAB           0000000000000000  009b4cf8
       000000000003c0c3  0000000000000000           0     0     1

How could this pass the trybots for ppc64x?

I think there are currently no trybots for linux/ppc64x

/cc @bradfitz

Last time I asked, I was told there was some compile-only testing done, but obviously not since in this case the toolchain doesn't even build.

@laboger, @tklauser, we do indeed compile for ppc64 on trybots, but it's cross-compiled from Linux, not run on real ppc64 hardware, which seems to be the issue here. Looks like it does cross-compile fine from Linux.

The definition from x/build/dashboard/builders.go is:

        addMiscCompile := func(suffix, rx string) { 
                addBuilder(BuildConfig{ 
                        Name:        "misc-compile" + suffix, 
                        HostType:    "host-linux-jessie", 
                        TryBot:      true, 
                        TryOnly:     true, 
                        CompileOnly: true, 
                        Notes:       "Runs buildall.sh to cross-compile std packages for " + rx + ", but doesn't run any tests.",
                        allScriptArgs: []string{
                                // Filtering pattern to buildall.bash:                                                                                                                
                                rx,
                        },
                })
        }
        addMiscCompile("", "^(linux-arm64|linux-s390x|solaris-amd64|darwin-386)$") // 4 ports                                                                                         
        addMiscCompile("-mobile", "(^android|darwin-arm64)")                       // 5 ports                                                                                         
        addMiscCompile("-nacl", "^nacl")                                           // 3                                                                                               
        addMiscCompile("-mips", "^linux-mips")                                     // 4                                                                                               
        addMiscCompile("-ppc", "^linux-ppc64")                                     // 2                                                                                               
        addMiscCompile("-plan9", "^plan9-")                                        // 3                                                                                               
        addMiscCompile("-freebsd", "^freebsd-")                                    // 3                                                                                               
        addMiscCompile("-netbsd", "^netbsd-")                                      // 3                                                                                               
        addMiscCompile("-openbsd", "^openbsd-")                                    // 3                                                                                               

Again, the goal of TryBots is to be fast and have decent coverage to catch most mistakes, but not all.

Cross compiling doesn't crash, but the resulting binaries are still broken. It'd be hard to know that without running them though.

The most obvious problem is that the location of .shstrtab is wrong, which is why the section names are gibberish. In the hello world program I built, .shstrtab in the ELF headers:

  [ 3]                   STRTAB           0000000000000000  000d3d00
       0000000000000193  0000000000000000           0     0     1

But looking at the actual binary, the correct location is 0xd4d00. I'll poke around for my own curiosity but Austin is probably better suited for this.

Actually the whole section table appears to be wrong. Before this CL for the cmd/go binary "There are 37 section headers, starting at offset 0x270", after this CL "There are 24 section headers, starting at offset 0x1c8". And the section header types don't match up at all (which you would expect if it was just the names that were broken).

Okay. I was able to fix the problems with the section headers, As always with the linker I'm using a mix of guesswork and brute force, but I think the immediate problem is in datoff(). It only looks at Segtext and Segdata, but .shstrtab is in Segrodata. On amd64, rodata comes right after text (other than a bit of padding) and so that doesn't matter, but on arm64 and ppc64le there seems to be a sizeable gap for some reason. Changing datoff() to check against Segrodata fixes the section headers:

	for _, seg := range []*sym.Segment{&Segdwarf, &Segdata, &Segrodata, &Segtext} {
		if uint64(addr) >= seg.Vaddr {
			return int64(uint64(addr) - Segdata.Vaddr + Segdata.Fileoff)
		}
	}

but probably the bigger problem is the gap. I presume it's new after Austin's CL, though I didn't check.

Austin, I think you were probably external linking and the external threw away a bunch of sections that it didn't know what to do with? I only see 24 sections in my binaries.

FWIW, when I originally wrote the patch I verified that binaries were identical on all supported OSes and arches. Either I did that wrong (certainly possible) or something happened in the rebase to master.

@heschik, you're right that it looked completely different for me because I was external linking.

You're also right that there are some gaps that were added by my CL. In some cases (the strings test binary on linux/mips64) the binaries are even different sizes.

The gaps are happening because previously we computed Fileoff as round(seg.Vaddr) - prev.Vaddr + prev.Fileoff and now we're computing it as round(seg.Vaddr - prev.Vaddr) + prev.Fileoff. The previous computation was "more correct" in the sense the it applied the rounding to virtual addresses rather than file offsets, but of course the point of the change was to get away from depending on the virtual address layout to match the file layout.

Change https://golang.org/cl/118716 mentions this issue: cmd/link: separate virtual address layout from file layout