CC @golang/s390x, @golang/compiler.

I'll take a look, most likely a bit later this week.

Thanks. I should have added that the error is about the manager Go executable (from which the core was produced) and that also looks okay:

# readelf -a manager | head
ELF Header:
  Magic:   7f 45 4c 46 02 02 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, big endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           IBM S/390
  Version:                           0x1

Hi @kgibm , I spent a little while exploring this issue.

I have limited access to our s390x machine (no ssh, only gomote commands), and our machine does not have "gcore", but I did get as far as creating a core dump using a toy program and then running 'viewcore' on the dump.

The good news is that viewcore was able to read the DWARF info for my toy program; the bad news is that it ran into problems later on (crash due to a different error of some sort). The crash I am seeing is

goroutine 1 [running]:
golang.org/x/debug/internal/core.(*Thread).SP(...)
	/data/golang/workdir/xxx/debug/internal/core/thread.go:32
golang.org/x/debug/internal/gocore.(*Process).readG(0xc000394000, {0xc000394000, 0xc000002340, 0xc000124b40})
	/data/golang/workdir/xxx/debug/internal/gocore/process.go:688 +0xa7e
golang.org/x/debug/internal/gocore.(*Process).readGs(0xc000394000)
	/data/golang/workdir/xxx/debug/internal/gocore/process.go:648 +0x118
golang.org/x/debug/internal/gocore.Core(0xc00038a000)
	/data/golang/workdir/xxx/debug/internal/gocore/process.go:173 +0x4f4
main.readCore()
	/data/golang/workdir/xxx/debug/cmd/viewcore/main.go:277 +0xc2
main.runMappings(0x17dd9a0, {0xc000098f20, 0x0, 0x1})
	/data/golang/workdir/xxx/debug/cmd/viewcore/main.go:406 +0x2a

which is fairly early in the coredump reading process, it's having trouble making sense of the "g" structure for one of the program's goroutines.

Viewcore is only sporadically/sparsely maintained, so it is entirely possible that something has changed in the runtime datastructures in the 1.21 timeframe that is throwing it off.

Some questions for you:

• what sort of output do you see from the command

  readelf --debug-dump=info manager | head -40

• in your crash the DWARF reader is reporting a zero offset for the .debug_info section; is this section actually present in your binary? what do you see when you run the command

  readelf --wide -S manager | fgrep debug_info

Thanks.

@thanm

• what sort of output do you see from the command
  readelf --debug-dump=info manager | head -40

Empty

• in your crash the DWARF reader is reporting a zero offset for the .debug_info section; is this section actually present in your binary? what do you see when you run the command
  readelf --wide -S manager | fgrep debug_info

Empty

I guess this explains my proximate issue. The executable also notes "stripped" which I presume is another symptom of lacking DWARF information:

$ file manager
manager: ELF 64-bit MSB executable, IBM S/390, version 1 (SYSV), statically linked, Go BuildID=Kz1lgHB4w_kxUOvlQqtu/tq5tP0TciDudwbRV8f6M/f7kzpDaG_HNG7APeD9w6/3cdjWHx8fvnsL3NCm5MQ, stripped

I will pass this on to the team producing the package. This is based on the Kubernetes operator-lifecycle-manager so I'm not sure if OLM is doing this or our build scripts.

If you have any tips on how to check what is causing the stripping and/or how to maximize symbols, that would be helpful.

Thanks!

Yup, does indeed look like a stripped executable.

Kubernetes for a number of other big binaries like kubelet adds Go linker options to strip things that aren't explicitly being used for debugging (if I recall correctly). E.g.

https://github.com/kubernetes/kubernetes/blob/3c268b752448f66ba8338cb62ff9a4a14f77873b/hack/lib/golang.sh#L887

So I think the thing to do would be to check if the build is being done with go build -o manager ... -ldflags="-s -w" (where "-s" strips symbols and "-w" strips DWARF). HTH.

Yes, confirmed we are using go build -ldflags="-s -w". We'll re-consider. This explains the issue so I'll close it out. For now, we'll switch to debug.WriteHeapDump on SIGUSR1 to produce a heapdump instead of trying to use viewcore. Thanks.