CC @thanm

Would it be sufficient to require a host linker when using this flag? Then the host linker could be in charge of writing out the archive in the standard format, and Go tools wouldn't have to know those details.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Most applications large enough to be running into these issues (e.g. binary with DWARF is "too big") are already using external linking due to the use of CGO in a dependency somewhere. When I do a build of k8s with "ldflags=-v", I can see that the external linker is already being invoked by default for a regular build. Given that k8s is already carefully curating linker flags, it seems that the easiest path forward would be to just pass the proper options to the host linker, as Russ suggests.

Great. Doing it via the host linker sounds like a small amount of work.
If we did need to do the bigger amount of work for internal linking we might want to reconsider though.

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

Separate .debug files are just elf (or mach-o) files so, in theory, the linker already knows how to write them. Letting the host linker do it is of course less work. Doing .pdb is much more complicated since the format is different and (afaik) only partially documented.

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc Doing this via the host linker would be great, but as far as I can tell neither GNU ld nor llvm's lld know how to generate external debuginfo files in a single pass, so you can't just use go tool link's -extldflags to pass a flag to delegate the work to the host linker transparently.

C and C++ builds will usually compile and link an integrated binary with full debuginfo and symbols, then post-process it to produce a stripped release binary and an external debuginfo archive.

gcc and clang also have the -gsplit-dwarf flag which causes the compilation phase to emit .dwo files alongside each .o file. Then the binutils dwp command or llvm-dwp can create easily distributed DWARF-v5 debuginfo archives from the .dwo files. See https://gcc.gnu.org/wiki/DebugFission . This article has a handy demo https://www.productive-cpp.com/improving-cpp-builds-with-split-dwarf/ . The result is different to a regular external debuginfo archive and requires support in the tools loading symbols, but is widely supported by ELF symbol loaders already. See also https://dwarfstd.org/issues/140421.1.html

In any case I don't see how this can be done with external linker flags alone.

I suspect that to implement this, either:

• cmd/link would need to process the temp result binary to generate a stripped binary and optional external debuginfo in a similar manner to how binaries are split into stripped binary and external debuginfo using llvm's dsymutil or using llvm-objcopy, eu-strip or objcopy; or
• cmd/compile would need to emit .dwo files that could be aggregated into a DWARFv5 debuginfo archive for distribution, initially using dwp or llvm-dwp, and later by a golang native tool.

It'd be good to make this easy for the golang ecosystem to adopt, so it becomes normal to publish debug symbols alongside release builds on github artifacts etc, and ideally even publish them on an elfutils debuginfod or llvm-debuginfod server for on-demand fetching. Depending on CGO makes that harder.

A lot of useful info on this topic can be found at https://maskray.me/blog/2022-10-30-distribution-of-debug-information

Sorry for the delayed response by the way. I tend to be flooded with noisy github notifications and it's hard to find the relevant ones.

@thanm Are you aware of a set of linker flags that'll produce external debuginfo in a single pass? I'd love to do it that way, at least for now, but am not aware of a way to do so.

cmd/link would need to process the temp result binary to generate a stripped binary and optional external debuginfo in >a similar manner to how binaries are split into stripped binary and external debuginfo using
llvm's dsymutil or using llvm-objcopy, eu-strip or objcopy

Yes, this is what I had in mind. In external linking the Go linker prepares an intermediate object file (e.g. "go.o") that it then hands off to the external linker as a final step. Before doing that it seems reasonable to run llvm-objcopy (or equivalent) to split off the debug info to a separate file and then pass the striped go.o to the external linker.

Implementing full split DWARF would obviously be more work. I don't think we would need to touch the compiler to do that, but it would require some changes in the linker.

I'm a little late here, just discovered this proposal.

I would prefer to keep PDB generation out of this proposal. If someday we support generating PDB files, then passing -external-debuginfo will be redundant, as PDB files can't be stored in the binary, and the user intent won't be clear: should we generate a PDB file or a split DWARF (Windows would supports both).

Having said this, would it make sense to change the flag name to -external-dwarf, or something else more DWARF-centric?