Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: compress debug info #11799

Closed
josharian opened this issue Jul 20, 2015 · 44 comments
Closed

cmd/link: compress debug info #11799

josharian opened this issue Jul 20, 2015 · 44 comments

Comments

@josharian
Copy link
Contributor

Compressing our debug info might offer a significant, cheap file size win.

Things to do:

  • Figure out which targets to compress debug info for by default.
  • Provide a linker flag to control compression on/off.
  • Teach the linker to compress debug info.
  • Teach the Go packages how to read compressed debug info.

@ianlancetaylor commented off-list:

Compressed debug info is supported by gdb and the GNU binutils.
Yes, we should do it in our toolchain for relevant targets.

Related: #11773

@josharian josharian added this to the Unplanned milestone Jul 20, 2015
@mwhudson
Copy link
Contributor

It's only tangentially related, but it's been at the back of my mind to try to figure out what would be required to make stripping Go binaries (and so being able to use detached debugging symbols) more of a supported thing.

@josharian
Copy link
Contributor Author

cc @crawshaw @derekparker

I think that this could reduce the size of cmd/go by about 15%.

How I got that number (OS X commands):

$ dwarfdump -R `which go`
[snip]
Segments
Segment Name     vmaddr           vmsize           fileoff          filesize         maxprot  initprot nsects   flags
---------------- ---------------- ---------------- ---------------- ---------------- -------- -------- -------- --------
__PAGEZERO       0000000000000000 0000000000001000 0000000000000000 0000000000000000 00000000 00000000 00000000 00000000
__TEXT           0000000000001000 00000000007bf000 0000000000000000 00000000007bf000 00000007 00000005 00000006 00000000
__DATA           00000000007c0000 0000000000045060 00000000007bf000 0000000000020ac0 00000003 00000003 00000005 00000000
__LINKEDIT       0000000000806000 0000000000083c4c 0000000000ac6000 0000000000083c4c 00000007 00000003 00000000 00000000
__DWARF          0000000000805060 0000000000000000 00000000007e0000 00000000002e5968 00000000 00000000 00000008 00000000

Sections
Section Name     Segment Name     addr             size             offset   align    reloff   nreloc   flags    reserv1  reserv2  reserv3  size     size %
---------------- ---------------- ---------------- ---------------- -------- -------- -------- -------- -------- -------- -------- -------- ======== ======
__text           __TEXT           0000000000002000 00000000003a6f38 00001000 00000004 00000000 00000000 00000400 00000000 00000000 00000000    3.65M 33.49%
__rodata         __TEXT           00000000003a8f40 0000000000283f0c 003a7f40 00000005 00000000 00000000 00000000 00000000 00000000 00000000    2.52M 23.06%
__typelink       __TEXT           000000000062ce50 000000000000e7f0 0062be50 00000003 00000000 00000000 00000000 00000000 00000000 00000000   57.98K  0.52%
__gosymtab       __TEXT           000000000063b640 0000000000000000 0063a640 00000000 00000000 00000000 00000000 00000000 00000000 00000000    0      0.00%
__gopclntab      __TEXT           000000000063b640 000000000018415b 0063a640 00000005 00000000 00000000 00000000 00000000 00000000 00000000    1.52M 13.90%
__symbol_stub1   __TEXT           00000000007bf7a0 00000000000000d8 007be7a0 00000005 00000000 00000000 80000408 00000000 00000006 00000000  216      0.00%
__nl_symbol_ptr  __DATA           00000000007c0000 0000000000000138 007bf000 00000002 00000000 00000000 00000006 00000024 00000000 00000000  312      0.00%
__noptrdata      __DATA           00000000007c0140 0000000000014008 007bf140 00000005 00000000 00000000 00000000 00000000 00000000 00000000   80.01K  0.72%
__data           __DATA           00000000007d4160 000000000000c960 007d3160 00000005 00000000 00000000 00000000 00000000 00000000 00000000   50.34K  0.45%
__bss            __DATA           00000000007e0ac0 000000000001e138 00000000 00000005 00000000 00000000 00000001 00000000 00000000 00000000  120.30K  1.08%
__noptrbss       __DATA           00000000007fec00 0000000000006460 00000000 00000005 00000000 00000000 00000001 00000000 00000000 00000000   25.09K  0.22%
__debug_abbrev   __DWARF          0000000000805060 00000000000000ff 007e0000 00000000 00000000 00000000 02000000 00000000 00000000 00000000  255      0.00%
__debug_line     __DWARF          000000000080515f 00000000000ac904 007e00ff 00000000 00000000 00000000 02000000 00000000 00000000 00000000  690.25K  6.18%
__debug_frame    __DWARF          00000000008b1a63 000000000005c574 0088ca03 00000000 00000000 00000000 02000000 00000000 00000000 00000000  369.36K  3.31%
__debug_info     __DWARF          000000000090dfd7 0000000000154b49 008e8f77 00000000 00000000 00000000 02000000 00000000 00000000 00000000    1.33M 12.20%
__debug_pubnames __DWARF          0000000000a62b20 000000000005f4c1 00a3dac0 00000000 00000000 00000000 02000000 00000000 00000000 00000000  381.19K  3.41%
__debug_pubtypes __DWARF          0000000000ac1fe1 000000000002897f 00a9cf81 00000000 00000000 00000000 02000000 00000000 00000000 00000000  162.37K  1.45%
__debug_aranges  __DWARF          0000000000aea960 0000000000000030 00ac5900 00000000 00000000 00000000 02000000 00000000 00000000 00000000   48      0.00%
__debug_gdb_scri __DWARF          0000000000aea990 0000000000000035 00ac5930 00000000 00000000 00000000 02000000 00000000 00000000 00000000   53      0.00%
$ # manually convert fileoff and filesize from hex to dec, calculate rough compressed dwarf size
$ head -c 8257536 `which go` | tail -c 3037544 | gzip | wc -c
 1048123
$ # find current binary size
$ wc -c `which go`
 11836492
$ # calculate savings as a percent of current size
$ python -c "print(100*(3037544-1048123)/11836492.)"
16.807522026

I also double-checked that I'm interpreted fileoff and filesize correctly by using otool and xxd to dump just the __debug_info section and compressed it; the compression ratio for __debug_info is even better than for the entirety of the dwarf segment.

Note that this probably includes updating the dwarf stdlib to handle compression.

See also #5158.

@minux
Copy link
Member

minux commented Mar 3, 2016 via email

@josharian
Copy link
Contributor Author

I took yet another stab at this, but I am having a hard time making friends with the linker, so I'm going to leave this for @aarzilli or @heschik.

Here are some very useful implementation details from @ianlancetaylor, copied from #20463:


DWARF compression is done at the file container level. There are two ways to do it. The older way is to prefix the DWARF section name with a 'z', as in '.zdebug_info'. The newer way, which applies to sections of all types, not just DWARF sections, is to set the SHF_COMPRESSED flag in the ELF section flags.

Clearly Mach-O doesn't support the SHF_COMPRESSED flag, but I don't see any reason that it wouldn't support .zdebug_*. Looking at the gdb source code, it seems that it ought to work. I suppose the only way to find out is to try it and see what happens.

A .zdebug-* section starts with the four characters "ZLIB" followed by a uint64 of the uncompressed size in big-endian order followed by the compressed data.

SHF_COMPRESSED works differently. The section data starts with a compression header: a 32-bit word with the compression type (1 for ZLIB), on 64-bit systems a 32-bit padding word, a 32-bit or 64-bit word with the uncompressed section size, a 32-bit or 64-bit word with the uncompressed alignment.

@heschi
Copy link
Contributor

heschi commented Jul 14, 2017

I've spent a little time digging into this and I'll leave some notes for myself or whoever comes next.

The linker doesn't have the .debug_info section as a whole unit in memory at any point. dwarfp contains each top-level DIE as an individual symbol, and they're written to the output file individually by Dwarfblk in data.go. So that's the only place that we can insert compression of the whole section, perhaps by checking a new flag in the Section type.

This won't work for external linking, because the .debug_info section contains relocations to text symbols that will be placed by the external linker. New versions of GNU ld (I'm not sure when it was added exactly) support --compress-debug-sections to compress debug sections passed to. (gcc has another flag -gz, but presumably that's just for the individual CUs it's creating, not the whole section. That doesn't help with the Go CUs.)

@ianlancetaylor
Copy link
Contributor

I don't think we should compress the debug info when doing external linking. The object file that we create is only going to live briefly, probably in /tmp, likely on a RAM disk. It will be deleted after the external linker runs. Compressing the debug info in that case will buy us nothing; we should leave it to the external linker.

@heschi
Copy link
Contributor

heschi commented Dec 15, 2017

I agree, it's both impossible and undesirable :)

I might work on this for 1.11 as part of getting DWARF location lists enabled by default, to offset the increase in binary size.

@ghost
Copy link

ghost commented Dec 30, 2017

What about providing the option to completely strip debug info? The -w and -s flags don't remove this info.

@ianlancetaylor
Copy link
Contributor

@r04r That is a completely different problem that should be discussed on a different issue, not here. Thanks.

@ghost
Copy link

ghost commented Dec 30, 2017

@ianlancetaylor This issue seems to be about acquiring a "significant, cheap file size win.". Instead of compressing, just removing this information seems more significant, and cheaper. Is it viable? Should I make another issue?

@ianlancetaylor
Copy link
Contributor

@r04r This issue is about a cheap file size win without losing any information. Removing the debug info produces a binary that can not be used with a debugger, so that is a different matter--not a win at all for people who want to use a debugger.

Yes, if the linker's -s and -w flags do not work as documented, please open a different issue.

@josharian
Copy link
Contributor Author

Marking as release-blocker for 1.11, since https://go-review.googlesource.com/c/go/+/100738 just went in and executable sizes are up >15%.

@heschi
Copy link
Contributor

heschi commented Mar 20, 2018

Spent a little more time. More notes, adding to #11799 (comment).

Revisiting it, I'm not totally sure where to put the compression any more. The linker calculates the output's layout in (*Link).address(), then uses it in (*Link).reloc() to compute relocation targets and rewrite symbols as necessary. The DWARF cannot have been compressed before that, since it contains relocations. But compression could alter the layout, invalidating the previous results. In practice, the only things that come after the .debug sections on my Linux/AMD64 machine are .note.go.buildid, .symtab, and .strtab. I don't think any of those ought to be relocation targets. As long as we don't mind making that a guarantee, we can add a compression step beween reloc and Asmb, using the .zdebug approach for both Mach-O and ELF based on Ian's commentary above. I don't know about PE.

@aarzilli
Copy link
Contributor

Other options:

  1. Move all .debugs all the way to the end of the file
  2. Do the relocations for .debugs only, compress, do the relocations for everything else.

@aclements
Copy link
Member

To expand on @heschik's comment, perhaps the root of the problem here is that (*Link).address() does two things: it assigns virtual addresses, which has to be done before reloc; and it assigns file offsets. These are logically independent. If we could disentangle them, we could assign virtual addresses, then apply relocations, then compress the DWARF section, then do file layout. I suspect the reason they're entangled is simply that there's no simple list of segments anywhere. The list is all unrolled into code and it would be awful to duplicate that code, so it's all in address.

@bradfitz
Copy link
Contributor

bradfitz commented May 4, 2018

I'd like to stress that this really should happen for Go 1.11, as our binaries have gotten 20% bigger since Go 1.10 and that's not something users will be super happy about.

@aclements
Copy link
Member

I'll see if I can put together a fix for this, but it's definitely not an easy change. We'll see how it turns out.

@ianlancetaylor
Copy link
Contributor

@aclements I'm also happy to look at this if you prefer.

@gopherbot
Copy link

Change https://golang.org/cl/118276 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

@gopherbot
Copy link

Change https://golang.org/cl/118277 mentions this issue: cmd/link: compress debug sections in external linking mode

gopherbot pushed a commit that referenced this issue Jun 12, 2018
Currently these two forms of layout are done in a single pass. This
makes it difficult to compress DWARF sections because that must be
done after relocations are applied, which must happen after virtual
address layout, but we can't layout the file until we've compressed
the DWARF sections.

Fix this by separating the two layout steps. In the process, we can
also unify the copy-pasted code in Link.address to compute file
offsets. Currently, each instance of this is slightly different, but
there's no reason for it to be. For example, we don't perform
PEFILEALIGN alignment on Segrodata or Selreltodata even when HeadType
== Hwindows, but it turns out it doesn't matter whether you do or
don't because these segments simply don't exist on Windows. Hence, in
the unified code path, we do this alignment for all segments.
Likewise, there are two ways of computing Fileoff:
  seg.Vaddr - prev.Vaddr + prev.Fileoff
and
  prev.Fileoff + uint64(Rnd(int64(prev.Filelen), int64(*FlagRound)))
At the moment, these always have the same value, but the latter will
continue to work after we start compressing sections on disk.

Tested by comparing test binaries for all packages in std before and
after this change for GOOS={linux,windows,darwin,plan9}. All binaries
are identical.

For #11799.

Change-Id: If09f28771bb4d78dd392fd58b8d7c9d5f22b0b9f
Reviewed-on: https://go-review.googlesource.com/111682
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@jcajka
Copy link
Contributor

jcajka commented Jun 13, 2018

bd83774 seems to be cause of issue #25863

@gopherbot
Copy link

Change https://golang.org/cl/118716 mentions this issue: cmd/link: separate virtual address layout from file layout

dna2github pushed a commit to dna2fork/go that referenced this issue Jun 14, 2018
Currently these two forms of layout are done in a single pass. This
makes it difficult to compress DWARF sections because that must be
done after relocations are applied, which must happen after virtual
address layout, but we can't layout the file until we've compressed
the DWARF sections.

Fix this by separating the two layout steps. In the process, we can
also unify the copy-pasted code in Link.address to compute file
offsets. Currently, each instance of this is slightly different, but
there's no reason for it to be. For example, we don't perform
PEFILEALIGN alignment on Segrodata or Selreltodata even when HeadType
== Hwindows, but it turns out it doesn't matter whether you do or
don't because these segments simply don't exist on Windows. Hence, in
the unified code path, we do this alignment for all segments.
Likewise, there are two ways of computing Fileoff:
  seg.Vaddr - prev.Vaddr + prev.Fileoff
and
  prev.Fileoff + uint64(Rnd(int64(prev.Filelen), int64(*FlagRound)))
At the moment, these always have the same value, but the latter will
continue to work after we start compressing sections on disk.

Tested by comparing test binaries for all packages in std before and
after this change for GOOS={linux,windows,darwin,plan9}. All binaries
are identical.

For golang#11799.

Change-Id: If09f28771bb4d78dd392fd58b8d7c9d5f22b0b9f
Reviewed-on: https://go-review.googlesource.com/111682
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue Jun 15, 2018
Currently these two forms of layout are done in a single pass. This
makes it difficult to compress DWARF sections because that must be
done after relocations are applied, which must happen after virtual
address layout, but we can't layout the file until we've compressed
the DWARF sections.

Fix this by separating the two layout steps. In the process, we can
also unify the copy-pasted code in Link.address to compute file
offsets, which currently has some unnecessary variation.

Unlike the current file offset computation, which depends on virtual
addresses, the new computation only uses file offsets and sizes. This
will let us compress the file representation of a segment and create
the file layout based on its on-disk size rather than its original
in-memory size.

Tested by comparing the test binary for the "strings" package on all
supported GOOS/GOARCH combinations. All binaries are identical
(except, of course, their build IDs).

This is a second attempt at CL 111682.

For #11799.
Fixes #25863.

Change-Id: If09f28771bb4d78dd392fd58b8d7c9d5f22b0b9e
Reviewed-on: https://go-review.googlesource.com/118716
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue Jun 15, 2018
Forked from CL 111895.

For #11799.

Change-Id: Ie1346ac2c9122de494823b9058df3a0971e9dfe1
Reviewed-on: https://go-review.googlesource.com/118277
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@robpike
Copy link
Contributor

robpike commented Jun 18, 2018

This should be left open until all platforms, not just ELF, do this.

@robpike robpike reopened this Jun 18, 2018
@agnivade
Copy link
Contributor

Then #25927 should be closed I believe.

@aclements
Copy link
Member

I think it's fine to leave #25927 open for the Windows-specific aspects of this. There's already been a decent amount of discussion there.

For macOS, I'm concerned because my understanding is that lldb doesn't support compressed DWARF sections in Mach-O binaries (in fact, it didn't support them in ELF binaries until last month). And I believe dsymutil doesn't support them either (@heschik?)

@heschi
Copy link
Contributor

heschi commented Jun 18, 2018

Well, dsymutil doesn't matter so much since we only use it during external linking, and we would need the host linker to do the compression in that case. But there's no --compress-debug-sections on macOS. I suppose we could do the compression in machoCombineDwarf.

Then there's the question of whether we want to. dwarfdump and lldb won't support it, but Delve could. So I suppose it's technically feasible as long as we're willing to leave lldb behind. I would want a flag so that I could use dwarfdump when necessary though.

@gopherbot
Copy link

Change https://golang.org/cl/119816 mentions this issue: cmd/link: enable DWARF compression on Windows

@gopherbot
Copy link

Change https://golang.org/cl/119815 mentions this issue: debug/elf,macho,pe: support compressed DWARF

gopherbot pushed a commit that referenced this issue Jun 19, 2018
Simple follow-on to CL 118276. Everything worked except that the
compressed sections need to be aligned at PEFILEALIGN.

Fixes #25927
Updates #11799

Change-Id: Iec871defe30e3e66055d64a5ae77d5a7aca355f5
Reviewed-on: https://go-review.googlesource.com/119816
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
gopherbot pushed a commit that referenced this issue Jun 19, 2018
Since we're going to start compressing DWARF on Windows and maybe
Darwin, copy the ELF support for .zdebug sections to macho and pe. The
code is almost completely the same across the three.

While I was here I added support for compressed .debug_type sections,
which I presume were overlooked before.

Tests will come in a later CL once we can actually generate compressed
PE/Mach-O binaries, since there's no other good way to get test data.

Updates #25927, #11799

Change-Id: Ie920b6a16e9270bc3df214ce601a263837810376
Reviewed-on: https://go-review.googlesource.com/119815
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/120155 mentions this issue: cmd/link: support DWARF compression on Darwin

gopherbot pushed a commit that referenced this issue Jun 22, 2018
We want to compress DWARF even on macOS, but the native toolchain isn't
going to understand it. Add a flag that can be used to disable
compression, then add Darwin to the whitelist used during internal
linking.

Unlike GNU ld, the Darwin linker doesn't have a handy linker flag to do
compression. But since we're already doing surgery to put the DWARF in
the output executable in the first place, compressing it at the same
time isn't unduly difficult. This does have the slightly odd effect of
compressing some Apple proprietary debug sections, which absolutely
nothing will understand. Leaving them uncompressed didn't make much
sense, though, since I doubt they're useful without (say) __debug_info.

Updates #11799

Change-Id: Ie00b0215c630a798c59d009a641e2d13f0e7ea01
Reviewed-on: https://go-review.googlesource.com/120155
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@heschi
Copy link
Contributor

heschi commented Jun 22, 2018

Here's the state of things as of that last commit. We now enable DWARF compression by default on almost all platforms. I didn't do anything for plan9, wasm, and nacl, but I'm not actually sure if they get debug information anyway.

To debug a binary with DWARF compression, you'll need a modern version of GDB or a version of Delve after derekparker/delve@440b440 compiled with tip Go. Tip versions of lldb might work, but I haven't tested them. Compression can be disabled with -ldflags=-compressdwarf=false if there's any problems, or if you need to use inspection tools like macOS dwarfdump that don't understand the compressed DWARF.

I'm not aware of any significant remaining work to do here, so closing. Please comment if I missed something.

@heschi heschi closed this as completed Jun 22, 2018
@gopherbot
Copy link

Change https://golang.org/cl/138182 mentions this issue: doc: mention -compressdwarf=false on gdb page

gopherbot pushed a commit that referenced this issue Sep 27, 2018
Update #11799

Change-Id: I2646a52bfb8aecb67a664a7c6fba25511a1aa49f
Reviewed-on: https://go-review.googlesource.com/138182
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: David Chase <drchase@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/138184 mentions this issue: [release-branch.go1.11] doc: mention -compressdwarf=false on gdb page

@golang golang locked and limited conversation to collaborators Sep 27, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests