Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: compress debug info on windows #25927

Closed
dajoo75 opened this issue Jun 17, 2018 · 19 comments
Closed

cmd/link: compress debug info on windows #25927

dajoo75 opened this issue Jun 17, 2018 · 19 comments
Labels
Debugging FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Windows
Milestone
Go1.12

Comments

@dajoo75
Copy link
Contributor

dajoo75 commented Jun 17, 2018 •
edited

What did you do?

Compiled a basic hello world program on 64-bit Windows 10 with go tip (+b459e00), go 1.8.1 and go 1.10.3 and compared the binary sizes.

               hello_go1.8.1.exe  |  hello_go1.10.3.exe  |  hello_+b459e00.exe
binary size    1 633 280 bytes    |  2 057 728 bytes     |  2 848 768 bytes
                                  |                      |
.rdata                     N/A    |             N/A      |    752 128 bytes
.text          1 068 032 bytes    |  1 181 696 bytes     |    575 488 bytes
.debug_info      260 608 bytes    |    396 800 bytes     |    457 216 bytes
.debug_loc                 N/A    |             N/A      |    404 480 bytes
.symtab           98 304 bytes    |    141 312 bytes     |    142 336 bytes
.debug_frame      70 144 bytes    |     75 264 bytes     |     76 288 bytes
.debug_line       68 519 bytes    |     72 192 bytes     |    139 264 bytes
.debug_pubtypes   30 720 bytes    |     42 496 bytes     |     44 544 bytes
.debug_pubnames   17 408 bytes    |     40 960 bytes     |     41 472 bytes
.data             15 360 bytes    |     77 824 bytes     |     78 336 bytes
.debug_ranges              N/A    |     25 600 bytes     |    133 632 bytes
.idata             1 536 bytes    |      1 024 bytes     |      1 024 bytes
.debug_aranges       512 bytes    |              N/A     |              N/A
.debug_abbrev        512 bytes    |        512 bytes     |        512 bytes
.debug_gdb_scripts         N/A    |        512 bytes     |        512 bytes

What did you expect to see?

The binary size to be about the same at tip (+b459e00) as the previous release (go1.10.3 at the time of writing this). And if possible, the binary size to have gone down.

Other related issues: #6853, #11799

What did you see instead?

The binary size at tip (+b459e00) has increased by 38 percent since the previous release (go1.10.3 at the time of writing this).

Does this issue reproduce with the latest release (go1.10.3)?

Yes

System details

go version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000 windows/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="C:\Users\loki\AppData\Local\go-build"
GOEXE=".exe"
GOHOSTARCH="amd64"
GOHOSTOS="windows"
GOOS="windows"
GOPATH="C:\home\projects\go"
GOPROXY=""
GORACE=""
GOROOT="C:\home\installs\go\go.tip"
GOTMPDIR=""
GOTOOLDIR="C:\home\installs\go\go.tip\pkg\tool\windows_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\loki\AppData\Local\Temp\go-build103655438=/tmp/go-build -gno-record-gcc-switches"
VGOMODROOT=""
GOROOT/bin/go version: go version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000 windows/amd64
GOROOT/bin/go tool compile -V: compile version devel +b459e00 Sat Jun 16 13:35:45 2018 +0000
gdb --version: GNU gdb (GDB) 8.0
@ALTree
Copy link
Member

ALTree commented Jun 17, 2018

As you word it, this would technically be a dup of #6853.

What I think it's worth noting is that CL 118276 (cmd/link: compress DWARF sections in ELF binaries) only helped with ELF binaries. So I think it would be better if you renamed this issue to point out this specific issue (i.e., make this issue a TODO reminder to compress debug info on windows, too).

In theory we had #11799 (cmd/link: compress debug info) for the general problem, but it got closed after the ELF work.

@dajoo75 dajoo75 changed the title Windows binaries too big and growing cmd/link: compress debug info on windows Jun 17, 2018
@ALTree ALTree added this to the Go1.12 milestone Jun 17, 2018
@ALTree ALTree added NeedsFix The path to resolution is known, but the work has not been done. Debugging OS-Windows labels Jun 17, 2018
@dajoo75
Copy link
Contributor Author

dajoo75 commented Jun 17, 2018 •
edited

@ALTree I've renamed the issue to cmd/link: compress debug info on windows as you suggested.

Others more in the know about Windows, the PE file format and the debug story around Windows (gdb, delve etc.) have to chime in on the correct way forward.

@dajoo75
Copy link
Contributor Author

dajoo75 commented Jun 17, 2018

I tried to get this working on Windows (made some small changes to cmd/link and, debug/pe) and hello world.exe went from 2.8MB to 1.9MB. I almost got delve to start but didn't have time to finish it.

There doesn't seem to be an insurmountable amount of work left to do to get compression working on Windows as well (since most of the work seems to be done already in #11799) but I may be wrong (the devil is in the details and all that).

I would feel better if the original authors of #11799 looked at this as well (together with Alex) since they know what they are doing :).

cc @heschik, @alexbrainman, @ianlancetaylor, @aclements

@agnivade agnivade mentioned this issue Jun 18, 2018
Closed
@aarzilli
Copy link
Contributor

@dajoo75 what approach are you taking? Elf supports compression but PE doesn't (based on a ctrl-f search in the relevant MSDN page), so .zdebug?

@dajoo75
Copy link
Contributor Author

dajoo75 commented Jun 18, 2018 via email

@aarzilli
Copy link
Contributor

Ah, I didn't notice the switch to zdebug. In that case: https://github.com/derekparker/delve/pull/1237

@alexbrainman
Copy link
Member

Thank you @dajoo75 for creating the issue.

If I understand correctly, CL 118276 added compression for ELF DWARF sections. I would try and add some code to make .zdebug_... sections appear in Windows executables, and see where it gets us. I doubt I would have time before weekend. So others are welcome to this.

I also wonder if debuggers used by our users will support this. Lucky we have @aarzilli here, so he can fix Delve. Not so easy to fix gdb. We will see.

Alex

@aarzilli
Copy link
Contributor

Not so easy to fix gdb. We will see.

I think gdb was the original user of the zdebug convention, I imagine that should be fine. The problem is lldb.

@alexbrainman
Copy link
Member

The problem is lldb.

Does lldb work on Windows?

Alex

@aarzilli
Copy link
Contributor

Does lldb work on Windows?

Partially: https://lldb.llvm.org/status.html

@heschi
Copy link
Contributor

heschi commented Jun 18, 2018

@dajoo75 I'm happy to review a CL and help you over some bumps; I'd have to set up a Windows machine to even start on this and it sounds like you're well into it. We are pretty deep into the freeze so it'd be nice to get it done soon.

Can someone confirm that on Windows we're basically ignoring the platform convention for debug info, which I guess would either be COFF or PDB? As long as that's the case and none of the standard tools work anyway, I think it makes sense to go ahead and compress it. I doubt lldb is very popular there, even if it does work.

@ianlancetaylor
Copy link
Contributor

@heschik That is correct: we only generate DWARF. COFF debug info is entirely inadequate. As far as I know PDB is not publicly documented.

@dajoo75
Copy link
Contributor Author

dajoo75 commented Jun 18, 2018 via email

@heschi
Copy link
Contributor

heschi commented Jun 18, 2018

Okay. @alexbrainman, if you don't claim this by tomorrow I'll take a crack at it.

@alexbrainman
Copy link
Member

Okay. @alexbrainman, if you don't claim this by tomorrow I'll take a crack at it

Please, do. I will try it myself, but I do not know when.

Alex

@gopherbot
Copy link

Change https://golang.org/cl/119816 mentions this issue: cmd/link: enable DWARF compression on Windows

@gopherbot
Copy link

Change https://golang.org/cl/119815 mentions this issue: debug/elf,macho,pe: support compressed DWARF

gopherbot pushed a commit that referenced this issue Jun 19, 2018
@heschi
debug/elf,macho,pe: support compressed DWARF
2036f16 
Since we're going to start compressing DWARF on Windows and maybe
Darwin, copy the ELF support for .zdebug sections to macho and pe. The
code is almost completely the same across the three.

While I was here I added support for compressed .debug_type sections,
which I presume were overlooked before.

Tests will come in a later CL once we can actually generate compressed
PE/Mach-O binaries, since there's no other good way to get test data.

Updates #25927, #11799

Change-Id: Ie920b6a16e9270bc3df214ce601a263837810376
Reviewed-on: https://go-review.googlesource.com/119815
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@be5invis
Copy link

@ianlancetaylor
PDB's definition: https://github.com/Microsoft/microsoft-pdb

@ianlancetaylor
Copy link
Contributor

@be5invis Thanks.

@golang golang locked and limited conversation to collaborators Oct 19, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Debugging FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Windows
Projects
None yet
Milestone
Go1.12
Development

No branches or pull requests
8 participants
@be5invis @aarzilli @ianlancetaylor @ALTree @dajoo75 @gopherbot @alexbrainman @heschi