I think this is a good idea.
We use -ldflags="-s -w" by default in our team for years to keep the binary size down and make the build process fast.
I delete the build those flags occasionally to use go tool objdump to see which function is in the binary.I was surprised that the binary size go from 1.6MB to 2.6MB with golang 1.9 with a simple windows exe.
I even do not know what Delve or GDB is after six years of golang developing.
It looks like this proposal has not effect to me, because i already use -ldflags=-w in my build tool.

I too have tried to disable DWARF globally before, without much success. This proposal sounds great, and I agree with the change in default behavior. If a developer wants to use a debugger, they should be able to set an environment variable.

I also assume this wouldn't affect cmd/link's -s, since it doesn't seem to involve DWARF.

This is not as radical as it sounds.

This is pretty much what every other compiler does, I wouldn't call it radical.

I propose we change the Go build environment to suppress DWARF by default, saving lots of CPU time and disk space. Instead, a global shell environment variable, say GODWARF=1, could be set to cause it to be written out

My preference would be for a go build flag. For example -g.

My preference would be for a go build flag. For example -g.

My preference would be a golang package that have a function that can build golang package with all possible switches with good default value and nice field name like GoOs, GoArch, GoPathList, TagList, GoRoot, EnableDataRace, EnableDwarf ...

Command line flags or shell environment variables are very difficult for me to understand all the possible and the correct type of that value.
I use golang to replace all bash scripts or ugly command line flags include the scripts to build my golang program.
Is it the reason why we use golang in the first place?

I suggest the command line of golang compiler should run a host os and arch golang package which is my golang program build script. Then I will write everything else in my golang program build script.

Yes please. As I understand it delve suggest compiling with -N -l already so adding another flag to the “I want to build for the debugger” step shouldn’t be a large burden.

@bronze1man custom Go build programs have been discussed and rejected in the past - please keep this thread on topic. You're welcome to create a new proposal for your idea, though.

@davecheney, I agree with you that "recompile for debugging with Delve/gdb/lldb/etc" is not too onerous and well-established. I just want to point out for the record that the recent work is supposed to make it the case that you don't need -N -l (and the consequent performance loss) just to run those debuggers. While I don't mind training people that if you're going to use Delve you might need a different build, it would be good if we could get away from "if you're going to use Delve you have to give up significant performance", and I believe Go 1.11 is a decent step toward that.

/cc @aclements @cherrymui @dr2chase @heschik @thanm

The DWARF tables are present only for the debuggers.

Profilers read DWARF as well, for what it is worth. It's nice to be able to collect a detailed profile without having to recompile. On the other hand (as with debuggers) very few people run profilers.

How about a tweak -- continue to emit DWARF, but when linking "user" programs, don't include the DWARF from the standard packages (people who use debuggers are rare, but people who debug the Go runtime are even fewer).

How about a tweak -- continue to emit DWARF, but when linking "user" programs, don't include the DWARF from the standard packages (people who use debuggers are rare, but people who debug the Go runtime are even fewer).

That's not that easy. First you'd still want the full debug_frame (at minimum) to unwind the stack (possibly also skeleton DIEs for all functions too). Secondly, delve checks the destination of all CALL instructions to filter away the calls to runtime functions inserted by the compiler. Thirdly delve needs a bunch of DIEs from runtime (off the top of my head: runtime.allgs, runtime.g, runtime.firstmoduledata) for various reasons. Basically you would need to do something like microsoft's symbol files for system dlls.

@aarzilli
Everything you say makes sense, but I thought we had previously established that delve users would already be recompiling with special delve-friendly compiler flags?

@thanm: I was under the impression that this would be a compromise so they wouldn't have to. Also I assume most other tools will have needs similar to delve.

I am not the least bit happy that I'm expected to debug something different than what I run, and that a recompilation is expected. I don't run the debugger much because it's not easy to run the debugger much, and I'm trying to fix that ("not too many people swimming across the river, why do we need a bridge?"). I do think it is onerous; and I view it as one of the things that Java got relatively right.

There's the additional problem that the -N -l flag combo (treating it as a single flag) is not tested nearly as well as the default; if we honestly support it, we'll need to run all.bash in that mode too, and anyone with widely-used libraries will want to also test using that mode, just in case. Doubling the testing load is not going to speed up developer workflow. And of course, we must document the flags, and people learning to use the compiler will need to learn about them if they plan to create binaries that delve or gdb can debug.

The main point of adding all this debugging information to Go binaries was these two things; to remove speed bumps from Go debugging, and to make progress on the do-we/don't-we support problem for these flags.

Debugging information is also not just for debuggers. Years ago a friend of mine showed me what he called the "killer app for Python", which was just really good backtraces that did a nice job of reporting the values of all the local variables in the stack trace. Their app would wrap all that up in a neat little bundle, ask the customer if they were willing to send it in as a bug report (with certain assurances about information confidentiality that probably wouldn't fly now), and his next interaction with said customer was not to badger them for more information, but instead "here's the dot-release that fixes your problem, download it at your convenience". "And the customer thinks I'm a wizard!" We could/should do this for Go, at least as an option.

There are some alternatives, for instance, with reproducible builds we can generate debugging information on demand -- but when we mention that to nearby colleagues working on other languages and platforms, their reaction is "yeah, right". Their best practice is to build with debugging, make a copy (of either the debugging info alone, or the entire binary) and strip for shipping.

Is binary size causing some other problem that can be discussed publicly? Are we filling up disks? (not mine -- with 10 copies of the Go repo in various states of build, plus all the go-gotten binaries for tools, plus testing and benchmarking, about 2% of my "disk", counting both bin and pkg) . Is there some quota that we're exceeding? Are builds taking too long, and if so, by how much, and have we explored other ways of saving that time that don't reduce functionality? (the linker is due for a rewrite, the register allocator is definitely a time hog, Josh has done some interesting work on moving code from the front end to SSA that seems to save a little time, and has the potential to expose more of the compiler to multithreading).

"Debugging information" does not mean only "DWARF". As I said in my post, we can improve self-debugging support independently of improving debugger support, which means DWARF.

@dr2chase can always get the DWARF he wants all the time by just setting the flag once in his profile. I am not proposing to turn DWARF off, I am proposing to turn it off by default as few programmers need the support it provides every day. You are not expected to run without DWARF, you are expected just to set the flag once, for yourself, if you want DWARF on.

Binary size matters. We copy binaries around, we push binaries over networks, we put them in containers, we store them in the cloud, we fill our bin directories with many programs; in those operations, the accumulations are significant.

As to your performance problems, my experiments with building the go command (cmd/go) show that generating DWARF for a significant program is about 40% of the build CPU time. That is a major chunk.

% rm go
rm: go: No such file or directory
bismarck=% time go build

real	0m1.205s
user	0m1.235s
sys	0m0.238s
% rm go
% time go build -ldflags=-w

real	0m0.730s
user	0m0.703s
sys	0m0.223s
% hoc -e 730/1235
0.5910931174089069
% 

You propose we double our testing load to account for the presence/absence of this option?

Some compiler and tools work may require more testing, but only modest amounts. The great majority of Go programmers would be unaffected that way.

If the DWARF is redundant -- once we add the missing information to what we store in the binary -- we could generate it from built-in debugging information, thus allowing us to replace before deprecation, instead of vice-versa.

We'd also want to be careful about how any compatibility guarantees interact with the debugging information; what "variables" can be examined at which "statements" depends on all sorts of stuff.

Another worry is to what degree we'd allow a Go program to introspect on its own execution through this debugging facility.

If the binary records an accurate list of all modules (with versions) used for the build and all other relevant flags, it seems like delve could transparently generate the necessary information by retrieving the source and invoking the compiler.

Would building out two separate binaries one with debugging information and one without (Using go build or maybe a different command go release or some other command ) would maybe help with people that don't like the idea of having there final binaries without DWARF or having to set a environment variable right off the bat? Don't know if this is in scope / on-topic for this proposal, just wanted to throw out a thought.

@dr2chase respectfully the “why build bridges” argument applies equally to binary size “disks are hella big, why optimise for space?”

Now that DWARF is compressed on Mac (and I assume also on Windows), it seems like we're back to a 20% or so space overhead, and the main issue is now link latency.

Even on Linux I see DWARF aggregation approximately doubling the time spent in the linker (and it's not like the non-DWARF parts of the linker are terribly fast). So this is not a Mac-specific problem. Of course, compression might be what's taking all the time.

Putting this on hold pending better understanding of where all the CPU time is going and that there aren't significant optimizations remaining that might make it faster. If 2X really is the cost of DWARF, then we should resume this conversation about whether it makes sense to pay that cost speculatively all the time when the probability of needing it is near zero.

Filed #26318 for the linker time issue.

Change https://golang.org/cl/126656 mentions this issue: cmd/go: add $GOFLAGS environment variable

Almost all my production server code has the prof built in. As far as I know this is standard practice. I would hate for this to become harder to do. Also this would be a breaking change.

Almost all my production server code has the prof built in.

As mentioned in the very first post of this issue, profiling is not affected when DWARF information is not present:

Even without DWARF at all, stack traces cased by panic would be unchanged and would contain symbols and line numbers. Pprof, objdump, and many other tools would still work.

What are the alternatives to reduce binary size and compile/link time? (Are these issues filed for them already?)