-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: cmd/compile: appropriately disable garbage collector #24299
Comments
Potentially unbounded memory grow while compiling for a ~20% speed-up in compilation times for a typical package seems a bad trade-off. I this had the potential to cut in half compilation times it could be worth it, but 20% is not much. CPU-time reduction is also very small, I wonder if filling up the memory when compiling many packages in parallel could make it even less worthwhile. And anyway users can already do this with
which cases? It's not clear from the proposal. Like on certain systems? Or when compiling certain packages? Or both? |
I may misunderstand, but I think the benchmark means 5% in kernel and 20-25% in application, which is a noticeable difference by a person.
We’re missing data from everything not amd64/linux. I’d like to try with large open source projects. @ianlancetaylor mentioned very large packages built by generating code. The proposal is to define these cases and the feature that meets all needs. My thought and guess is disabling it may help 80% of people without any crashing, and otherwise we can reenable it by checking something.
Yes, but I’d prefer to not worry about that and a free noticeable improvement is good for the project. |
Would you mind if I'd label this proposal as "on hold" until you have all the data you need to come up with a concrete plan? The proposal process is usually used for concrete proposals with most of the details already worked out. |
@ALTree that's fine. I could go back to the golang-dev thread and work through it there too. Thanks. |
I am a bit apprehensive about this. IMO, this really feels like a slippery slope. There are whole lot of cases where disabling GC gives a boost. But changing the compiler to dynamically switch off GC seems like a cop-out to me. We should instead optimize the runtime further instead of switching off GC to improve performance. Especially when there is already a switch ( |
On the disabling front: what about a goroutine in cmd/compile that periodically (every 100ms?) checks memory usage and turns on regular garbage collection and returns if over a platform default that can be adjusted with an environment variable? |
There's always making the compiler generate less garbage. For example, at the moment it parses files via cmd/compile/internal/syntax, and translates that AST to cmd/compile/internal/gc's. That results in every AST node being allocated twice, and lots of little objects for the GC to keep track of. That will eventually be cleaned up, though. I would imagine that once the compiler gets better at generating less garbage, turning the GC off will have less of an impact. |
Long term plan is indeed to use less memory. Skipping the intermediate ast is one big piece of that. The other is lazy importing, since much of what gets imported is unused. @mdempsky is actively working on the latter, I believe. |
If this isn’t the case by the end of the Go 1.11 development cycle then I think invisibly (no regressions because of memory use changes) disabling the collector for (assumed) widespread 10-30% time reduction is the right move. This would require a Go 1.12 issue to revisit the workaround. |
That seems to imply that making the compiler 5% faster is an immediate priority. Sure, the compiler could be faster, and it is made faster every release. But I don't see the need for this kind of urgency, especially when this workaround has many potential downsides. And also since it's already available via GOGC. For example, if one compiles very large programs, I wouldn't be surprised if turning off the GC doubled the peak memory use of the compiler. Have you measured the downsides to disabling the GC in any way? Also remember the machines that have low RAM - if I remember correctly, even with GC on some ARM builders were having issues with memory. |
5% doesn’t seem worth a workaround effort, but 20-30% does to me. I may be misunderstanding the benchmark. The benchmark system is Ubuntu server without a GUI, and I’m assuming the time and user-time add.
From compilebench I assume the total allocations (without each package’s memory being released to the OS taken into account) generally don’t go beyond a few GB, which is fine for a typical desktop compile. Disabling the GC doesn’t change that number, and I assume most large programs consist of packages that fit within the compilebench constraints.
Having a dynamic reenable like I suggested earlier would cover these cases. An idea is that if memory is overused then the memory threshold environment variable could be updated by the toolchain (so all future compiles use that) then the compile could be retried. Worst case the GC is back to always on and the user didn’t see any difference. |
linux/amd64 with 1 GB of memory (kernel flag
Disabling on platforms with virtual memory (all of them?) shouldn’t cause crashes (assuming ample drive space), but for large cases performance can be severely impacted by memory swapping to the point of being unusable. The compile command is used on each package separately but has to load the entirety of its dependency object code, so it appears a lot of memory can be used in a single call especially at the root package. But for compilebench cases this number appears to be under 8 GB. Conclusion The cmd/compile memory needs are unbounded in relation to program size and performance is majorly helped by the garbage collector after a knee point of memory use, but before that point, which is significant on most development computers for small to medium programs, we can see 10%-30% cmd/compile time reduction by disabling the garbage collector. |
I misunderstood the benchmark. The first table (time/op) is wall time, and the second time (user-time/op) is the value reported by os.ProcessState.UserTime() (I was thinking this was kernel vs app time - also I can’t just add percentages for this even though the values are somewhat close). For the user the wall time is what they’ll perceive, so we are looking at ~5% like @mvdan said. That doesn’t seem like a worthwhile increase for a workaround, so I’ll close this. Thanks. |
Disabling the garbage collector in cmd/compile via
debug.SetGCPercent(-1)
saves significant time according to compilebench. Since the compile command is used per-package there are cases where the garbage collector is freeing memory the OS is about to free anyway, so this proposal is to define a feature to disable the collector in certain cases for cmd/compile.(with
go version devel +1b1c8b3 Sat Feb 17 18:35:41 2018 +0000 linux/amd64
, four cores, and 'performance' CPU frequency governor)Running the benchmark and compiling the Go toolchain worked on an 8GB linux/amd64 computer with the garbage collector disabled.
Two concerns from https://groups.google.com/forum/#!topic/golang-dev/atj2hJIJj4o are for limited systems such as the Raspberry Pi and for large packages that may be created by generating code, but a conclusion is that there may be a careful worthwhile cmd/compile change to make.
I plan to report results here from:
The text was updated successfully, but these errors were encountered: