runtime: scavenger is too eager on Darwin #36507
Comments
mknyszek
added
Performance
NeedsFix
|
Change https://golang.org/cl/214517 mentions this issue: |
|
Digging deeper, I think I know better why I'm not sure what to do here. The patch I uploaded helps the problem a little bit, but to completely solve the problem would require figuring out exactly which pages are scavenged and only calling Because the allocator doesn't propagate up which pages are scavenged (though it does know this information precisely!) currently we just do the heavy-handed thing. But, we could instead have the allocator, which actually clears the bits, do the Unfortunately, lowering the |
|
On second thought, I can't really find evidence of the number of scavenged pages in an allocation being less than the size of the allocation in the above benchmarks. Perhaps this isn't a problem... |
|
OK going to walk back my second thought. Turns out, yes, #36507 (comment) is a real problem. It's slightly worse now, but only by a little bit. This is basically going back to Go 1.11 behavior where if any memory was scavenged you would just scavenge the whole thing. |
Currently on Darwin the scavenger is too eager and causing performance regressions. The issue mainly stems from the fact that in Go 1.14 the scavenger is paced empirically according to costs of scavenging, most of which comes from
sysUnusedwhich makes anmadvisesyscall on most platforms, including Darwin.However, the problem on Darwin is that we don't just do
MADV_FREEanymore, we doMADV_FREE_REUSABLEin thesysUnusedpath andMADV_FREE_REUSEin thesysUsedpath (instead ofsysUsedbeing a no-op). It turns out the source of the regression is mostly the fact thatsysUsedis actually quite expensive relative to other systems where it's just a no-op and we instead incur an implicit page fault once the memory is actually touched. The benefits of using this API outweigh the costs, since it updates process RSS counters in the kernel appropriately andMADV_FREE_REUSABLEis properly lazy.So since we don't account for
sysUsedwe end up scavenging a lot more frequently than we should to maintain the scavenger's goal of only using 1% of the CPU time of one CPU.The actual size of the regression can be quite large, up to 5%, as seen in #36218, so we should fix this before 1.14 goes out.
The fix here is relatively simple: we just need to account for this extra cost somehow. We could measure it directly in the runtime but this then slows down allocation unnecessarily, and even then it's unclear how we should attribute that cost to the scavenger (maybe as a debt it needs to pay down?). Trying to account for this cost on non-Darwin platforms is also tricky because the costs aren't actually coming from
sysUsedbut from the page fault.Instead, I think it's a good idea to do something along the lines of what we did last release: get some empirical measurements and use that to get an order-of-magnitude approximation. In this particular case, I think we should compute an empirical ratio "r" of using a scavenged page and
sysUnusedand turn this into a multiplicative constant, "1+r" for the time spent scavenging. So for example ifsysUsedis roughly as expensive assysUnusedthe factor would be 2.The text was updated successfully, but these errors were encountered: