A safe way to inject our own stack size (even if it is static).

The Go project has explicitly avoided providing knobs like this. It adds complexity that we have to live with forever. I don't think we'd want to change that stance to get just a few percent performance improvement.

A histogram with all the stack sizes it has seen, this would helps decide what size would gives us the best results without using too much memory.

This might be possible with the new runtime/metrics API, as it supports returning histograms. I don't think the runtime currently tracks goroutine sizes, so this might incur some additional cost to track. We'd also have to decide what "size" means here, exactly. Peak use? Average use? Randomly sampled use? Sampled at GC time use?

And given above, I don't see a compelling need to provide this data given that we don't intend to provide a way to use it.

The assumption a good global average size exists appears to be false for many applications. Allowing the developer to control the global value doesn't improve the situation.

It would be better to automatically track/set the initial stack size per go invocation - I suspect this assumption is likely to hold in many more cases. The assumption could still break for goroutine pools running wildly different tasks, any automated solution is going to struggle there.

Average is not a very representative metric. It deserves some more exploration of other statistical measures: e.g., median/mode/...

You could also imagine incorporating this into PGO somehow.

Thanks for the response @randall77!

(disclaimer, I work in the same corporation as @cdvr1993, but on a different domain)

The Go project has explicitly avoided providing knobs like this. It adds complexity that we have to live with forever. I don't think we'd want to change that stance to get just a few percent performance improvement.
[...]
I don't see a compelling need to provide this data given that we don't intend to provide a way to use it.

This is a bad phrasing on our part. We would also strongly prefer not to have knobs and for things to work (mostly) okay. The goal thing was to report the situation, list what we tried/thought about, and ask for advice.

To answer your points 1 by 1:

• knobs: we agree that less is more. But then there are GOMAXPROCS, GOGC, GOMEMLIMIT - all modifiable at runtime. It was not immediately clear what is "too much". We mentioned the extra knob because we imagined it could (theoretically) be easier to expose a knob rather than changing the algorithm in general, like in runtime: new goroutines can spend excessive time in morestack #18138.
• "just a few percent" - 7% total CPU reduction is a significant amount for us. Especially compared to things like PGO, which seem to have lower wins (?)
• stats - well, we could at least observe this better in production and experiment in "user-space". We could then report back what we see.

As it stands, we'll likely end up using the unsafe/linkname tricks in at least the worst affected applications.

@randall77 could I maybe ask you to reconsider the stat emission? Or is that an absolute strong no?

Even without more knobs, we can very well start a "goroutine ballast" already today. It would be much easier to do if we had stats to base it on.

I don't have any objection to exposing such a stat by itself. I just think it would take a fair amount of work to implement - it isn't just exporting something the runtime is already keeping track of. Given the extra work required and limited payoff I don't think it is worth doing.
If someone can propose / design a stat that's easy & cheap to collect, then I think it would be ok to export it.

Instead of exposing this via runtime/metrics, would it be cheaper to expose as part of the pprof or some other explicit call? E.g., include the stack size for each goroutine in the pprof goroutines output.

I was actually able to get this info from cpu profiles. Not sure if it is the correct approach but at least I have some info.

So basically I focus on copystack on one profile and then disassemble the binary so that I can get the stack usage of each stack trace.

Then I just order them based on total size.

With that I was able to determine that the best starting stack size for the application I mentioned previously was 32KB which allowed me to reduce copystack from 9.5% to 0.8%.

The remaining would require a somewhat big jump on stack size, so I decided to stay on 32KB.

I don't have any objection to exposing such a stat by itself. I just think it would take a fair amount of work to implement - it isn't just exporting something the runtime is already keeping track of. Given the extra work required and limited payoff I don't think it is worth doing. If someone can propose / design a stat that's easy & cheap to collect, then I think it would be ok to export it.

A very simple stat that only keeps track of "maxStackSize" can be implemented with some very small additions to scanstack and gcComputeStartingStackSize to save the max in addition to the current sum and count that are being saved.
This will at least give people the "upper bound" of how high the startingStackSize needs to be to make sure their app never needs to waste CPU on copystack.

@randall77 - I think I can open such a PR, do you think if I do so it has a viable chance of getting merged?

Just tracking max would be pretty easy, yes.
I'm not sure it would be terribly useful though. Max has the problem that a single outlier can skew everything. avg*N for some small N is probably more robust.

I think for a proposal to be viable we would need a concrete description what to collect (and how), and how that information would be used. "Sizing my stack ballast" is not a very compelling use...

I'm not sure it would be terribly useful though. Max has the problem that a single outlier can skew everything.

While that is true in general, web servers deployed at scale usually have a pretty consistent flow of requests and every "type" of request (method/route/some parameter combination)- will have some size that it reaches for most requests, and each type will have some frequency.
The problem with anything that uses AVG is that it might be majorly skewed towards something that creates the majority of goroutines, while a small minority (like 10% of request being a POST and getting to a different flow for example) of request that reach a larger stack still cause significant wasted CPU.
For those same apps - the cost of memory is often insignificant compared to the cost of CPU, making the decision to use max a very easy one (for a subset of apps that we know how to pinpoint).

"Sizing my stack ballast" is not a very compelling use...

This is a "means to an end" - the goal is to reduce costs. We have seen services waste up to 15% of their CPU on morestack (from CPU profiling). For large services this sums up to very significant compute cost.