CC @golang/runtime

This is closely related to the accepted proposal #15490. The difference here is this is a histogram of run time rather than just a count.

cc @mknyszek

That proposal includes /sched/goroutines/running:goroutines, a count of how many goroutines are running at this instant, rather than a cumulative histogram of how long they remained in that state. They're related, but I don't think we'd be able to use the value of one to derive the other.

Throughout: Being able to calculate a continuously-updating (sampled) sum of on-CPU time allows comparisons against other perspectives such as RUSAGE_SELF or CLOCK_PROCESS_CPUTIME_ID. This lets the application notice that it's exceeding the OS-provided resources.

This would indeed be interesting.

But, histograms have a downside: values are only added to the bucket count when the event has passed. In this case, the duration for which a specific goroutine was RUNNING can only be determined after it has transitioned to another state.

On the edge, it's possible that goroutines rarely (or ever) transition to other states. It is also possible that the scheduler itself is having hiccups,

This is closely related to the accepted proposal #15490. The difference here is this is a histogram of run time rather than just a count.

As @rhysh highlights, the instantaneous count of RUNNING/RUNNABLE/... goroutines would be interesting from a debugging point of view, but (I suspect) less useful from a runtime tuning/controller perspective than having a sum of the time spent in each state. In general cumulative values are better for this, as one create ones own deltas based on some polling period, and not lose data. A general strategy to make something cumulative is to add a time component, which naturally leads to something like /sched/running/total:seconds. Contrasting:

/sched/latencies:seconds: Histogram of time spent in RUNNABLE state.
/sched/latencies/total:seconds: Sum of time spent RUNNABLE by all goroutines (up until metric collection time). NOTE: This value would be different from summing the buckets*counts of /sched/running:seconds, not only due to the imprecision of the bucket boundaries, but also because samples are only added to /sched/running:seconds when goroutines switch state, whereas they are sampled at collection time in /sched/running/total:seconds.
/sched/running:seconds: Histogram of time spent in RUNNING state.
/sched/running/total:seconds: Sum of time spent RUNNING by all goroutines (up until metric collection time). NOTE: This value would be different from summing the buckets*counts of /sched/running:seconds, not only due to the imprecision of the bucket boundaries, but also because samples are only added to /sched/running:seconds when goroutines switch state, whereas they are sampled at collection time in /sched/running/total:seconds.

Does this make sense? What do others think? Should this go in a separate proposal?

Implementation wise, this would be either a single atomic value (incremented when goroutines switch state), to which the current time-spent-in-RUNNABLE for all RUNNABLE goroutines is added at collection time. Alternatively, a per-thread counter that can be updated without atomics, and is summed when read. I would prefer the latter, so as to not introduce more possible contention points in the runtime. Metric reads are far less frequent than writes, and could take the hit.

References:

• /sched/running/total:seconds is not the same as the /sched/goroutine/running:nanoseconds proposed by @irfansharif in https://go.dev/cl/387874. The latter would only return RUNNING time for the current (querying) goroutine, while the former returns the sum over all goroutines
• The values returned by /sched/running/total:seconds would be similar to those returned by CumulativeNormalizedRunnableGoroutines mentioned in proposal: runtime: add per-goroutine CPU stats #41554 (comment). But, the /sched/running/total:seconds proposal seems better as it does not have a discretization step in-between (losing data).