-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime/pprof: TestCPUProfileMultithreadMagnitude rusage too low #50232
Comments
Change https://golang.org/cl/372800 mentions this issue: |
The CPU profiler asks for SIGPROF signals at 100Hz: https://github.com/golang/go/blob/go1.17.5/src/runtime/pprof/pprof.go#L771 The runtime allows goroutines to run for 10ms before preempting them (via SIGURG): https://github.com/golang/go/blob/go1.17.5/src/runtime/proc.go#L5476 Are we seeing this genre of failure—a small fraction of times the CPU profiler runs include many samples pointing to code in the scheduler—because those intervals are in sync? |
I meant it for both. This has samples "in" asyncPreempt. |
Ah true, good point. |
…tely Currently TestCPUProfileMultithreadMagnitude runs two CPU consumption functions in a single profile and then analyzes the results as separate subtests. This works fine, but when debugging failures it makes manual analysis of the profile dump a bit annoying. Refactor the test to collect separate profiles for each subtest for easier future analysis. For #50097. For #50232. Change-Id: Ia1c8bb86aaaf652e64c5e660dcc2da47d2194c2b Reviewed-on: https://go-review.googlesource.com/c/go/+/372800 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Rhys Hiltner <rhys@justin.tv> Reviewed-by: Bryan Mills <bcmills@google.com>
This appears not to be specific to
2022-03-16T20:10:07-91631bc/linux-amd64-wsl |
I'm working on this. I plan to convert it back to the usual profiling test structure, so The early versions of TestCPUProfileMultithreadMagnitude ran the serial and parallel versions as part of the same profile, which got in the way of using I'm also going to restructure the rusage reporting so the test can log the user and system time separately. I expect the SIGPROF deliveries to match the sum of those, but there may be a helpful clue in that breakdown when the test fails. |
Change https://go.dev/cl/393934 mentions this issue: |
Restructure TestCPUProfileMultithreadMagnitude so it will run again with a longer duration on failure. Log the split between the user vs system CPU time that rusage reports. For #50232 Change-Id: Ice5b38ee7594dbee1eaa5686d32b968c306e3e85 Reviewed-on: https://go-review.googlesource.com/c/go/+/393934 Run-TryBot: Rhys Hiltner <rhys@justin.tv> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com> Trust: Michael Knyszek <mknyszek@google.com>
So far, so good. (Is there anything still planned for this issue, or can it be closed?)
|
…tely Currently TestCPUProfileMultithreadMagnitude runs two CPU consumption functions in a single profile and then analyzes the results as separate subtests. This works fine, but when debugging failures it makes manual analysis of the profile dump a bit annoying. Refactor the test to collect separate profiles for each subtest for easier future analysis. For golang#50097. For golang#50232. Change-Id: Ia1c8bb86aaaf652e64c5e660dcc2da47d2194c2b Reviewed-on: https://go-review.googlesource.com/c/go/+/372800 Trust: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Rhys Hiltner <rhys@justin.tv> Reviewed-by: Bryan Mills <bcmills@google.com>
2021-12-06T19:10:14-7a84066/linux-s390x-ibm
This failure is the opposite of #50097. Here, the rusage is lower than the pprof samples, which is rather surprising. It also seems like we have way too many cpuHog1 samples. I'd expect 10, but see 16.
cc @rhysh @bcmills
The text was updated successfully, but these errors were encountered: