Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime/pprof: CPU profiles incorrect for kernels with broken setitimer support #13841

Open
rsc opened this issue Jan 6, 2016 · 6 comments
Open

runtime/pprof: CPU profiles incorrect for kernels with broken setitimer support #13841

rsc opened this issue Jan 6, 2016 · 6 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime.
Milestone
Unplanned

Comments

@rsc
Copy link
Contributor

rsc commented Jan 6, 2016

As of Go 1.6, pprof's CPU profiles are known to be incorrect on a few systems due to what are arguably kernel bugs. This issue documents those systems.

The text below distinguishes a profile being incomplete (missing profile samples for code that was running) from being incorrect (containing samples for code that wasn't running).

  • DragonflyBSD (all known versions): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (20ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete.
  • Linux (without CONFIG_HIGH_RES_TIMERS=y): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (often 10ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete. Most Linux kernels in use today do enable high-resolution timers and therefore do not suffer from this problem.
  • NetBSD (all known versions): Delivers signals to the wrong thread. On such systems, profiles are commonly very incorrect.
  • OpenBSD (all known versions): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (20ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete.
  • OS X (fixed in OS X 10.11 El Capitan): Deliver signals to the wrong thread. On such systems, profiles are commonly very incorrect. See rsc.io/pprof_mac_fix for a workaround on those early systems.
  • Solaris (fixed in Solaris 8): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (10ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete. Solaris 8 fixes the problem on systems with APIC hardware (most x86 systems). On systems that continue to exhibit the problem, adding set hires_tick = 1 to /etc/system can mitigate this problem somewhat by reducing the clock tick to 1ms.

Please comment on this issue only if the text above is incomplete or incorrect; we will keep this top-level comment up to date.
@rsc rsc added this to the Unplanned milestone Jan 6, 2016
@rsc rsc mentioned this issue Jan 6, 2016
Closed
@bradfitz bradfitz mentioned this issue Jan 6, 2016
Closed
@bradfitz
Copy link
Contributor

bradfitz commented Jan 6, 2016

Sent a change to skip the test on dragonfly too: https://go-review.googlesource.com/18306 after failures:

http://build.golang.org/log/9e126b69f4629739f0c7951ce4d42cc46db54a31
http://build.golang.org/log/56a3af05f787150fb668c99bfd30ebaae3e67bf8
etc

@rsc
Copy link
Contributor Author

rsc commented Jan 6, 2016

Added Dragonfly, thanks.

@bradfitz bradfitz mentioned this issue Jan 8, 2016
Closed
@aclements
Copy link
Member

FreeBSD has the same problem as OpenBSD and DragonflyBSD, but by default runs at 1000Hz, so the problem is less noticeable.

@mdempsky
Copy link
Contributor

I don't believe the description of OpenBSD is entirely accurate. In particular, OpenBSD's "hard clock" interrupt runs at 100Hz (i.e., 10ms clock tick intervals) on all CPUs supported by Go. Also, I don't see any evidence that OpenBSD's kernel cares about whether the process was running for the full time slice; only that it was running when the interrupt fired. The process may only have actually run for the last 1ms of the clock slice, but it will still have its timer credited for the full 10ms.

Lastly, I'm not sure it's relevant to the runtime/pprof tests, but for completeness: OpenBSD sends SIGPROF to the process, not the thread. It's just that it favors sending to the running thread when possible. If the interrupted thread has blocked SIGPROF and the process has other threads that are not blocking SIGPROF, the kernel will send SIGPROF to one of those instead.

gopherbot pushed a commit that referenced this issue Feb 2, 2016
@bradfitz
runtime/pprof: mark dragonfly and solaris as bad at pprof
beceea7 
Updates #13841

Change-Id: I121bce054e2756c820c76444e51357f474b7f3d6
Reviewed-on: https://go-review.googlesource.com/19161
Reviewed-by: Russ Cox <rsc@golang.org>
@alk alk mentioned this issue Feb 20, 2016
Closed
@timdarbydotnet
Copy link

From Matt Dillon on Dragonfly: That one really isn't a bug. DragonFly will profile at any point (not just on a full tick), but we use a low resolution profiling timer so the collected statistics will not be very good for short tests.

@aalexand aalexand mentioned this issue Jul 21, 2017
Closed
@ALTree ALTree mentioned this issue Nov 6, 2017
Open
@rictorlome rictorlome mentioned this issue Jan 19, 2019
Closed
@tobiaskohlbau tobiaskohlbau mentioned this issue Sep 27, 2019
Closed
@rhysh rhysh mentioned this issue Oct 21, 2019
Closed
@bcmills bcmills mentioned this issue Feb 19, 2021
Closed
@rhysh rhysh mentioned this issue Oct 29, 2021
Closed
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
@adonovan
Copy link
Member

Hi Russ, do you want to add #57722 to the list at the top?

@adonovan adonovan mentioned this issue May 10, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
7 participants
@bradfitz @mdempsky @rsc @aclements @adonovan @gopherbot @timdarbydotnet