runtime/pprof: CPU profiles incorrect for kernels with broken setitimer support

[rsc]

As of Go 1.6, pprof's CPU profiles are known to be incorrect on a few systems due to what are arguably kernel bugs. This issue documents those systems.

The text below distinguishes a profile being incomplete (missing profile samples for code that was running) from being incorrect (containing samples for code that wasn't running).

• DragonflyBSD (all known versions): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (20ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete.
• Linux (without CONFIG_HIGH_RES_TIMERS=y): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (often 10ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete. Most Linux kernels in use today do enable high-resolution timers and therefore do not suffer from this problem.
• NetBSD (all known versions): Delivers signals to the wrong thread. On such systems, profiles are commonly very incorrect.
• OpenBSD (all known versions): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (20ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete.
• OS X (fixed in OS X 10.11 El Capitan): Deliver signals to the wrong thread. On such systems, profiles are commonly very incorrect. See rsc.io/pprof_mac_fix for a workaround on those early systems.
• Solaris (fixed in Solaris 8): Delivers a profiling signal only when a thread runs continuously for an entire clock tick (10ms). In workloads with high context-switch or garbage collection rates, this may cause profiles to be incomplete. Solaris 8 fixes the problem on systems with APIC hardware (most x86 systems). On systems that continue to exhibit the problem, adding set hires_tick = 1 to /etc/system can mitigate this problem somewhat by reducing the clock tick to 1ms.

Please comment on this issue only if the text above is incomplete or incorrect; we will keep this top-level comment up to date.

[bradfitz]

Sent a change to skip the test on dragonfly too: https://go-review.googlesource.com/18306 after failures:

http://build.golang.org/log/9e126b69f4629739f0c7951ce4d42cc46db54a31
http://build.golang.org/log/56a3af05f787150fb668c99bfd30ebaae3e67bf8
etc

[rsc]

Added Dragonfly, thanks.

[aclements]

FreeBSD has the same problem as OpenBSD and DragonflyBSD, but by default runs at 1000Hz, so the problem is less noticeable.

[mdempsky]

I don't believe the description of OpenBSD is entirely accurate. In particular, OpenBSD's "hard clock" interrupt runs at 100Hz (i.e., 10ms clock tick intervals) on all CPUs supported by Go. Also, I don't see any evidence that OpenBSD's kernel cares about whether the process was running for the full time slice; only that it was running when the interrupt fired. The process may only have actually run for the last 1ms of the clock slice, but it will still have its timer credited for the full 10ms.

Lastly, I'm not sure it's relevant to the runtime/pprof tests, but for completeness: OpenBSD sends SIGPROF to the process, not the thread. It's just that it favors sending to the running thread when possible. If the interrupted thread has blocked SIGPROF and the process has other threads that are not blocking SIGPROF, the kernel will send SIGPROF to one of those instead.

[timdarbydotnet]

From Matt Dillon on Dragonfly: That one really isn't a bug. DragonFly will profile at any point (not just on a full tick), but we use a low resolution profiling timer so the collected statistics will not be very good for short tests.