Process Tracing Tips and Tools

Tracing Processes You Don't Control

e.g. busybox ash on an Alpine machine, for reducing bugs from regtest/aports

These techniques do NOT require recompilation, debug symbols, or source code modification

Single Process Tracing

• Shell interpreter tracing with set -x aka set -o xtrace - show shell statements executed

  • Example from Andy: I used this to discover that autotool configure scripts call rm and cat commonly, and then we implemented builtin rm and builtin cat

• strace myprog - show syscalls

  • the -e flag filters by syscall
  • the -c flag creates a histogram of syscalls
  • Samuel used this for the $PWD bug

• ltrace myprog - show libc calls

  • has an -e flag
  • Example from Andy: I used this to debug setlocale() calls in bash versus OSH

Multi Process Tracing

• shell tracing with set -x; export SHELLOPTS - bash and OSH have a feature where you can export SHELLOPTS to make all processes trace

  • Note: bash also has an XTRACE_FD= env var to send all the traces to the same place; I don't think OSH has this yet

• strace -ff -o prefix - follow forks, creates prefix.$PID files

Debugging / Tracing Processes You Can Modify

Require Symbols / Recompilation

• gdb --args myprog --flag
• ASAN gives nicer stack traces
• We are using uftrace - user space function tracing
  • it traces all function call entry and exit, by putting probes in the code

Require Source Code Instrumentation

• systemtap - we have DTRACE_PROBE() calls in the Oils C++ source code

Related

• Process Tracing Projects
• How Debuggers Work