Process Tracing Tips and Tools

Tracing Processes You Don't Control

e.g. busybox ash on an Alpine machine, for reducing bugs from regtest/aports. These techniques do NOT require recompilation, debug symbols, or source code modification

• Shell interpreter tracing with set -x aka set -o xtrace - show shell statements executed

  • Example from Andy: I used this to discover that autotool configure scripts call rm and cat commonly, and then we implemented builtin rm and builtin cat
  • multi-process tracing with set -x; export SHELLOPTS - bash and OSH have a feature where you can export SHELLOPTS to set -x on all child processes
  • Note: bash also has an XTRACE_FD= env var to send all the traces to the same place; I don't think OSH has this yet

• strace myprog - show syscalls

  • Samuel used this for the $PWD bug
  • It can be helpful to use -f -o FILE and then start replacing the PIDs with a human name in the file to analyze e.g. races
  • use something like -e t=chdir,file,read,write, to limit the shown syscalls (e.g. memory is usually irrelevant)
  • -v to not abbreviate
  • -s 1000 to abbreviate strings only after 1000 characters (needs to be in addition to -v
  • the -c flag creates a histogram of syscalls
  • the -f flag follows child processes
  • multi-process tracing with strace -ff -o prefix - follow forks, creates prefix.$PID files

• ltrace myprog - show libc calls

  • has an -e flag
  • Example from Andy: I used this to debug setlocale() calls in bash versus OSH

Tracing or Debugging Processes You Can Modify

Require Symbols / Recompilation

• gdb --args myprog --flag
• ASAN gives nicer stack traces
• We are using uftrace - user space function tracing
  • it traces all function call entry and exit, by putting probes in the code

Require Source Code Instrumentation

• systemtap - we have DTRACE_PROBE() calls in the Oils C++ source code

Related

• Process Tracing Projects
• Implementing Debuggers
• Tips on Reducing Bugs to Small Test Cases - for regtest/aports