Record and replay debugging with "rr"

Robert O'Callahan
https://linux.conf.au/schedule/30013/...
Debugging is expensive and not fun, especially debugging intermittent failures. rr changes that by recording Linux user-space process execution and letting you play back a recording and debug it using gdb, getting exactly the same execution as was recorded, as many times as you need. rr is used by Mozilla developers to debug Firefox, so it works on real applications. It runs on stock hardware, requires no kernel changes, and has low recording overhead (usually less than 1.5x). rr supports gdb's reverse execution features, which (combined with breakpoints and hardware data watchpoints) make debugging fast and fun. This presentation will describe how rr works, discuss some of the kernel issues that created difficulties for rr, and speculate about what we can do to make rr and related tools even better.