Linux 4.x Tracing Tools Using BPF Superpowers
Brendan Gregg, Ne lix bgregg@ne lix.com
December 4–9, 2016 | Boston, MA www.usenix.org/lisa16 #lisa16
Demo me GIVE ME 15 MINUTES AND I'LL CHANGE YOUR VIEW OF LINUX TRACING
inspired by Greg Law's: Give me fi een minutes and I'll change your view of GDB Demo LISA 2014 perf-tools ( race) LISA 2016
bcc tools (BPF) Wielding Superpowers WHAT DYNAMIC TRACING CAN DO Previously
• Metrics were vendor chosen, closed source, and incomplete • The art of inference & making do
# ps alx F S UID PID PPID CPU PRI NICE ADDR SZ WCHAN TTY TIME CMD 3 S 0 0 0 0 0 20 2253 2 4412 ? 186:14 swapper 1 S 0 1 0 0 30 20 2423 8 46520 ? 0:00 /etc/init 1 S 0 16 1 0 30 20 2273 11 46554 co 0:00 –sh […] Crystal Ball Observability
Dynamic Tracing Linux Event Sources Event Tracing Efficiency
Eg, tracing TCP retransmits Kernel Old way: packet capture
tcpdump 1. read send 2. dump buffer receive
Analyzer 1. read 2. process file system disks 3. print
New way: dynamic tracing
Tracer 1. configure tcp_retransmit_skb() 2. read New CLI Tools
# biolatency Tracing block device I/O... Hit Ctrl-C to end. ^C usecs : count distribution 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 1 | | 128 -> 255 : 12 |******** | 256 -> 511 : 15 |********** | 512 -> 1023 : 43 |******************************* | 1024 -> 2047 : 52 |**************************************| 2048 -> 4095 : 47 |********************************** | 4096 -> 8191 : 52 |**************************************| 8192 -> 16383 : 36 |************************** | 16384 -> 32767 : 15 |********** | 32768 -> 65535 : 2 |* | 65536 -> 131071 : 2 |* | New Visualiza ons and GUIs Ne lix Intended Usage Self-service UI:
Flame Graphs Tracing Reports …
should be open sourced; you may also build/buy your own Conquer Performance
Measure anything Introducing BPF BPF TRACING A Linux Tracing Timeline
• 1990’s: Sta c tracers, prototype dynamic tracers • 2000: LTT + DProbes (dynamic tracing; not integrated) • 2004: kprobes (2.6.9) • 2005: DTrace (not Linux), SystemTap (out-of-tree) • 2008: race (2.6.27) • 2009: perf (2.6.31) • 2009: tracepoints (2.6.32) • 2010-2016: race & perf_events enhancements • 2014-2016: BPF patches also: LTTng, ktap, sysdig, ... Ye Olde BPF Berkeley Packet Filter
# tcpdump host 127.0.0.1 and port 22 -d (000) ldh [12] (001) jeq #0x800 jt 2 jf 18 (002) ld [26] (003) jeq #0x7f000001 jt 6 jf 4 (004) ld [30] (005) jeq #0x7f000001 jt 6 jf 18 (006) ldb [23] (007) jeq #0x84 jt 10 jf 8 (008) jeq #0x6 jt 10 jf 9 (009) jeq #0x11 jt 10 jf 18 (010) ldh [20] (011) jset #0x1fff jt 18 jf 12 (012) ldxb 4*([14]&0xf) (013) ldh [x + 14] (014) jeq #0x16 jt 17 jf 15 (015) ldh [x + 16] (016) jeq #0x16 jt 17 jf 18 (017) ret #65535 (018) ret #0 BPF Enhancements by Linux Version
• 3.18: bpf syscall eg, Ubuntu: • 3.19: sockets • 4.1: kprobes
• 4.4: bpf_perf_event_output 16.04 • 4.6: stack traces • 4.7: tracepoints 16.10 • 4.9: profiling Enhanced BPF is in Linux BPF
• aka eBPF == enhanced Berkeley Packet Filter – Lead developer: Alexei Starovoitov (Facebook) • Many uses – Virtual networking – Security – Programma c tracing • Different front-ends – C, perf, bcc, ply, … BPF mascot BPF for Tracing
User Program Kernel
1. generate verifier
BPF bytecode kprobes
2. load BPF uprobes per- event 3. perf_output tracepoints data 3. async sta s cs maps read Raw BPF
samples/bpf/sock_example.c 87 lines truncated C/BPF
samples/bpf/tracex1_kern.c 58 lines truncated bcc
• BPF Compiler Collec on Tracing layers: – h ps://github.com/iovisor/bcc – Lead developer: Brenden Blanco bcc tool bcc tool … (PlumGRID) • Includes tracing tools bcc … • Front-ends Python lua front-ends – Python user – Lua kernel Kernel – C helper libraries BPF Events bcc/BPF
bcc examples/tracing/bitehist.py en re program ply/BPF
h ps://github.com/wkz/ply/blob/master/README.md en re program The Tracing Landscape, Dec 2016 (my opinion)
dtrace4L. ply/BPF
(less brutal) ktap sysdig perf stap