Kernel Tracing With eBPF Unlocking God Mode on Linux Jeff Dileo Andy Olsen . @chaosdatumz @0lsen_ . 35C3 Who are we? Jeff Dileo (@chaosdatumz) Andy Olsen (@0lsen_) • Unix aficionado • Ultimate frisbee enthusiast • Agent of chaos • Amateur chiptune artist • Consultant / Research Director @ NCC Group • Security Consultant @ NCC Group • I like to do terrible things to/with/in: • Il ne parle pas Français • programs • languages • runtimes • memory • kernels • packets • bytes • ... Outline • eBPF • Tracing with eBPF • Defensive eBPF • eBPF Secure Coding Gotchas • Offensive eBPF • Q&A • But what is BPF? eBPF — Background • ”extended” BPF eBPF — Background • ”extended” BPF • But what is BPF? eBPF — BPF • Berkeley Packet Filter • Limited instruction set for a bytecode virtual machine • Originally created to implement FAST programmatic network filtering in kernel • has a few (2) 32-bit registers (and a hidden frame pointer) • load/store, conditional jump (forward), add/sub/mul/div/mod, neg/and/or/xor, bitshift • tcpdump -i any -n 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0' (000) ldh [14] (001) jeq #0x800 jt 2 jf 10 (002) ldb [25] (003) jeq #0x6 jt 4 jf 10 (004) ldh [22] (005) jset #0x1fff jt 10 jf 6 (006) ldxb 4*([16]&0xf) (007) ldb [x + 29] (008) jset #0x12 jt 9 jf 10 (009) ret #262144 (010) ret #0 eBPF — eBPF • ”extended” Berkeley Packet Filter • ”designed to be JITed with one to one mapping” • ”originally designed with the possible goal in mind to write programs in ’restricted C’” • socket filters, packet processing, tracing, internal backend for ”classic” BPF, and more... • File descriptor-based API through bpf(2) syscall • Provide: • An array of bytecode instructions • Type of eBPF program (e.g. BPF_PROG_TYPE_SOCKET_FILTER, BPF_PROG_TYPE_KPROBE, etc.) • Other type-specific metadata • Receive: • (on success) A file descriptor referring to the in-kernel compiled eBPF program • The power of eBPF is really in the kernel APIs that will accept an eBPF descriptor and plug it into things eBPF — eBPF static int add_lookup_instructions(BPFProgram *p, int map_fd, int protocol, bool is_ingress, int verdict){ ... struct bpf_insn insn[] = { BPF_JMP_IMM(BPF_JNE, BPF_REG_7, htobe16(protocol), 0), ... BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), BPF_MOV32_IMM(BPF_REG_2, addr_offset), BPF_MOV64_REG(BPF_REG_3, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_3,-addr_size), BPF_MOV32_IMM(BPF_REG_4, addr_size), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_skb_load_bytes), ... BPF_LD_MAP_FD(BPF_REG_1, map_fd), BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2,-addr_size - sizeof(uint32_t)), BPF_ST_MEM(BPF_W, BPF_REG_2, 0, addr_size * 8), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1), BPF_ALU32_IMM(BPF_OR, BPF_REG_8, verdict), }; ... Listing 1: systemd/src/core/bpf-firewall.c eBPF — Important BPF to eBPF Changes • now 10 64-bit registers, directly mapped to HW CPU registers • R0: return value from in-kernel function, and exit value for eBPF program • R1-R5: arguments from eBPF program to in-kernel function • R6-R9: callee saved registers that in-kernel function will preserve • R10: read-only frame pointer to access stack • new bpf_call instruction • HW-based register passing convention for zero overhead calls from/to other kernel functions • Used to call other eBPF programs and ”helper” functions • Bytecode validator (”verifier”) • Helper functions • Set of native kernel functions exposed to eBPF code • Context-dependent (e.g. packet processing eBPF cannot call kernel memory read helper) • Argument registers validated against call spec for each helper function • network tunneling • custom iptables rules • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • Reduce need for kernel modules • firewall subsystem with rules implemented entirely in eBPF As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively Why eBPF? • HIGH PERFORMANCE in-plane packet processing • network tunneling • custom iptables rules • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • Reduce need for kernel modules • firewall subsystem with rules implemented entirely in eBPF As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively Why eBPF? • (safe) HIGH PERFORMANCE in-plane packet processing • custom iptables rules • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • Reduce need for kernel modules • firewall subsystem with rules implemented entirely in eBPF As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively Why eBPF? • (safe) HIGH PERFORMANCE in-plane packet processing • network tunneling • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • Reduce need for kernel modules • firewall subsystem with rules implemented entirely in eBPF As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively Why eBPF? • (safe) HIGH PERFORMANCE in-plane packet processing • network tunneling • custom iptables rules • Reduce need for kernel modules • firewall subsystem with rules implemented entirely in eBPF As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively Why eBPF? • (safe) HIGH PERFORMANCE in-planekernel packet processing programmatic operations • network tunneling • custom iptables rules • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • firewall subsystem with rules implemented entirely in eBPF As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively Why eBPF? • (safe) HIGH PERFORMANCE in-planekernel packet processing programmatic operations • network tunneling • custom iptables rules • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • Reduce need for buggy kernel modules • firewall subsystem with rules implemented entirely in eBPF As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively Why eBPF? • (safe) HIGH PERFORMANCE in-planekernel packet processing programmatic operations • network tunneling • custom iptables rules • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • Reduce need for buggy kernel modules As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively Why eBPF? • (safe) HIGH PERFORMANCE in-planekernel packet processing programmatic operations • network tunneling • custom iptables rules • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • Reduce need for buggy kernel modules • firewall subsystem with rules implemented entirely in eBPF Why eBPF? • (safe) HIGH PERFORMANCE in-planekernel packet processing programmatic operations • network tunneling • custom iptables rules • syscall filtering (mix of classic [for now] BPF for seccomp, eBPF for cgroups shenanigans) • Reduce need for buggy kernel modules • firewall subsystem with rules implemented entirely in eBPF As more eBPF features have been added in newer kernel versions, the ”why” of eBPF has changed retroactivively • Linux will not likely gain such unified facilities • It provides a base to build more complicated analysis tooling on • eBPF has the potential to give DTrace a run for its money • The power of DTrace is in its providers (event/data sources) • eBPF is more programmatic, but lower level • DTrace is amazing at one-off human-driven system analysis • But eBPF enables very efficient dynamic always-on whole system analysis Why eBPF? — OK, but really, why? • eBPF is different things to different people • Personally, we like being able to selectively instrument an entire OS without making it crawl • The title of this talk is ”Kernel Tracing With eBPF” :) • Linux will not likely gain such unified facilities • It provides a base to build more complicated analysis tooling on • The power of DTrace is in its providers (event/data sources) • eBPF is more programmatic, but lower level • DTrace is amazing at one-off human-driven system analysis • But eBPF enables very efficient dynamic always-on whole system analysis Why eBPF? — OK, but really, why? • eBPF is different things to different people • Personally, we like being able to selectively instrument an entire OS without making it crawl • The title of this talk is ”Kernel Tracing With eBPF” :) • eBPF has the potential to give DTrace a run for its money • It provides a base to build more complicated analysis tooling on • eBPF is more programmatic, but lower level • DTrace is amazing at one-off human-driven system analysis • But eBPF enables very efficient dynamic always-on whole system analysis Why eBPF? — OK, but really, why? • eBPF is different things to different people • Personally, we like being able to selectively instrument an entire OS without making it crawl • The title of this talk is ”Kernel Tracing With eBPF” :) • eBPF has the potential to give DTrace a run for its money • The power of DTrace is in its providers (event/data sources) • Linux will not likely gain such unified facilities • DTrace is amazing at one-off human-driven system analysis • But eBPF enables very efficient dynamic always-on whole system analysis Why eBPF? — OK, but really,