Evolution of kernel fuzzers in NetBSD Siddharth Muralee Team bi0s >_ $ whoami ● Siddharth Muralee (R3x) ● Third year BTech CSE @ Amrita Vishwa Vidyapeetham ● CTF player - Team bi0s ● Reverse Engineering and Exploitation ● Core organising team @ InCTF and InCTFj ● Contributor to NetBSD (GSoC ‘18) ○ Kernel Code Quality improvement team ○ Security team 2 >_ The NetBSD Project “Of course it runs NetBSD” ❏ Unix - like BSD Operating System ❏ Open Source ❏ Portability PowerPC, Alpha, SPARC, MIPS, SH3, ARM, amd64, i386, m68k, VAX, ... 3 >_ Agenda ● Issues faced while fuzzing the kernel ● Sanitizers ● Kernel Code Coverage ● Syzkaller ● Future work 4 >_ Issues faced while fuzzing the kernel ● Setup with fuzzer and VMs ○ Handle Crashes ○ Multiarch support ● Scraping Crashes/logs to generate reports ○ Console output ● Restricted kernel APIs ○ Sandboxing ○ Usermode privilege protection 5 >_ Issues faced while fuzzing the kernel (Contd.) ● Generating proper inputs for fuzzing ○ Need to identify proper contexts ● Getting proper reproducers ○ Kernel bug ~= Kernel Panic ○ Identify root cause ● Indetermination of Execution ○ Threads ○ Scheduling 6 Coverage Fuzzer Sanitizers Future Work 7 >_ Sanitizers ● Dynamic testing tools ● Compiler Instrumented ● Available with GCC and Clang ● Fuzzing Aid 8 >_ Types of Sanitizers >_ Address Sanitizer >_ Undefined Behaviour Sanitizer detects invalid address usage bugs finds unspecified code semantic bugs >_ Memory Sanitizer >_ Thread Sanitizer finds uninitialized memory access detects threading bugs bugs >_ Kernel Address Sanitizer (KASAN) ● Overflows ● Use after free (UAFs) Compile NetBSD kernel with : makeoptions KASAN=1 options KASAN Supported in NetBSD : ● amd64 ● aarch64 10 >_ KASAN - Overview ● Poisoning ● Shadow buffer ● Interceptors 11 >_ KASAN - sample report ifconfig gif0 create ifconfig gif0 up [ 50.682919] kASan: Unauthorized Access In 0xffffffff80f22655: \ Addr 0xffffffff81b997a0 [8 bytes, read] [ 50.682919] #0 0xffffffff8021ce6a in kasan_memcpy <netbsd> [ 50.692999] #1 0xffffffff80f22655 in m_copyback_internal <netbsd> ……. [ 50.703622] #13 0xffffffff80fde694 in gif_ioctl <netbsd> [ 50.703622] #14 0xffffffff80fcdb1f in doifioctl <netbsd> 12 >_ KASAN - sample report ifconfig gif0 create ifconfig gif0 up [ 50.682919] kASan: Unauthorized Access In 0xffffffff80f22655: \ Addr 0xffffffff81b997a0 [8 bytes, read] [ 50.682919] #0 0xffffffff8021ce6a in kasan_memcpy <netbsd> [ 50.692999] #1 0xffffffff80f22655 in m_copyback_internal <netbsd> ……. [ 50.703622] #13 0xffffffff80fde694 in gif_ioctl <netbsd> [ 50.703622] #14 0xffffffff80fcdb1f in doifioctl <netbsd> 13 >_ KASAN - sample report ifconfig gif0 create ifconfig gif0 up [ 50.682919] kASan: Unauthorized Access In 0xffffffff80f22655: \ Addr 0xffffffff81b997a0 [8 bytes, read] [ 50.682919] #0 0xffffffff8021ce6a in kasan_memcpy <netbsd> [ 50.692999] #1 0xffffffff80f22655 in m_copyback_internal <netbsd> ……. [ 50.703622] #13 0xffffffff80fde694 in gif_ioctl <netbsd> [ 50.703622] #14 0xffffffff80fcdb1f in doifioctl <netbsd> 14 >_ KASAN - sample report ifconfig gif0 create ifconfig gif0 up [ 50.682919] kASan: Unauthorized Access In 0xffffffff80f22655: \ Addr 0xffffffff81b997a0 [8 bytes, read] [ 50.682919] #0 0xffffffff8021ce6a in kasan_memcpy <netbsd> [ 50.692999] #1 0xffffffff80f22655 in m_copyback_internal <netbsd> ……. [ 50.703622] #13 0xffffffff80fde694 in gif_ioctl <netbsd> [ 50.703622] #14 0xffffffff80fcdb1f in doifioctl <netbsd> 15 >_ Kernel Undefined Behaviour Sanitizer ● Signed overflows ● unportable bit shift Compile NetBSD kernel with : ● Unaligned memory access options KUBSAN 16 >_ KLEAK ● Developed by ○ Thomas Barabosch ○ Maxime Villard ● Detect kernel information leaks ● Uses taint tracking ● Scanned buffers passed from kernel to userspace 17 >_ TODO ● Kernel Memory Sanitizer ● Kernel Thread Sanitizer ● Improve existing Sanitizers 18 Coverage Fuzzer Sanitizers Future Work 19 >_ Coverage Guided Fuzzing ● Instrument the target (Compiler/Binary) ● Get path coverage of each input ● Mutate the input based on the coverage. ● Proved to be more efficient than ordinary fuzzing ○ AFL ○ Honggfuzz ○ Syzkaller 20 >_ Kernel Code Coverage (KCov) ● Compiler instrumentation ● Supported in GCC and Clang Compile NetBSD kernel with : ● Recently ported to NetBSD makeoptions KCOV=1 ● Implemented as a Kernel Module options KCOV ○ Ioctl - Enable/Disable 21 >_ Kernel Code Coverage (KCov) ● Two major modes : ○ Trace PC mode ○ Trace Compare mode ● Just add the compiler flag : ○ `-fsanitize-coverage=trace-pc` - PC mode ○ `-fsanitize-coverage=trace-cmp` - Compare mode 22 >_ Trace PC mode 23 >_ Trace PC mode (Contd.) 24 >_ Trace PC mode (Contd.) ● We do not trace during ○ Boot ○ Interrupts ● Returns back addresses traced ● addr2line - used to find the function names. 25 >_ KCov Example 26 >_ KCov Example 27 >_ KCov Example 28 >_ KCov Example 29 >_ KCov Trace 30 >_ KCov Trace | addr2line sy_invoke /src/sys/sys/syscallvar.h:84 sy_invoke /src/sys/sys/syscallvar.h:86 sys_read /src/sys/kern/sys_generic.c:110 x86_curlwp /repos/obj1/sys/arch/amd64/compile/GENERIC/./machine/cpu.h:67 fd_getfile /src/sys/kern/kern_descrip.c:416 sys_read /src/sys/kern/sys_generic.c:123 sy_invoke /src/sys/sys/syscallvar.h:97 syscall /src/sys/arch/x86/x86/syscall.c:142 31 >_ KCov Trace | addr2line sy_invoke /src/sys/sys/syscallvar.h:84 sy_invoke /src/sys/sys/syscallvar.h:86 sys_read /src/sys/kern/sys_generic.c:110 x86_curlwp /repos/obj1/sys/arch/amd64/compile/GENERIC/./machine/cpu.h:67 fd_getfile /src/sys/kern/kern_descrip.c:416 sys_read /src/sys/kern/sys_generic.c:123 sy_invoke /src/sys/sys/syscallvar.h:97 syscall /src/sys/arch/x86/x86/syscall.c:142 32 Coverage Fuzzer Sanitizers Future Work 33 >_ Syzkaller ● Coverage Guided kernel fuzzer ● Google ● Written in Go and C ● Supported OS’s ○ Linux ○ Fuchsia ○ OpenBSD ○ Akaros ○ NetBSD ○ FreeBSD 34 >_ Syzkaller (Contd.) ● Uses KCov feature ● Primary target : System Calls ● Secondary target : ○ Network Stack ○ Filesystem Stack (Under progress) ● Close to around 2000 bugs 35 VM.. Syz-Manager Kernel. 36 >_ Sample config (Syz-Manager) { "target": "netbsd/amd64", //Target OS and Architecture "workdir": "work", //Directory to store crash details "syzkaller": "./", // Syzkaller Directory "image": "netbsd.img", "sshkey":"netbsdkey", "type": "qemu", "vm": { "qemu": "qemu-system-x86_64", "count": 2, "cpu": 2, "mem": 4048 } } 37 >_ Sample config (Contd.) { "target": "netbsd/amd64", "workdir": "work", "syzkaller": "./", "image": "netbsd.img", // Disk with NetBSD OS installed "sshkey":"netbsdkey", // Ssh Key for the Installed OS "type": "qemu", // VM used "vm": { "qemu": "qemu-system-x86_64", "count": 2, "cpu": 2, "mem": 4048 } } 38 >_ Sample config (Contd.) { "target": "netbsd/amd64", "workdir": "work", "syzkaller": "./", "image": "netbsd.img", "sshkey":"netbsdkey", "procs": 2, "type": "qemu", "vm": { // Config for VM "qemu": "qemu-system-x86_64", // Qemu binary to be used "count": 2, // Number of parallel VMs "cpu": 2, // Number of CPU’s in each VM "mem": 1024 // RAM memory for each VM } } 39 VM.. Syz-executor Syz-Manager Kernel. Syz-fuzzer 40 VM.. Executes Syz-executor Syz-Manager Sends Program Kernel. Syz-fuzzer 41 >_ Creating programs ● Scrape kernel codebase ● Generate function prototypes ● Uses a pseudo formal grammar for representation 42 >_ Creating programs (Contd.) ● System calls are executed using Syz-executor ● Mutating inputs ○ Random inputs ○ bit flips ● Syz-prog2c - converts Syzkaller representation in C programs 43 VM.. Executes Syz-executor Syz-Manager Sends Program Kernel. Coverage Info Syz-fuzzer /dev/kcov V 44 VM.. Executes Syz-executor Syz-Manager Sends Program Kernel. Coverage Info Syz-fuzzer /dev/kcov V 45 VM.. Corpus Executes Syz-executor Data Syz-Manager Sends Program Kernel. Coverage Info Syz-fuzzer /dev/kcov V 46 VM.. Corpus Executes Syz-executor Data Syz-Manager Sends Program Kernel. Coverage Info Syz-fuzzer /dev/kcov V PoCs 47 Role of Coverage Mutate a Generate a new program from the program corpus Execute the Modify the Program Corpus No Did we Yes Collect Coverage encounter new from kcov code paths? 48 Role of Coverage (Contd) Mutate a Generate a new program from the program corpus Execute the Modify the Program Corpus No Did we Yes Collect Coverage encounter new from kcov code paths? 49 Role of Coverage (Contd) Mutate a Generate a new program from the program corpus Execute the Modify the Program Corpus No Did we Yes Collect Coverage encounter new from kcov code paths? 50 Role of Coverage (Contd) Mutate a Generate a new program from the program corpus Execute the Modify the Program Corpus No Did we Yes Collect Coverage encounter new from kcov code paths? 51 Role of Coverage (Contd) Mutate a Generate a new program from the program corpus Execute the Modify the Program Corpus No Did we Yes Collect Coverage encounter new from kcov code paths? 52 Role of Coverage (Contd) Mutate a Generate a new program from the program corpus Execute the Modify the Program Corpus No Did we Yes Collect Coverage encounter new from kcov code paths? 53 Role of Coverage (Contd) Mutate a Generate a new program from the program corpus Execute the Modify the Program Corpus No Did we Yes Collect Coverage encounter new from kcov code paths? 54 1 weekend = 18 bugs 3 fixed! 55 56 >_ Syzbot ● 24/7 automatic fuzzing ● Google Cloud Engine ● Automatically sends bug reports to a mailing list ● Maintains a dashboard ● Also provides possible reproducers 57 https://syzkaller.appspot.com/#netbsd 58 https://syzkaller.appspot.com/#netbsd 59 https://syzkaller.appspot.com/#netbsd