FIX ALL THE BUGS AMERICAN FUZZY LOP AND ADDRESS SANITIZER Hanno Böck 1 INTRODUCTION Hanno Böck, freelance journalist and hacker. Writing for Golem.de and others. Author of monthly Bulletproof TLS Newsletter. Fuzzing Project, funded by Linux Foundation's Core Infrastructure Initiative. 2 BUG EXAMPLE (QT file src/plugins/platforminputcontexts/compose/qcomposeplatforminputcontext.cpp) 3 KDE / QT BUGS Use aer free in qmake Underflow / out of bounds read in QT / QCompose Out of bounds read in QtGui Out of bounds read in Kwin (last 2 not sure if I should blame Xorg/xcb API) 4 FOR FAIRNESS: GNOME #762417: Out of bounds read in glib / token parser #762483: Out of bounds read in glib / unicode parser #768441: Heap overflow in gnome-session parameter parsing #770027: Out of bounds read in pango / test suite 5 BUG EXAMPLE /* +2 for our new arguments, +1 for NULL */ new_argv = g_malloc (argc + 3 * sizeof (*argv)); (gnome-session 3.20.1, bug #768441) 6 ADDRESS SANITIZER (ASAN) All this bugs can be trivially found with Address Sanitizer. Just add -fsanitize=address to the compiler flags in GCC/CLANG. 7 FIND BUGS WITH ASAN ./configure CFLAGS="-fsanitize=address -g" \ CXXFLAGS="-fsanitize=address -g" \ LDFLAGS="-fsanitize=address" make make check 8 WHAT IS ASAN DOING? Shadow memory tracking which memory areas are valid. Finds out of bounds access (read/write) and use aer free bugs (and other less common issues). 9 OUT OF BOUNDS READ #include <stdio.h> int main() { int a[2] = {3,1}; int i = 2; printf("%i\n", a[i]); } 10 USE AFTER FREE #include <stdio.h> #include <stdlib.h> int main() { char *c = calloc(10,1); printf("%i\n", c[0]); free(c); printf("%i\n", c[1]); } 11 MISSION: TEST EVERYTING WITH ASAN Every project using C/C++ code should test with ASAN. 12 GENTOO WITH ASAN Why not build everything in a Linux system with ASAN? Gentoo + ASAN: It runs! Found bugs in Bash, Coreutils/Shred, man-db, Pidgin-OTR, Courier, Syslog-NG, Screen, Claws-Mail, ProFTPD ICU, TCL, Dovecot, Glib, GNOME, Qt, KDE, Libarchive, Squid, CMake, Gettext, SpamAssassin, ... 13 FUZZING Throw garbage at soware. 14 FUZZING Example: Image parser Take valid image, add random errors to it, see if parser crashes. 15 16 DARPA CYBER GRAND CHALLENGE DARPA's Cyber Grand Challenge: Early Highlights from the C... 17 RARELY TOLD STORY Most teams and all three winners of the Darpa Cyber Grand Challenge used American Fuzzy Lop (AFL) with some addons as a bug finding tool. 18 AMERICAN FUZZY LOP 19 FUZZING STRATEGIES Dumb fuzzing: Easy, but not very effective. Template-based fuzzing: More effective, lots of work, doesn't scale. Coverage-based fuzzing: Easy and effective. 20 AMERICAN FUZZY LOP American Fuzzy Lop (AFL) made the idea of coverage-based fuzzing popular. Step 1: Compile with afl-wrapper (afl-gcc or afl-clang-fast) Step 2: Fuzz 21 AFL IS EASY ./configure CC=afl-clang-fast CXX=afl-clang-fast++ --disable- shared; make [put sample file into directory in/] afl-fuzz -i in -o out [path_to_parser_executable] @@ 22 AMERICAN FUZZY LOP 23 AFL FOUND BUGS IN ... OpenSSL, OpenSSH, libjpeg-turbo, libpng, sqlite, GnuPG, Bash, Stagefright, BIND, NTPD, ... There isn't any major piece of C parser code where AFL hasn't found bugs. 24 FUZZING WITH SUPERPOWERS AFL finds bugs, ASAN finds more bugs. Best to combine AFL and ASAN. Set AFL_USE_ASAN=1 and add "-m none". 25 26 AFL/ASAN MEET HEARTBLEED Could Fuzzing find the Heartbleed bug? Experiment: Implement wrapper that accepts handshake messages as file input. Success aer ~ 6 hours. (Kostya Serebryany showed that LibFuzzer finds it in 5 minutes) 27 LIBFUZZER Also coverage based fuzzing. Part of LLVM/CLANG. AFL fuzzes executables, LibFuzzer fuzzes functions. Faster, but more initial work (write code). 28 LIBFUZZER EXAMPLE #include <stdint.h> #include <stddef.h> #include <openssl/asn1.h> extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) { ASN1_STRING *out = 0; ASN1_mbstring_copy(&out, Data, Size, MBSTRING_BMP, 0); if (out!=0) ASN1_STRING_free(out); return 0; } 29 DIFFERENTIAL TESTING Typical fuzzing: look for crashes / memory corruption. Differential testing: Feed two different implementations doing the same thing the same input, compare the output. 30 DIFFERENTIAL TESTING ON MATH Crypto is important. Crypto uses mathematics - but is the math correct? 31 BN_SQR() BUG (CVE-2014-3570) OpenSSL had a bug in the squaring function. On very rare inputs (1 in 2^128) it produced wrong results. Surprising: AFL found this bug (first tested by Ralph-Philipp Weinmann). 32 AFL IS GOOD AT THIS OpenSSL / BN_mod_exp (CVE-2015-3193) Nettle / ECC (CVE-2015-8803, CVE-2015-8804) NSS / mp_div() / exptmod() (CVE-2016-1938) OpenSSL / Poly1305 MatrixSSL / pstm_exptmod (CVE-2016-6885, CVE-2016-6886, CVE-2016-6887) 33 WHAT IS A VULNERABILIT? The vast majority of bugs found with AFL+ASAN are heap out of bounds reads. Are these vulnerabilities? Sometimes (Heartbleed!) Be prepared for pointless discussions whether these should be called vulnerabilities. IMHO: Just fix them and skip that discussion. 34 MORE TOOLS 35 OTHER SANITIZERS Undefined Behavior Sanitizer (UBSAN) - easy to use, but finds many bugs, mostly not very interesting. Memory Sanitizer (MSAN) - finds uninitialized memory, tricky to use. Thread Sanitizer (TSAN) - mostly interesting for larger C++ projects. 36 UNDEFINED BEHAVIOR SANITIZER (UBSAN) #include <limits.h> int main() { int i = 10; int j = -1; i <<= j; i = INT_MAX; i++; } 37 MEMORY SANITIZER (MSAN) int main(int argc, char **argv) { int x[10]; x[1] = 1; if (x[argc]) return 1; } 38 KASAN, KUBSAN, KTSAN, SYZCALLER Sanitizers and coverage-based fuzzing have been adapted for the Linux Kernel. 39 NETWORK-FUZZING Tricky - no really good solution yet. Preeny - uses LD_PRELOADing. Patch from Doug Birdwell for AFL, fragile. Wrappers to parser functions. 40 AFL + SYMBOLIC EXECUTION Some work on this (e. g. in Darpa Challenge), but nothing easily usable yet. Will have to proove it's useful. 41 IT'S FREE All presented tools (AFL, LibFuzzer, ASAN, other Sanitizers, Preeny, KASAN, Syzcaller) are published as Free and Open Source Soware. 42 THE C/C++ PROBLEM Most fuzzing/ASAN-related bugs are typical C/C++ problems. Maybe we should just rewrite everything in Rust. 43 COMPARING VENDOR REACTIONS 44 DPKG 45 DPKG 2015-11-18: Reported 2 bugs in .deb parsing 2015-11-26: Debian and Ubuntu publish updates and security advisories (USN-2820-1, DSA-3407-1) 46 RPM 47 RPM 2015-11-20: Reported 3 bugs in .rpm parsing to Red Hat Answer: We already got 30 crash reports, may take some time. 48 RPM... RPM is an independent project since 2007, used by Red Hat, Suse and others. Or not? rpm.org belongs to Red Hat Inc. Red Hat Security: "However, we don't own rpm.org domain, it's upstream project, so there's not much we can do about it." 49 RPM.ORG Trac installation. Trying to register account: Certificate error. To create a bug you should ask for permission in IRC or on the mailing list. 50 RPM REPOSITORY RPM development happens on Github these days. The rpm.org webpage does not mention that. 51 WHAT'S THE LATEST VERSION OF RPM? According to rpm.org/releases: 4.12.0.1 According to Github repository: 4.12.0 According to Fedora: 4.13.0 52 STATUS RPM One Stack Buffer Overflow still unfixed in the latest Git Code. No release (that can be found) since 2014. There are more bugs, including ones that happen pre- signature-check. 53 ADVERTISEMENT BLOCK: BERLINSEC MEETUP Tomorrow (5th Sept) at Mozilla Berlin Community space https://berlinsec.github.io/ 54 THANKS FOR LISTENING Test with Address Sanitizer Fuzz your soware Questions? https://fuzzing-project.org/ 55.