SM2-TES: Functional Programming and Property-Based Testing, Day 11 Jan Midtgaard MMMI, SDU Outline Motivation Challenges in C Programming Fuzz Testing Fuzzing in the Bigger Picture 2/44 Motivation Positive vs. negative testing QuickCheck (as we’ve covered so far) focuses mainly on positive testing: Testing that systems/programs/libraries act as desired on valid input. These days it is increasingly important to also perform negative testing: Testing that systems/programs/libraries also act meaningfully on invalid input. Security issues typically fall in the second category: SQL or JS injections, buffer overflows, . Many of these are memory safety issues 4/44 A real problem in practice! 2014 - Heartbleed was a security bug in the OpenSSL cryptography library From https://en.wikipedia.org/wiki/Heartbleed 5/44 A real problem in practice! https://www.zdnet.com/article/microsoft-70-percent-of-all-security-bugs-are-memory-safety-issues/ 6/44 A real problem in practice! https://www.zdnet.com/article/chrome-70-of-all-security-bugs-are-memory-safety-issues/ 6/44 2020 Most Dangerous Software Weaknesses Here’s the 2020 CWE Top 25: # ID Name 1 79 Improper Neutralization of Input During Web Page Generation (XSS) 2 787 Out-of-bounds Write 3 20 Improper Input Validation 4 125 Out-of-bounds Read 5 119 Improper Restriction of Operations within the Bounds of a Memory Buffer 6 89 Improper Neutralization of Special Elements used in an SQL Command 7 200 Exposure of Sensitive Information to an Unauthorized Actor 8 416 Use After Free 9 352 Cross-Site Request Forgery (CSRF) 10 78 Improper Neutralization of Special Elements used in an OS Command 11 190 Integer Overflow or Wraparound 12 22 Improper Limitation of a Pathname to a Restricted Directory 13 476 NULL Pointer Dereference 14 287 Improper Authentication 15 434 Unrestricted Upload of File with Dangerous Type . http://cwe.mitre.org/top25/archive/2020/2020_cwe_top25.html 7/44 Memory safety is a security concern These are mostly caused by using C and C++ which (unlike OCaml and Java) are memory unsafe languages To address these challenges working software engineers have developed: Sanitizers – to help raise alarms Fuzz testing (or fuzzing) – another randomized testing approach(!) Today we will study both. First: Let’s better understand why this is an issue... 8/44 Challenges in C Programming The challenges of C programming. (1/2) Writing C code is challenging: C’s manual memory management requires you to: allocate chunks of memory with malloc (or similar) release the allocated chunks again with free Furthermore: Allocated memory is (generally) not initialized for you Indexing out of bounds is not caught... What could possibly go wrong? 10 / 44 The challenges of C programming. (2/2) Writing safe and proper C code is error prone: writing out-of-bounds in the heap or stack reading out-of-bounds in the heap or stack uninitialized reads use-after-free double freeing . Historically compilers did not help catch these. With a crash you would be lucky. More likely: program sometimes exhibits weird behavior 11 / 44 Enter ASan Google software engineers at some point designed AddressSanitizer (ASan) which was a game changer. https://github.com/google/sanitizers/wiki/AddressSanitizer http://clang.llvm.org/docs/AddressSanitizer.html ASan’s job is simply to detect memory addressing errors In a language like Java we are spoiled and take, e.g., IndexOutOfBounds exceptions, for granted. These days LLVM (and GCC) has ASan support built in. LLVM supports Linux, *BSD, Mac, Android, Windows Historically, a range of tools predated ASan: Valgrind, Electric Fence, ... 12 / 44 An example C program #include <stdio.h> #include <stdlib.h> int main() { char *p = malloc(5); p[0] = 'h'; p[1] = 'e'; p[2] = 'l'; p[3] = 'l'; p[4] = 'o'; p[5] = '\0'; /* this index is out of bounds */ printf("%s\n",p); return 0; } 13 / 44 An example C program #include <stdio.h> #include <stdlib.h> int main() { char *p = malloc(5); p[0] = 'h'; p[1] = 'e'; p[2] = 'l'; p[3] = 'l'; p[4] = 'o'; p[5] = '\0'; /* this index is out of bounds */ printf("%s\n",p); return 0; } $ clang -Wall -Wextra -pedantic -o example2 example2.c $ ./example2 hello $ Hm, it runs without any flags raised... 13 / 44 Same example compiled with ASan $ clang -Wall -Wextra -pedantic -g -o example2 -fsanitize=address example2.c $ 14 / 44 Same example compiled with ASan $ clang -Wall -Wextra -pedantic -g -o example2 -fsanitize=address example2.c $ ./example2 ================================================================= ==4922==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000000b5 at pc 0x000102604df5 bp 0x7fff5d5fb7d0 sp 0x7fff5d5fb7c8 WRITE of size 1 at 0x6020000000b5 thread T0 #0 0x102604df4 in main example2.c:12 #1 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234) 0x6020000000b5 is located 0 bytes to the right of 5-byte region [0x6020000000b0,0x6020000000b5) allocated by thread T0 here: #0 0x102666e9c in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x58e9c) #1 0x102604bfa in main example2.c:6 #2 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234) SUMMARY: AddressSanitizer: heap-buffer-overflow example2.c:12 in main Shadow bytes around the buggy address: 0x1c03ffffffc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03fffffff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c0400000000: fa fa fd fd fa fa fd fd fa fa 00 04 fa fa 00 00 =>0x1c0400000010: fa fa 00 06 fa fa[05]fa fa fa fa fa fa fa fa fa 0x1c0400000020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [...] ==4922==ABORTING Abort trap: 6 14 / 44 Same example compiled with ASan $ clang -Wall -Wextra -pedantic -g -o example2 -fsanitize=address example2.c $ ./example2 ================================================================= ==4922==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000000b5 at pc 0x000102604df5 bp 0x7fff5d5fb7d0 sp 0x7fff5d5fb7c8 WRITE of size 1 at 0x6020000000b5 thread T0 #0 0x102604df4 in main example2.c:12 #1 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234)Line causing overflow 0x6020000000b5 is located 0 bytes to the right of 5-byte region [0x6020000000b0,0x6020000000b5) allocated by thread T0 here: #0 0x102666e9c in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x58e9c) #1 0x102604bfa in main example2.c:6 #2 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234) SUMMARY: AddressSanitizer: heap-buffer-overflow example2.c:12 in main Shadow bytes around the buggy address: 0x1c03ffffffc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03fffffff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c0400000000: fa fa fd fd fa fa fd fd fa fa 00 04 fa fa 00 00 =>0x1c0400000010: fa fa 00 06 fa fa[05]fa fa fa fa fa fa fa fa fa 0x1c0400000020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [...] ==4922==ABORTING Abort trap: 6 14 / 44 Same example compiled with ASan $ clang -Wall -Wextra -pedantic -g -o example2 -fsanitize=address example2.c $ ./example2 ================================================================= ==4922==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000000b5 at pc 0x000102604df5 bp 0x7fff5d5fb7d0 sp 0x7fff5d5fb7c8 WRITE of size 1 at 0x6020000000b5 thread T0 #0 0x102604df4 in main example2.c:12 #1 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234)Line causing overflow 0x6020000000b5 is located 0 bytes to the right of 5-byte region [0x6020000000b0,0x6020000000b5) allocated by thread T0 here: #0 0x102666e9c in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x58e9c) #1 0x102604bfa in main example2.c:6 #2 0x7fffe6664234 in start (libdyld.dylib:x86_64+0x5234)Line that allocated SUMMARY: AddressSanitizer: heap-buffer-overflow example2.c:12 in main Shadow bytes around the buggy address: 0x1c03ffffffc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03ffffffe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c03fffffff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x1c0400000000: fa fa fd fd fa fa fd fd fa fa 00 04 fa fa 00 00 =>0x1c0400000010: fa fa 00 06 fa fa[05]fa fa fa fa fa fa fa fa fa 0x1c0400000020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x1c0400000060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [...] ==4922==ABORTING Abort trap: 6 14 / 44 ASan doesn’t catch everything For example: #include <stdio.h> #include <stdlib.h> int main(int argc, char** argv) /* argc is an argument counter */ { char a[10]; /* stack allocation */ a[5] = 0; printf("running %s argc is %i\n", argv[0], argc); if (a[argc]) /* Reading uninitialized memory */ printf("what?