Enhancing Memory Error Detection for Large-Scale Applications and Fuzz testing
Wookhyun Han, Byunggil Joe, Byoungyoung Lee*, Chengyu Song†, Insik Shin
KAIST, *Purdue, †UCR
1 Memory error
glibc: getaddrinfo Heartbleed Shellshock stack-based buffer overflow • Information leakage – Heartbleed • Privilege escalation – Shellshock • Remote code execution – Shellshock, glibc, Conficker
2 Memory error detection
• Pointer-based [SoftBound+CETS, Intel MPX] • Hardware support (cannot detect temporal memory errors) • Challenges to support complex applications
• Redzone-based [AddressSanitizer (ASan)] • Compatible to complex applications • Most popular in practice Google Chrome, Mozilla Firefox, Linux Kernel American Fuzzy Lop (AFL), ClusterFuzz, OSS-Fuzz
3 Redzone-based memory error detection
• Buffer overflow (spatial memory errors) ptrX Shadow memory: a bitmap to validate all addresses objX Check before access
Shadow memory Accessible
4 Redzone-based memory error detection
• Buffer overflow (spatial memory errors) ptrX Shadow memory: a bitmap to validate all addresses objX Check before access Redzone: inaccessible region between objects Shadow memory Accessible Inaccessible (redzone) 4 Redzone-based memory error detection
• Buffer overflow (spatial memory errors) ptrX Shadow memory: a bitmap to validate all addresses objX
Error! Redzone: inaccessible region between objects Shadow memory Accessible Inaccessible (redzone) 4 Redzone-based memory error detection
• Use-after-free (temporal memory errors) ptrX
objX
Accessible Inaccessible Shadow memory 5 Redzone-based memory error detection
• Use-after-free (temporal memory errors) ptrX ptrX
objX free(ptrX) Quarantined Region is invalidated and quarantined, but not actually deallocated
Accessible Inaccessible Shadow memory 5 Redzone-based memory error detection
• Use-after-free (temporal memory errors) ptrX ptrX Hold the objX free(ptrX) Quarantined region until quarantine zone is full (FIFO)
Accessible Inaccessible Shadow memory 5 Redzone-based memory error detection
• Use-after-free (temporal memory errors) ptrX ptrX ptrY
ptrY = Quarantined malloc() TheobjX regionfree( ptrXis )actually objY deallocated, and can be allocated to a new object
Accessible Inaccessible Shadow memory 5 Limitations of redzone-based approach
1. What if a pointer accesses beyond redzone?
ptrX objX
objY
6 Limitations of redzone-based approach
1. What if a pointer accesses beyond redzone?
objX
ptrX objY
Spatial memory error 6 Limitations of redzone-based approach
1. What if a pointer 2. What if a dangling pointer accesses beyond accesses after another object redzone? is allocated in the region?
objX ptrX
ptrX objY objX
Spatial memory error 6 Limitations of redzone-based approach
1. What if a pointer 2. What if a dangling pointer accesses beyond accesses after another object redzone? is allocated in the region?
objX ptrX ptrX
ptrX objY objX objZ
Spatial memory error Temporal memory error 6 Limitations of redzone-based approach
1. What if a pointer 2. What if a dangling pointer accesses beyond accesses after another object redzone? is allocated in the region?
objX ptrX ptrX
ptrX objY Cannot detect!objX objZ
Spatial memory error Temporal memory error 6 Motivation P1
• To enhance detectability of redzone- obj1 based memory error detection • P1. Large gap to detect spatial memory errors P1 • P2. Large quarantine zone to detect temporal memory errors obj1
P1
7 Motivation P1
• To enhance detectability of redzone- P2 obj1 based memory error detection • P1. Large gap to detect spatial memory errors P1 • P2. Large quarantine zone to detect temporal memory errors P2 obj2obj1
P1
7 Motivation P1
• To enhance detectability of redzone- P2 obj1 based memory error detection • P1. Large gap to detect spatial memory errors P1 • P2. Large quarantine zone to detect temporal memory errors P2 obj2obj1
Huge physical memory P1 required
7 MEDS overview
• Enhances detectability of redzone-based memory error detection
• Idea: Fully utilize 64-bit virtual address space to support • P1. Large gap to detect spatial error • P2. Large quarantine zone to detect temporal error
• Approach: minimize physical memory use • Page aliasing allocator and page protection • Hierarchical memory error detection
8 Page aliasing (P1)
• Maps multiple virtual pages to single physical page Virtual obj1
A memory page obj2 Allocated
Redzone
Page aliasing
obj4 9 Page aliasing (P1)
• Maps multiple virtual pages to single physical page Virtual obj1
Physical obj1 obj2 A memory page obj2 obj3 obj4 Allocated
Redzone
Page aliasing
obj4 9 Page aliasing (P1)
• Maps multiple virtual pages to single physical page Virtual obj1 Redzone itself does not Physical occupy physical memory obj1 obj2 A memory page obj2 obj3 obj4 Allocated
Redzone
Page aliasing
obj4 9 Page protection (P1)
• Redzone only pages are unmapped
Virtual
obj1 Physical
obj1 A memory page obj2 obj3 Unmapped page obj4 obj2 Allocated
Redzone
Page aliasing 10 Page protection (P1)
• Redzone only pages are unmapped Do not occupy shadow memory and physical Virtual memory obj1 Physical
obj1 A memory page obj2 obj3 Unmapped page obj4 obj2 Allocated
Redzone
Page aliasing 10 Page aliasing & Page protection (P2)
Virtual obj1
Physical obj1 obj2 A memory page obj3 obj4 obj4 Unmapped page
Allocated
Redzone
Page aliasing 11 Page aliasing & Page protection (P2)
Virtual Virtual obj1
Quarantined
Physical Physical obj1 obj2 obj2 A memory page obj3 obj3 obj4 obj4 obj4 obj4 Unmapped page
Allocated
Redzone
Page aliasing 11 Page aliasing & Page protection (P2)
Virtual Virtual obj1
Quarantined
Physical Physical obj1 objX obj2 obj2 A memory page obj3 obj3 obj4 obj4 obj4 obj4 Unmapped page
Allocated objX Redzone
Page aliasing 11 Page aliasing & Page protection (P2)
Virtual Virtual Reuse physical obj1 memory immediately, Quarantined while not reusing Physical Physical virtual addresses obj1 objX obj2 obj2 A memory page obj3 obj3 obj4 obj4 obj4 obj4 Unmapped page
Allocated objX Redzone
Page aliasing 11 Hierarchical memory error detection
• Many different ways to represent redzones Further optimizing physical memory uses ptr
12 Hierarchical memory error detection
• Many different ways to represent redzones Further optimizing physical memory uses ptr
#1. Shadow memory is invalid
12 Hierarchical memory error detection
• Many different ways to represent redzones Further optimizing physical memory uses ptr
#1. Shadow memory is invalid
#2. Virtual page is unmapped
12 Hierarchical memory error detection
• Many different ways to represent redzones Further optimizing physical memory uses ptr
#1. Shadow memory is invalid
#2. Virtual page is unmapped
#3. Shadow memory is unmapped
12 Evaluation
• Configuration ASan MEDS Improv. Redzone 8-1024 bytes 4MB 16,384x Quarantine 128MB 80TB 65,536x • ASan cannot use configuration for MEDS (lack of memory) • Compatibility • Performance: 2 times slowdown • Detection (fuzz testing): 68% more detection
13 Compatibility
• Unit tests from real-world applications • Test cases in Chrome, Firefox, Nginx • All Passed
• Memory error unit tests • ASan unit tests • All Passed • NIST Juliet test suites • All Passed except random access tests ASan: 35% vs. MEDS: 98%
14 Micro-scale performance overhead
• TLB misses • 5 times more than ASan (more virtual pages with page aliasing)
• Number of system calls • mmap(), munmap(), and mremap() • 32 times more than ASan (page aliasing and page protection)
• Memory footprint • 218% more than baseline • 68% more than ASan (much larger redzone and quarantine)
15 End-to-end performance overhead
• 108% compared to baseline, 86% to ASan 4
3
2
1 Baseline
0 Chrome Firefox Apache Nginx ASan MEDS 16 End-to-end performance overhead
• 108% compared to baseline, 86% to ASan 4
3 41% to baseline 22% to ASan 2
1 Baseline
0 Chrome Firefox Apache Nginx ASan MEDS 16 End-to-end performance overhead
• 108% compared to baseline, 86% to ASan Large number of 4 small objects on stack 41% to baseline 3 243% to baseline 22% to ASan 2 211% to ASan
1 Baseline
0 Chrome Firefox Apache Nginx ASan MEDS 16 Detection (fuzz testing)
• Run AFL (8 cores, 6 hours) • Despite the performance overhead, explore 68.3% more unique crashes than ASan 4 3.5 3 2.5 2 1.5 1 ASan 0.5 0
17 Detection (fuzz testing)
• Run AFL (8 cores, 6 hours) • Despite the performance overhead, explore 68.3% more unique crashes than ASan 4 MEDS finds more unique crashes in 3.5 3 initial phase, but saturated in the end 2.5 2 1.5 1 ASan 0.5 0
17 Detection (fuzz testing)
• Number of unique crashes with time spent (metacam)
70 Saturated 60 50 40 30 20
Found crashes Found 10 0 1 2 3 4 5 6 7 8 Time spent (hrs)
ASan MEDS
18 How MEDS explores more crashes?
• More input sets can be detected • Higher probability to detect int a[10]; • Bugs can be found earlier than ASan a[x] = x; • Fuzzer can focus on the other paths
• MEDS can detect the cases that ASan cannot detect struct A { • Always bypass redzone int num[10]; }; • e.g., Miscalculation of structure array size struct A *a = • Size of the structure is larger than redzone malloc(sizeof(struct A)); size ... • Access to certain element cannot be detected. (a+i)->num[8] = i;
19 How MEDS explores more crashes?
• More input sets can be detected • Higher probability to detect int a[10]; • Bugs can be found earlier than ASan a[x] = x; • Fuzzer can focus on the other paths
• MEDS can detect the cases that ASan cannot detect struct A { • Always bypass redzone int num[10]; }; • e.g., Miscalculation of structure array size struct A *a = • Size of the structure is larger than redzone malloc(sizeof(struct A)); size ... • Access to certain element cannot be detected. (a+i)->num[8] = i;
19 How MEDS explores more crashes?
• More input sets can be detected • Higher probability to detect int a[10]; • Bugs can be found earlier than ASan a[x] = x; • Fuzzer can focus on the other paths
• MEDS can detect the cases that ASan cannot detect struct A { • Always bypass redzone int num[10]; }; • e.g., Miscalculation of structure array size struct A *a = • Size of the structure is larger than redzone malloc(sizeof(struct A)); size ... • Access to certain element cannot be detected. (a+i)->num[8] = i;
19 How MEDS explores more crashes?
• More input sets can be detected • Higher probability to detect int a[10]; • Bugs can be found earlier than ASan a[x] = x; • Fuzzer can focus on the other paths
• MEDS can detect the cases that ASan cannot detect struct A { • Always bypass redzone int num[10]; }; • e.g., Miscalculation of structure array size struct A *a = • Size of the structure is larger than redzone malloc(sizeof(struct A)); size ... • Access to certain element cannot be detected. (a+i)->num[8] = i;
19 Conclusion
• Idea • Support large gap and large quarantine zone • Approach • Page aliasing and page protection • Hierarchical memory error detection • Despite overhead (108%), MEDS finds more crashes during fuzz testing (68.3%) • Open source – will be available soon • https://github.com/purdue-secomp-lab/MEDS • Please use to detect bugs
20 Thank you for listening!
21