Enhancing Memory Error Detection for Large-Scale Applications and Fuzz testing

Wookhyun Han, Byunggil Joe, Byoungyoung Lee*, Chengyu Song†, Insik Shin

KAIST, *Purdue, †UCR

1 Memory error

glibc: getaddrinfo Heartbleed Shellshock stack-based • Information leakage – Heartbleed • Privilege escalation – Shellshock • Remote code execution – Shellshock, glibc, Conficker

2 Memory error detection

• Pointer-based [SoftBound+CETS, Intel MPX] • Hardware support (cannot detect temporal memory errors) • Challenges to support complex applications

• Redzone-based [AddressSanitizer (ASan)] • Compatible to complex applications • Most popular in practice  Google Chrome, Mozilla Firefox, Linux Kernel  American Fuzzy Lop (AFL), ClusterFuzz, OSS-Fuzz

3 Redzone-based memory error detection

• Buffer overflow (spatial memory errors) ptrX Shadow memory: a bitmap to validate all addresses objX Check before access

Shadow memory Accessible

4 Redzone-based memory error detection

• Buffer overflow (spatial memory errors) ptrX Shadow memory: a bitmap to validate all addresses objX Check before access Redzone: inaccessible region between objects Shadow memory Accessible Inaccessible (redzone) 4 Redzone-based memory error detection

• Buffer overflow (spatial memory errors) ptrX Shadow memory: a bitmap to validate all addresses objX

Error! Redzone: inaccessible region between objects Shadow memory Accessible Inaccessible (redzone) 4 Redzone-based memory error detection

• Use-after-free (temporal memory errors) ptrX

objX

Accessible Inaccessible Shadow memory 5 Redzone-based memory error detection

• Use-after-free (temporal memory errors) ptrX ptrX

objX free(ptrX) Quarantined Region is invalidated and quarantined, but not actually deallocated

Accessible Inaccessible Shadow memory 5 Redzone-based memory error detection

• Use-after-free (temporal memory errors) ptrX ptrX Hold the objX free(ptrX) Quarantined region until quarantine zone is full (FIFO)

Accessible Inaccessible Shadow memory 5 Redzone-based memory error detection

• Use-after-free (temporal memory errors) ptrX ptrX ptrY

ptrY = Quarantined malloc() TheobjX regionfree( ptrXis )actually objY deallocated, and can be allocated to a new object

Accessible Inaccessible Shadow memory 5 Limitations of redzone-based approach

1. What if a pointer accesses beyond redzone?

ptrX objX

objY

6 Limitations of redzone-based approach

1. What if a pointer accesses beyond redzone?

objX

ptrX objY

Spatial memory error 6 Limitations of redzone-based approach

1. What if a pointer 2. What if a accesses beyond accesses after another object redzone? is allocated in the region?

objX ptrX

ptrX objY objX

Spatial memory error 6 Limitations of redzone-based approach

1. What if a pointer 2. What if a dangling pointer accesses beyond accesses after another object redzone? is allocated in the region?

objX ptrX ptrX

ptrX objY objX objZ

Spatial memory error Temporal memory error 6 Limitations of redzone-based approach

1. What if a pointer 2. What if a dangling pointer accesses beyond accesses after another object redzone? is allocated in the region?

objX ptrX ptrX

ptrX objY Cannot detect!objX objZ

Spatial memory error Temporal memory error 6 Motivation P1

• To enhance detectability of redzone- obj1 based memory error detection • P1. Large gap to detect spatial memory errors P1 • P2. Large quarantine zone to detect temporal memory errors obj1

P1

7 Motivation P1

• To enhance detectability of redzone- P2 obj1 based memory error detection • P1. Large gap to detect spatial memory errors P1 • P2. Large quarantine zone to detect temporal memory errors P2 obj2obj1

P1

7 Motivation P1

• To enhance detectability of redzone- P2 obj1 based memory error detection • P1. Large gap to detect spatial memory errors P1 • P2. Large quarantine zone to detect temporal memory errors P2 obj2obj1

Huge physical memory P1 required

7 MEDS overview

• Enhances detectability of redzone-based memory error detection

• Idea: Fully utilize 64-bit virtual address space to support • P1. Large gap to detect spatial error • P2. Large quarantine zone to detect temporal error

• Approach: minimize physical memory use • Page aliasing allocator and page protection • Hierarchical memory error detection

8 Page aliasing (P1)

• Maps multiple virtual pages to single physical page Virtual obj1

A memory page obj2 Allocated

Redzone

Page aliasing

obj4 9 Page aliasing (P1)

• Maps multiple virtual pages to single physical page Virtual obj1

Physical obj1 obj2 A memory page obj2 obj3 obj4 Allocated

Redzone

Page aliasing

obj4 9 Page aliasing (P1)

• Maps multiple virtual pages to single physical page Virtual obj1 Redzone itself does not Physical occupy physical memory obj1 obj2 A memory page obj2 obj3 obj4 Allocated

Redzone

Page aliasing

obj4 9 Page protection (P1)

• Redzone only pages are unmapped

Virtual

obj1 Physical

obj1 A memory page obj2 obj3 Unmapped page obj4 obj2 Allocated

Redzone

Page aliasing 10 Page protection (P1)

• Redzone only pages are unmapped Do not occupy shadow memory and physical obj1 Physical

obj1 A memory page obj2 obj3 Unmapped page obj4 obj2 Allocated

Redzone

Page aliasing 10 Page aliasing & Page protection (P2)

Virtual obj1

Physical obj1 obj2 A memory page obj3 obj4 obj4 Unmapped page

Allocated

Redzone

Page aliasing 11 Page aliasing & Page protection (P2)

Virtual Virtual obj1

Quarantined

Physical Physical obj1 obj2 obj2 A memory page obj3 obj3 obj4 obj4 obj4 obj4 Unmapped page

Allocated

Redzone

Page aliasing 11 Page aliasing & Page protection (P2)

Virtual Virtual obj1

Quarantined

Physical Physical obj1 objX obj2 obj2 A memory page obj3 obj3 obj4 obj4 obj4 obj4 Unmapped page

Allocated objX Redzone

Page aliasing 11 Page aliasing & Page protection (P2)

Virtual Virtual Reuse physical obj1 memory immediately, Quarantined while not reusing Physical Physical virtual addresses obj1 objX obj2 obj2 A memory page obj3 obj3 obj4 obj4 obj4 obj4 Unmapped page

Allocated objX Redzone

Page aliasing 11 Hierarchical memory error detection

• Many different ways to represent redzones  Further optimizing physical memory uses ptr

12 Hierarchical memory error detection

• Many different ways to represent redzones  Further optimizing physical memory uses ptr

#1. Shadow memory is invalid

12 Hierarchical memory error detection

• Many different ways to represent redzones  Further optimizing physical memory uses ptr

#1. Shadow memory is invalid

#2. Virtual page is unmapped

12 Hierarchical memory error detection

• Many different ways to represent redzones  Further optimizing physical memory uses ptr

#1. Shadow memory is invalid

#2. Virtual page is unmapped

#3. Shadow memory is unmapped

12 Evaluation

• Configuration ASan MEDS Improv. Redzone 8-1024 bytes 4MB 16,384x Quarantine 128MB 80TB 65,536x • ASan cannot use configuration for MEDS (lack of memory) • Compatibility • Performance: 2 times slowdown • Detection (fuzz testing): 68% more detection

13 Compatibility

• Unit tests from real-world applications • Test cases in Chrome, Firefox, Nginx • All Passed

• Memory error unit tests • ASan unit tests • All Passed • NIST Juliet test suites • All Passed except random access tests  ASan: 35% vs. MEDS: 98%

14 Micro-scale performance overhead

• TLB misses • 5 times more than ASan (more virtual pages with page aliasing)

• Number of system calls • mmap(), munmap(), and mremap() • 32 times more than ASan (page aliasing and page protection)

• Memory footprint • 218% more than baseline • 68% more than ASan (much larger redzone and quarantine)

15 End-to-end performance overhead

• 108% compared to baseline, 86% to ASan 4

3

2

1 Baseline

0 Chrome Firefox Apache Nginx ASan MEDS 16 End-to-end performance overhead

• 108% compared to baseline, 86% to ASan 4

3 41% to baseline 22% to ASan 2

1 Baseline

0 Chrome Firefox Apache Nginx ASan MEDS 16 End-to-end performance overhead

• 108% compared to baseline, 86% to ASan Large number of 4 small objects on stack 41% to baseline 3 243% to baseline 22% to ASan 2 211% to ASan

1 Baseline

0 Chrome Firefox Apache Nginx ASan MEDS 16 Detection (fuzz testing)

• Run AFL (8 cores, 6 hours) • Despite the performance overhead, explore 68.3% more unique crashes than ASan 4 3.5 3 2.5 2 1.5 1 ASan 0.5 0

17 Detection (fuzz testing)

• Run AFL (8 cores, 6 hours) • Despite the performance overhead, explore 68.3% more unique crashes than ASan 4 MEDS finds more unique crashes in 3.5 3 initial phase, but saturated in the end 2.5 2 1.5 1 ASan 0.5 0

17 Detection (fuzz testing)

• Number of unique crashes with time spent (metacam)

70 Saturated 60 50 40 30 20

Found crashes Found 10 0 1 2 3 4 5 6 7 8 Time spent (hrs)

ASan MEDS

18 How MEDS explores more crashes?

• More input sets can be detected • Higher probability to detect int a[10]; • Bugs can be found earlier than ASan a[x] = x; • Fuzzer can focus on the other paths

• MEDS can detect the cases that ASan cannot detect struct A { • Always bypass redzone int num[10]; }; • e.g., Miscalculation of structure array size struct A *a = • Size of the structure is larger than redzone malloc(sizeof(struct A)); size ... • Access to certain element cannot be detected. (a+i)->num[8] = i;

19 How MEDS explores more crashes?

• More input sets can be detected • Higher probability to detect int a[10]; • Bugs can be found earlier than ASan a[x] = x; • Fuzzer can focus on the other paths

• MEDS can detect the cases that ASan cannot detect struct A { • Always bypass redzone int num[10]; }; • e.g., Miscalculation of structure array size struct A *a = • Size of the structure is larger than redzone malloc(sizeof(struct A)); size ... • Access to certain element cannot be detected. (a+i)->num[8] = i;

19 How MEDS explores more crashes?

• More input sets can be detected • Higher probability to detect int a[10]; • Bugs can be found earlier than ASan a[x] = x; • Fuzzer can focus on the other paths

• MEDS can detect the cases that ASan cannot detect struct A { • Always bypass redzone int num[10]; }; • e.g., Miscalculation of structure array size struct A *a = • Size of the structure is larger than redzone malloc(sizeof(struct A)); size ... • Access to certain element cannot be detected. (a+i)->num[8] = i;

19 How MEDS explores more crashes?

• More input sets can be detected • Higher probability to detect int a[10]; • Bugs can be found earlier than ASan a[x] = x; • Fuzzer can focus on the other paths

• MEDS can detect the cases that ASan cannot detect struct A { • Always bypass redzone int num[10]; }; • e.g., Miscalculation of structure array size struct A *a = • Size of the structure is larger than redzone malloc(sizeof(struct A)); size ... • Access to certain element cannot be detected. (a+i)->num[8] = i;

19 Conclusion

• Idea • Support large gap and large quarantine zone • Approach • Page aliasing and page protection • Hierarchical memory error detection • Despite overhead (108%), MEDS finds more crashes during fuzz testing (68.3%) • Open source – will be available soon • https://github.com/purdue-secomp-lab/MEDS • Please use to detect bugs

20 Thank you for listening!

21