Moonshine: Optimizing OS Fuzzer Seed Selection with Trace Distillation
Total Page:16
File Type:pdf, Size:1020Kb
MoonShine: Optimizing OS Fuzzer Seed Selection with Trace Distillation Shankara Pailoor, Andrew Aday, and Suman Jana Columbia University Abstract bug in system call implementations might allow an un- privileged user-level process to completely compromise OS fuzzers primarily test the system-call interface be- the system. tween the OS kernel and user-level applications for secu- OS fuzzers usually start with a set of synthetic seed rity vulnerabilities. The effectiveness of all existing evo- programs , i.e., a sequence of system calls, and itera- lutionary OS fuzzers depends heavily on the quality and tively mutate their arguments/orderings using evolution- diversity of their seed system call sequences. However, ary guidance to maximize the achieved code coverage. generating good seeds for OS fuzzing is a hard problem It is well-known that the performance of evolutionary as the behavior of each system call depends heavily on fuzzers depend critically on the quality and diversity of the OS kernel state created by the previously executed their seeds [31, 39]. Ideally, the synthetic seed programs system calls. Therefore, popular evolutionary OS fuzzers for OS fuzzers should each contain a small number of often rely on hand-coded rules for generating valid seed system calls that exercise diverse functionality in the OS sequences of system calls that can bootstrap the fuzzing kernel. process. Unfortunately, this approach severely restricts However, the behavior of each system call heavily de- the diversity of the seed system call sequences and there- pends on the shared kernel state created by the previous fore limits the effectiveness of the fuzzers. system calls, and any system call invoked by the seed In this paper, we develop MoonShine, a novel strat- programs without the correct kernel state will only trig- egy for distilling seeds for OS fuzzers from system call ger the shallow error handling code without reaching the traces of real-world programs while still preserving the core logic. Therefore, to reach deeper into a system call dependencies across the system calls. MoonShine lever- logic, the corresponding seed program must correctly set ages light-weight static analysis for efficiently detecting up the kernel state as expected by the system call. As dependencies across different system calls. user programs can only read/write kernel state through We designed and implemented MoonShine as an other system calls, essentially the seed programs must extension to Syzkaller, a state-of-the-art evolutionary identify the dependent system calls and invoke them in fuzzer for the Linux kernel. Starting from traces con- a certain system-call-specific order. For example, a seed taining 2.8 million system calls gathered from 3,220 program using the read system call must ensure that the real-world programs, MoonShine distilled down to just input file descriptor is already in an "opened" state with over 14,000 calls while preserving 86% of the original read permissions using the open system call. code coverage. Using these distilled seed system call Existing OS fuzzers [11, 37] rely on thousands of sequences, MoonShine was able to improve Syzkaller’s hand-coded rules to capture these dependencies and use achieved code coverage for the Linux kernel by 13% on them to generate synthetic seed programs. However, this average. MoonShine also found 17 new vulnerabilities approach requires significant manual work and does not in the Linux kernel that were not found by Syzkaller. scale well to achieve high code coverage. A promising alternative is to gather system call traces from diverse ex- 1 Introduction isting programs and use them to generate synthetic seed programs. This is because real programs are required to Security vulnerabilities like buffer overflow and use- satisfy these dependencies in order to function correctly. after-free inside operating system (OS) kernels are par- However, the system call traces of real programs are ticularly dangerous as they might allow an attacker to large and often repetitive, e.g., executing calls in a loop. completely compromise a target system. OS fuzzing is Therefore, they are not suitable for direct use by OS a popular technique for automatically discovering and fuzzers as they will significantly slow down the effi- fixing such critical security vulnerabilities. Most OS ciency (i.e., execution rate) of the fuzzers. The system fuzzers focus primarily on testing the system-call inter- call traces must be distilled while maintaining the correct face as it is one of the main points of interaction between dependencies between the system calls as mentioned ear- the OS kernel and user-level programs. Moreover, any lier to ensure that their achieved code coverage does not go down significantly after distillation. We call this pro- We discuss the design and implementation of MoonShine cess seed distillation for OS fuzzers. This is a hard prob- in Section 4 and present the results of our evaluation in lem as any simple strategy that selects the system calls Section 5. Finally, we describe related work in Section 8 individually without considering their dependencies is and conclude in Section 10. unlikely to improve coverage of the fuzzing process. For example, we find that randomly selecting system calls from existing program traces do not result in any cover- 2 Overview age improvement over hand-coded rules (see Section 5.4 for more details). 2.1 Problem Description In this paper, we address the aforementioned seed dis- Most existing OS fuzzers use thousands of hand-coded tillation problem by designing and implementing Moon- rules to generate seed system call sequences with valid Shine, a framework that automatically generates seed dependencies. As such an approach is fundamentally programs for OS fuzzers by collecting and distilling sys- unscalable, our goal in this paper is to design and imple- tem call traces from existing programs. It distills sys- ment a technique for automatically distilling system calls tem call traces while still maintaining the dependencies from traces of real existing programs while maintaining across the system calls to maximize coverage. Moon- the corresponding dependencies. However, system call Shine first executes a set a real-world programs and cap- traces of existing programs can be arbitrarily large and tures their system call traces along with the coverage repetitive, and as a result will significantly slow down achieved by each call. Next, it greedily selects the calls the performance of an OS fuzzer. Therefore, in this pa- that contribute the most new coverage and for each such per, we focus on distilling a small number of system calls call, identifies all its dependencies using lightweight from the traces while maintaining their dependencies and static analysis and groups them into seed programs. preserving most of the coverage achieved by the com- We demonstrate that MoonShine is able to distill a plete traces. trace consisting of a total of 2.8 million system calls Existing test case minimization strategies like afl- gathered from 3,220 real programs down to just over tmin [12] try to dynamically remove parts of an input 14,000 calls while still maintaining 86% of their origi- while ensuring that coverage does not decrease. How- nal coverage over the Linux kernel. We also demonstrate ever, such strategies do not scale well to program traces that our distilled seeds help Syzkaller, a state-of-the-art containing even a modest number of system calls due to system call fuzzer, to improve its coverage achieved for their complex dependencies. For example, consider the the Linux kernel by 13% over using manual rules for gen- left-hand trace shown in Figure 1. A dynamic test min- erating seeds. Finally, MoonShine’s approach led to the imization strategy similar to that of afl-tmin might take discovery of 17 new vulnerabilities in Linux kernel, none up to 256 iterations for finding the minimal distilled se- of which were found by Syzkaller while using its manual quence of calls. rule-based seeds. To avoid the issues described above, we use In summary, we make the following contributions: lightweight static analysis to identify the potential depen- • We introduce the concept of seed distillation, i.e., dencies between system calls and apply a greedy strategy distilling traces from real world programs while to distill the system calls (along with their dependen- maintaining both the system call dependencies and cies) that contribute significantly towards the coverage achieved code coverage as a means of improving achieved by the undistilled trace. Before describing our OS fuzzers. approach in detail, we define below two different types of dependencies that we must deal with during the distil- • We present an efficient seed distillation algorithm lation process. for OS fuzzers using lightweight static analysis. Explicit Dependencies. We define a system call ci to be explicitly dependent on another system call c if • We designed and implemented our approach as part j c produces a result that c uses as an input argument. of MoonShine and demonstrated its effectiveness by j i For example, in Figure 1, the open call in line 2 is an integrating it with Syzkaller, a state-of-the-art OS explicit dependency of the mmap call in line 3 because fuzzer. MoonShine improved Syzkaller’s test cov- open returns a file descriptor (3) that is used by mmap as erage for the Linux kernel by 13% and discovered its fourth argument. If open did not execute, then mmap 17 new previously-undisclosed vulnerabilities in the would not return successfully, which means it would take Linux kernel. a different execution path in the kernel. The rest of the paper is organized as follows. Section 2 Implicit Dependencies. A system call ci is defined to provides an overview of our techniques along with a mo- be implicitly dependent on c j if the execution of c j af- tivating example.