KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs
Cristian Cadar, Daniel Dunbar, Dawson Engler ∗ Stanford University
Abstract bolic values and replace corresponding concrete program operations with ones that manipulate symbolic values. We present a new symbolic execution tool, KLEE, ca- When program execution branches based on a symbolic pable of automatically generating tests that achieve value, the system (conceptually) follows both branches, high coverage on a diverse set of complex and on each path maintaining a set of constraints called the environmentally-intensive programs. We used KLEE to path condition which must hold on execution of that thoroughly check all 89 stand-alone programs in the path. When a path terminates or hits a bug, a test case GNU COREUTILS utility suite, which form the core can be generated by solving the current path condition user-level environment installed on millions of Unix sys- for concrete values. Assuming deterministic code, feed- tems, and arguably are the single most heavily tested set ing this concrete input to a raw, unmodified version of of open-source programs in existence. KLEE-generated the checked code will make it follow the same path and tests achieve high line coverage — on average over 90% hit the same bug. per tool (median: over 94%) — and significantly beat Results are promising. However, while researchers the coverage of the developers’ own hand-written test have shown such tools can sometimes get good cover- suites. When we did the same for 75 equivalent tools in age and find bugs on a small number of programs, it the BUSYBOX embedded system suite, results were even has been an open question whether the approach has any better, including 100% coverage on 31 of them. hope of consistently achieving high coverage on real ap- We also used KLEE as a bug finding tool, applyingit to plications. Two common concerns are (1) the exponen- 452 applications (over 430K total lines of code), where tial number of paths through code and (2) the challenges it found 56 serious bugs, including three in COREUTILS in handling code that interacts with its surrounding envi- that had been missed for over 15 years. Finally, we ronment, such as the operating system, the network, or used KLEE to cross-check purportedly identical BUSY- the user (colloquially: “the environment problem”). Nei- BOX and COREUTILS utilities, finding functional cor- ther concern has been much helped by the fact that most rectness errors and a myriad of inconsistencies. past work, including ours, has usually reported results on 1 Introduction a limited set of hand-picked benchmarks and typically has not included any coverage numbers. Many classes of errors, such as functional correctness This paper makes two contributions. First, we present bugs, are difficult to find without executing a piece of a new symbolic execution tool, KLEE, which we de- code. The importance of such testing — combined with signed for robust, deep checking of a broad range of ap- the difficulty and poor performance of random and man- plications, leveraging several years of lessons from our ual approaches — has led to much recent work in us- previous tool, EXE [16]. KLEE employs a variety of con- ing symbolic execution to automatically generate test in- straint solving optimizations, represents program states puts [11,14–16,20–22,24,26,27,36]. At a high-level, compactly, and uses search heuristics to get high code these tools use variations on the following idea: Instead coverage. Additionally, it uses a simple and straight- of running code on manually- or randomly-constructed forward approach to dealing with the external environ- input, they run it on symbolic input initially allowed to ment. These features improve KLEE’s performance by be “anything.” They substitute program inputs with sym- over an order of magnitude and let it check a broad range of system-intensive programs “out of the box.” ∗Author names are in alphabetical order. Daniel Dunbar is the main author of the KLEE system. Second, we show that KLEE’s automatically-generated tests hit over 90% of the lines in each tool (median: tests get high coverage on a diverse set of real, com- over 94%), achieving perfect 100% coverage in 16 plicated, and environmentally-intensive programs. Our COREUTILS tools and 31 BUSYBOX tools. most in-depth evaluation applies KLEE to all 89 pro- 2 KLEE can get significantly more code coverage than grams 1 in the latest stable version of GNU COREUTILS a concentrated, sustained manual effort. The roughly (version 6.10), which contains roughly 80,000 lines of 89-hourrun used to generate COREUTILS line cover- library code and 61,000 lines in the actual utilities [2]. age beat the developers’ own test suite — built incre- These programs interact extensively with their environ- mentally over fifteen years — by 16.8%! ment to provide a variety of functions, including man- 3 With one exception, KLEE achieved these high- aging the file system (e.g., ls, dd, chmod), display- coverage results on unaltered applications. The sole ing and configuring system properties (e.g., logname, exception, sort in COREUTILS, required a single printenv, hostname), controlling command invo- edit to shrink a large buffer that caused problems for cation (e.g., nohup, nice, env), processing text files the constraint solver. (e.g., sort, od, patch), and so on. They form the 4 KLEE finds important errors in heavily-tested code. It core user-level environment installed on many Unix sys- found ten fatal errors in COREUTILS (including three tems. They are used daily by millions of people, bug that had escaped detection for 15 years), which ac- fixes are handled promptly, and new releases are pushed count for more crashing bugs than were reported in regularly. Moreover, their extensive interaction with the 2006, 2007 and 2008 combined. It further found 24 environment stress-tests symbolic execution where it has bugs in BUSYBOX, 21bugsin MINIX, and a security historically been weakest. vulnerability in HISTAR– a total of 56 serious bugs. Further, finding bugs in COREUTILS is hard. They are 5 The fact that KLEE test cases can be run on the raw arguably the single most well-tested suite of open-source version of the code (e.g., compiled with gcc) greatly applications available (e.g., is there a program the reader simplifies debugging and error reporting. For exam- has used more under Unix than “ls”?). In 1995, ran- ple, all COREUTILS bugs were confirmed and fixed dom testing of a subset of COREUTILS utilities found within two days and versions of the tests KLEE gen- markedly fewer failures as compared to seven commer- erated were included in the standard regression suite. cial Unix systems [35]. The last COREUTILS vulnerabil- 6 KLEE is not limited to low-level programming er- ity reported on the SecurityFocus or US National Vulner- rors: when used to crosscheck purportedly identical ability databases was three years ago [5,7]. BUSYBOX and GNU COREUTILS tools, it automat- In addition, we checked two other UNIX utility suites: ically found functional correctness errors and a myr- BUSYBOX, a widely-used distribution for embedded sys- iad of inconsistencies. tems [1], and the latest release for MINIX [4]. Finally, we 7 KLEE can also be applied to non-application code. checked the HISTAR operating system kernel as a con- When applied to the core of the HISTAR kernel, it trast to application code [39]. achieved an average line coverage of 76.4% (with Our experiments fall into three categories: (1) those disk) and 67.1% (without disk) and found a serious where we do intensiveruns to both find bugs and get high security bug. coverage (COREUTILS,HISTAR, and75 BUSYBOX util- The next section gives an overview of our approach. ities), (2) those where we quickly run over many appli- Section 3 describes KLEE, focusing on its key optimiza- cations to find bugs (an additional 207 BUSYBOX util- tions. Section 4 discusses how to model the environment. ities and 77 MINIX utilities), and (3) those where we The heart of the paper is Section 5, which presents our crosscheck equivalent programs to find deeper correct- experimental results. Finally, Section 6 describes related ness bugs (67 BUSYBOX utilities vs. the equivalent 67 in work and Section 7 concludes. COREUTILS). In total, we ran KLEE on morethan 452 programs,con- 2 Overview taining over 430K total lines of code. To the best of our This section explains how KLEE works by walking the knowledge, this represents an order of magnitude more reader through the testing of MINIX’s tr tool. Despite code and distinct programs than checked by prior sym- its small size — 169 lines, 83 of which are executable — bolic test generation work. Our experiments show: it illustrates two problems common to the programs we 1 KLEE gets high coverage on a broad set of complex check: programs. Its automatically generated tests covered 1 Complexity. The code aims to translate and delete 84.5% of the total lines in COREUTILS and 90.5% in characters from its input. It hides this intent well be- BUSYBOX (ignoring library code). On average these neath non-obvious input parsing code, tricky bound- 1We ignored utilities that are simply wrapper calls to others, such ary conditions, and hard-to-follow control flow. Fig- as arch (“uname -m”) and vdir (“ls -l -b”). ure 1 gives a representative snippet.
2 2 Environmental Dependencies. Most of the code is 1 : void expand(char *arg, unsigned char *buffer) { 8 controlled by values derived from environmental in- 2 : int i, ac; 9 put. Command line arguments determine what pro- 3 : while (*arg) { 10* 4 : if (*arg == '\\') { 11* cedures execute, input values determine which way 5 : arg++; if-statements trigger, and the program depends on the 6 : i = ac = 0; ability to read from the file system. Since inputs can 7 : if (*arg >= '0' && *arg <= '7') { be invalid (or even malicious), the code must handle 8 : do { 9 : ac = (ac << 3) + *arg++ − '0'; these cases gracefully. It is not trivial to test all im- 10: i++; portant values and boundary cases. 11: } while (i<4 && *arg>='0' && *arg<='7'); The code illustrates two additional common features. 12: *buffer++ = ac; 13: } else if (*arg != '\0') KLEE First, it has bugs, which finds and generates test 14: *buffer++ = *arg++; cases for. Second, KLEE quickly achieves good code 15: } else if (*arg == '[') { 12* coverage: in two minutes it generates 37 tests that cover 16: arg++; 13 all executable statements. 2 17: i = *arg++; 14 18: if (*arg++ != '-') { 15! KLEE has two goals: (1) hit every line of executable 19: *buffer++ = '['; code in the program and (2) detect at each dangerous op- 20: arg −= 2; eration (e.g., dereference, assertion) if any input value 21: continue; 22: } exists that could cause an error. KLEE does so by running 23: ac = *arg++; programs symbolically: unlike normal execution, where 24: while (i <= ac) *buffer++ = i++; operations produce concrete values from their operands, 25: arg++; /* Skip ’]’ */ here they generate constraints that exactly describe the 26: } else 27: *buffer++ = *arg++; set of values possible on a given path. When KLEE de- 28: } tects an error or when a path reaches an exit call, KLEE 29: } solves the current path’s constraints (called its path con- 30: ... dition) to produce a test case that will follow the same 31: int main(int argc, char* argv[ ]) { 1 32: int index = 1; 2 path when rerun on an unmodifiedversion of the checked 33: if (argc > 1 && argv[index][0] == '-') { 3* program (e.g, compiled with gcc). 34: ... 4 KLEE is designed so that the paths followed by the 35: } 5 unmodified program will always follow the same path 36: ... 6 37: expand(argv[index++], index); 7 KLEE took (i.e., there are no false positives). However, 38: ... non-determinism in checked code and bugs in KLEE or 39: } its models have produced false positives in practice. The ability to rerun tests outside of KLEE, in conjunction with Figure 1: Code snippet from MINIX’s tr, representative standard tools such as gdb and gcov is invaluable for of the programs checked in this paper: tricky, non-obvious, diagnosing such errors and for validating our results. difficult to verify by inspection or testing. The order of the We next show how to use KLEE, then give an overview statements on the path to the error at line 18 are numbered on of how it works. the right hand side. 2.1 Usage
A user can start checking many real programs with KLEE The first option, --max-time, tells KLEE to check in seconds: KLEE typically requires no source modifi- tr.bc for at most two minutes. The rest describe the cations or manual work. Users first compile their code symbolic inputs. The option --sym-args 1 10 10 to bytecode using the publicly-available LLVM com- says to use zero to three command line arguments, the 3 piler [33] for GNU C. We compiled tr using: first one character long, the others 10 characters long. The option --sym-files 2 2000 says to use stan- llvm-gcc --emit-llvm -c tr.c -o tr.bc dard input and one file, each holding 2000 bytes of sym- bolic data. The option --max-fail 1 says to fail at Users then run KLEE on the generated bytecode, option- most one system call along each programpath (see § 4.2). ally stating the number, size, and type of symbolic inputs to test the code on. For tr we used the command: 2.2 Symbolic execution with KLEE klee --max-time 2 --sym-args 1 10 10 When KLEE runs on tr, it finds a buffer overflow error --sym-files 2 2000 --max-fail 1 tr.bc at line 18 in Figure 1 and then produces a concrete test
2The program has one line of dead code, an unreachable return 3Since strings in C are zero terminated, this essentially generates statement, which, reassuringly, KLEE cannot run. arguments of up to that size.
3 case (tr [ "" "") that hits it. Assuming the options 3.1 Basic Architecture of the previous subsection, KLEE runs tr as follows: At any one time, KLEE may be executing a large number 1 KLEE constructs symbolic command line string argu- of states. The core of KLEE is an interpreter loop which ments whose contents have no constraints other than selects a state to run and then symbolically executes a zero-termination. It then constrains the number of ar- single instruction in the context of that state. This loop guments to be between 1 and 3, and their sizes to be continues until there are no states remaining, or a user- 1, 10 and 10 respectively. It then calls main with defined timeout is reached. these initial path constraints. Unlike a normal process, storage locations for a state 2 When KLEE hits the branch argc > 1 at line 33, — registers, stack and heap objects — refer to expres- it uses its constraint solver STP [23] to see which di- sions (trees) instead of raw data values. The leaves of rections can execute given the current path condition. an expression are symbolic variables or constants, and For this branch, both directions are possible; KLEE the interior nodes come from LLVM assembly language forks execution and follows both paths, adding the operations (e.g., arithmetic operations, bitwise manipu- constraint argc > 1 on the false path and argc ≤ 1 lation, comparisons, and memory accesses). Storage lo- on the true path. cations which hold a constant expression are said to be 3 Given more than one active path, KLEE must pick concrete. which one to execute first. We describe its algorithm Symbolic execution of the majority of instructions is in Section 3.4. For now assume it follows the path straightforward. For example, to symbolically execute that reachesthe bug. As it doesso, KLEE adds further an LLVM add instruction: constraints to the contents of arg, and forks for a %dst = add i32 %src0, %src1 total of five times (lines denoted with a “*”): twice online 33, and thenon lines3, 4, and 15in expand. KLEE retrieves the addends from the %src0 and %src1 4 At each dangerous operation (e.g., pointer derefer- registers and writes a new expression Add(%src0, ence), KLEE checks if any possible value allowed by %src1) to the %dst register. For efficiency, the code the current path condition would cause an error. On that builds expressions checks if all given operands are the annotated path, KLEE detects no errors before line concrete (i.e., constants) and, if so, performs the opera- 18. At that point, however, it determines that input tion natively, returning a constant expression. values exist that allow the read of arg to go out of Conditional branches take a boolean expression bounds: after taking the true branch at line 15, the (branch condition) and alter the instruction pointer of code increments arg twice without checking if the the state based on whether the condition is true or false. string has ended. If it has, this increment skips the KLEE queries the constraint solver to determine if the terminating '\0' and points to invalid memory. branch condition is either provably true or provably false 5 KLEE generates concrete values for argc and argv along the current path; if so, the instruction pointer is (i.e., tr [ "" "") that when rerun on a raw ver- updated to the appropriate location. Otherwise, both sion of tr will hit this bug. It then continues follow- branches are possible: KLEE clones the state so that it ing the current path, adding the constraint that the can explore both paths, updating the instruction pointer error does not occur (in order to find other errors). and path condition on each path appropriately. Potentially dangerous operations implicitly generate 3 The KLEE Architecture branches that check if any input value exists that could KLEE is a complete redesign of our previous system cause an error. For example, a division instruction gen- EXE [16]. At a high level, KLEE functions as a hybrid erates a branch that checks for a zero divisor. Such between an operating system for symbolic processes and branches work identically to normal branches. Thus, an interpreter. Each symbolic process has a register file, even when the check succeeds (i.e., an error is detected), stack, heap, program counter, and path condition. To execution continues on the false path, which adds the avoid confusion with a Unix process, we refer to KLEE’s negation of the check as a constraint (e.g., making the representation of a symbolic process as a state. Programs divisor not zero). If an error is detected, KLEE generates are compiled to the LLVM [33] assembly language, a a test case to trigger the error and terminates the state. RISC-like virtual instruction set. KLEE directly inter- As with other dangerous operations, load and store in- prets this instruction set, and maps instructions to con- structions generate checks: in this case to check that the straints without approximation (i.e. bit-level accuracy). 4 address is in-bounds of a valid memory object. However, load and store operations present an additional compli- cation. The most straightforward representation of the 4 KLEE does not currently support: symbolic floating point, longjmp, threads, and assembly code. Additionally, memory objects memory used by checked code would be a flat byte ar- are required to have concrete sizes. ray. In this case, loads and stores would simply map to
4 array read and write expressions respectively. Unfortu- Thus, we spent a lot of effort on tricks to simplify ex- nately, our constraint solver STP would almost never be pressions and ideally eliminate queries (no query is the able to solve the resultant constraints (and neither would fastest query) before they reach STP. Simplified queries the other constraint solvers we know of). Thus, as in make solving faster, reduce memory consumption, and EXE, KLEE maps every memory object in the checked increase the query cache’s hit rate (see below). The main code to a distinct STP array (in a sense, mapping a flat query optimizations are: address space to a segmented one). This representation Expression Rewriting. The most basic optimizations dramatically improves performance since it lets STP ig- mirror those in a compiler: e.g., simple arithmetic sim- n nore all arrays not referenced by a given expression. plifications (x+0=0), strength reduction (x * 2 Many operations (such as bound checks or object-level = x << n), linear simplification (2*x-x=x). copy-on-write) require object-specific information. If a Constraint Set Simplification. Symbolic execution pointer can refer to many objects, these operations be- typically involves the addition of a large number of con- come difficult to perform. For simplicity, KLEE sidesteps straints to the path condition. The natural structure of this problem as follows. When a dereferenced pointer p programs means that constraints on same variables tend can refer to N objects, KLEE clones the current state N to become more specific. For example, commonly an in- times. In each state it constrains p to be within bounds exact constraint such as x < 10 gets added, followed of its respective object and then performs the appropri- some time later by the constraint x = 5. KLEE actively ate read or write operation. Although this method can simplifies the constraint set by rewriting previous con- be expensive for pointers with large points-to sets, most straints when new equality constraints are added to the programs we have tested only use symbolic pointers that constraint set. In this example, substituting the value for refer to a single object, and KLEE is well-optimized for x into the first constraint simplifies it to true, which this case. KLEE eliminates. Implied Value Concretization. When a constraint such 3.2 Compact State Representation as x +1 = 10 is added to the path condition, then the The number of states grows quite quickly in practice: value of x has effectively become concrete along that often even small programs generate tens or even hun- path. KLEE determines this fact (in this case that x =9) dreds of thousands of concurrent states during the first and writes the concrete value back to memory. This en- few minutes of interpretation. When we ran COREUTILS sures that subsequent accesses of that memory location with a 1GB memory cap, the maximum number of con- can return a cheap constant expression. current states recorded was 95,982 (for hostid), and Constraint Independence. Many constraints do not the average of this maximum for each tool was 51,385. overlap in terms of the memory they reference. Con- This explosion makes state size critical. straint independence (taken from EXE) divides con- Since KLEE tracks all memory objects, it can imple- straint sets into disjoint independent subsets based on the ment copy-on-write at the object level (rather than page symbolic variables they reference. By explicitly track- granularity), dramatically reducing per-state memory re- ing these subsets, KLEE can frequently eliminate irrel- quirements. By implementing the heap as an immutable evant constraints prior to sending a query to the con- map, portions of the heap structure itself can also be straint solver. For example, given the constraint set shared amongst multiple states (similar to sharing por- {i < j,j < 20,k > 0}, a query of whether i = 20 tions of page tables across fork()). Additionally, this just requires the first two constraints. heap structure can be cloned in constant time, which is Counter-example Cache. Redundant queries are fre- important given the frequency of this operation. quent, and a simple cache is effective at eliminating a This approach is in marked contrast to EXE, which large number of them. However, it is possible to build used one native OS process per state. Internalizing the a more sophisticated cache due to the particular struc- state representation dramatically increased the number ture of constraint sets. The counter-example cache maps of states which can be concurrently explored, both by sets of constraints to counter-examples (i.e., variable as- decreasing the per-state cost and allowing states to share signments), along with a special sentinel used when a set memory at the object (rather than page) level. Addition- of constraints has no solution. This mapping is stored ally, this greatly simplified the implementation of caches in a custom data structure — derived from the UBTree and search heuristics which operate across all states. structure of Hoffmann and Hoehler [28] — which al- lows efficient searching for cache entries for both sub- 3.3 Query Optimization sets and supersets of a constraint set. By storing the Almost always, the cost of constraint solving dominates cache in this fashion, the counter-example cache gains everything else — unsurprising, given that KLEE gen- three additional ways to eliminate queries. In the ex- erates complicated queries for an NP-complete logic. ample below, we assume that the counter-example cache
5 Optimizations Queries Time (s) STP Time (s) 400 None None 13717 300 281 Cex. Cache Independence 13717 166 148 300 Independence All Cex. Cache 8174 177 156 All 699 20 10 200
Table 1: Performance comparison of KLEE’s solver optimiza- Average Time (s) tions on COREUTILS. Each tool is run for 5 minutes without 100 optimization, and rerun on the same workload with the given optimizations. The results are averaged across all applications. 0 0 0.2 0.4 0.6 0.8 1 Num. Instructions (normalized) currently has entries for {i < 10,i = 10} (no solution) and {i < 10, j = 8} (satisfiable, with variable assign- Figure 2: The effect of KLEE’s solver optimizations over ments i → 5, j → 8). time, showing they become more effective over time, as the caches fill and queries become more complicated. The num- 1 When a subset of a constraint set has no solution, ber of executed instructions is normalized so that data can be then neither does the original constraint set. Adding aggregated across all applications. constraints to an unsatisfiable constraint set cannot make it satisfiable. For example, given the cache above, {i < 10,i = 10, j = 12} is quickly deter- erage number of STP queries are reduced to 5% of the mined to be unsatisfiable. original number and the average runtime decreases by 2 When a superset of a constraint set has a solution, more than an order of magnitude. that solution also satisfies the original constraint set. It is also worth noting the degree to which STP time Dropping constraints from a constraint set does not (time spent solving queries) dominates runtime. For the invalidate a solution to that set. The assignment original runs, STP accounts for 92% of overall execution i → 5, j → 8, for example, satisfies either i< 10 time on average (the combined optimizations reduce this or j =8 individually. by almost 300%). With both optimizations enabled this 3 When a subset of a constraint set has a solution, it is percentage drops to 41%, which is typical for applica- likely that this is also a solution for the original set. tions we have tested. Finally, Figure 2 shows the efficacy This is because the extra constraints often do not in- of KLEE’s optimizations increases with time — as the validate the solution to the subset. Because checking counter-example cache is filled and query sizes increase, a potential solution is cheap, KLEE tries substituting the speed-up from the optimizations also increases. in all solutions for subsets of the constraint set and returns a satisfying solution, if found. For example, 3.4 State scheduling the constraint set {i < 10, j = 8,i 6= 3} can still be KLEE selects the state to run at each instruction by inter- satisfied by i → 5, j → 8. leaving the following two search heuristics. To demonstrate the effectiveness of these optimiza- Random Path Selection maintains a binary tree record- tions, we performed an experiment where COREUTILS ing the program path followed for all active states, i.e. the applications were run for 5 minutes with both of these leaves of the tree are the current states and the internal optimizations turned off. We then deterministically reran nodes are places where execution forked. States are se- the exact same workload with constraint independence lected by traversing this tree from the root and randomly and the counter-example cache enabled separately and selecting the path to follow at branch points. Therefore, together for the same number of instructions. This exper- when a branch point is reached, the set of states in each iment was done on a large sample of COREUTILS utili- subtree has equal probability of being selected, regard- ties. The results in Table 1 show the averaged results. less of the size of their subtrees. This strategy has two As expected, the independence optimization by itself important properties. First, it favors states high in the does not eliminate any queries, but the simplifications it branch tree. These states have less constraints on their performs reduce the overall running time by almost half symbolic inputs and so have greater freedom to reach un- (45%). The counter-example cache reduces both the run- covered code. Second, and most importantly, this strat- ning time and the number of STP queries by 40%. How- egy avoids starvation when some part of the program is ever, the real win comes when both optimizations are en- rapidly creating new states (“fork bombing”) as it hap- abled; in this case the hit rate for the counter-example pens when a tight loop contains a symbolic condition. cache greatly increase due to the queries first being sim- Note that the simply selecting a state at random has nei- plified via independence. For the sample runs, the av- ther property.
6 Coverage-Optimized Search tries to select states likely 1 : ssize t read(int fd, void *buf, size t count) { to covernew code in the immediate future. It uses heuris- 2 : if (is invalid(fd)) { tics to compute a weight for each state and then ran- 3 : errno = EBADF; domly selects a state according to these weights. Cur- 4 : return −1; 5 : } rently these heuristics take into account the minimum 6 : struct klee fd *f = &fds[fd]; distance to an uncovered instruction, the call stack of the 7 : if (is concrete file(f)) { state, and whether the state recently covered new code. 8 : int r = pread(f−>real fd, buf, count, f−>off); 9 : if (r != −1) KLEE uses each strategy in a round robin fashion. 10: f−>off += r; While this can increase the time for a particularly effec- 11: return r; tive strategy to achieve high coverage, it protects against 12: } else { cases where an individual strategy gets stuck. Further- 13: /* sym files are fixed size: don’t read beyond the end. */ 14: if (f−>off >= f−>size) more, since strategies pick from the same state pool, in- 15: return 0; terleaving them can improve overall effectiveness. 16: count = min(count, f−>size − f−>off); The time to execute an individual instruction can vary 17: memcpy(buf, f−>file data + f−>off, count); 18: f−>off += count; widely between simple instructions (e.g., addition) and 19: return count; instructions which may use the constraint solver or fork 20: } execution (branches, memory accesses). KLEE ensures 21: } that a state which frequently executes expensive instruc- tions will not dominate execution time by running each Figure 3: Sketch of KLEE’s model for read(). state for a “time slice” defined by both a maximum num- for each whether the associated file is symbolic or con- ber of instructions and a maximum amount of time. crete. If fd refers to a concrete file, we use the operating system to read its contents by calling pread() (lines 4 Environment Modeling 7-11). We use pread to multiplex access from KLEE’s When code reads values from its environment — many states onto the one actual underlying file descrip- command-line arguments, environment variables, file tor. 5 If fd refers to a symbolic file, read() copies from data and metadata, network packets, etc — we conceptu- the underlying symbolic buffer holding the file contents ally want to return all values that the read could legally into the user supplied buffer (lines 13-19). This ensures produce, rather than just a single concrete value. When it that multiple read() that access the same file use con- writes to its environment, the effects of these alterations sistent symbolic values. should be reflected in subsequent reads. The combina- Our symbolic file system is crude, containing only a tion of these features allows the checked program to ex- single directory with N symbolic files in it. KLEE users plore all potential actions and still have no false positives. specify both the number N and the size of these files. Mechanically, we handle the environment by redirect- This symbolic file system coexists with the real file sys- ing library calls that access it to models that understand tem, so applications can use both symbolic and concrete the semantics of the desired action well enough to gen- files. When the program calls open with a concrete erate the required constraints. Crucially, these models name, we (attempt to) open the actual file. Thus, the call: are written in normal C code which the user can read- int fd = open("/etc/fstab", O_RDNLY); ily customize, extend, or even replace without having to understand the internals of KLEE. We have about 2,500 sets fd to point to the actual configuration file lines of code to define simple models for roughly 40 sys- /etc/fstab. tem calls (e.g., open, read, write, stat, lseek, On the other hand, calling open() with an uncon- ftruncate, ioctl). strained symbolic name matches each of the N symbolic 4.1 Example: modeling the file system files in turn, and will also fail once. For example, given N =1, calling open() with a symbolic command-line For each file system operation we check if the action is argument argv[1]: for an actual concrete file on disk or a symbolic file. For concrete files, we simply invoke the corresponding sys- int fd = open(argv[1], O_RDNLY); tem call in the running operating system. For symbolic files we emulate the operation’s effect on a simple sym- will result in two paths: one in which fd points to the bolic file system, private to each state. single symbolic file in the environment, and one in which fd -1 Figure 3 gives a rough sketch of the model for is set to indicating an error. read(), eliding details for dealing with linking, reads 5Since KLEE’s states execute within a single Unix process (the one on standard input, and failures. The code maintains a set used to run KLEE), then unless we duplicated file descriptors for each of file descriptors, created at file open(), and records (which seemed expensive), a read by one would affect all the others.
7 60 Unsurprisingly, the choice of what interface to model 52 has a big impact on model complexity. Rather than hav- 50 ing our models at the system call level, we could have in-
stead built them at the C standard library level (fopen, 40 fread, etc.). Doing so has the potential performance advantage that, for concrete code, we could run these op- 30 erations natively. The major downside, however, is that 20 the standard library contains a huge number of functions 16 — writing models for each would be tedious and error- 10 prone. By only modeling the much simpler, low-level 5 6 4 3 system call API, we can get the richer functionality by 1 2 0 just compiling one of the many implementations of the C standard library (we use uClibc [6]) and let it worry 2000-3000 3000-4000 4000-5000 5000-6000 6000-7000 7000-8000 8000-9000 9000-10000 about correctness. As a side-effect, we simultaneously Executable Lines of Code check the library for errors as well. 4.2 Failing system calls Figure 4: Histogram showing the number of COREUTILS tools that have a given number of executable lines of code The real environment can fail in unexpected ways (e.g., (ELOC). write() fails because of a full disk). Such failures can often lead to unexpected and hard to diagnose bugs. Even when applications do try to handle them, this code as reported by gcov, because it is widely-understood is rarely fully exercised by the regression suite. To help and uncontroversial. Of course, it grossly underestimates catch such errors, KLEE will optionally simulate envi- KLEE’s thoroughness, since it ignores the fact that KLEE ronmental failures by failing system calls in a controlled explores many different unique paths with all possible manner (similar to [38]). We made this mode optional values. We expect a path-based metric would show even since not all applications care about failures — a simple more dramatic wins. application may ignore disk crashes, while a mail server We measure coverage by running KLEE-generated test expends a lot of code to handle them. cases on a stand-alone version of each utility and using 4.3 Rerunning test cases gcov to measure coverage. Running tests independently of KLEE eliminates the effect of bugs in KLEE and veri- KLEE-generated test cases are rerun on the unmodified fies that the produced test case runs the code it claims. native binaries by supplying them to a replay driver we Note, our coverage results only consider code in the provide. The individual test cases describe an instance tool itself. They do not count library code since doing so of the symbolic environment. The driver uses this de- makes the results harder to interpret: scription to create actual operating system objects (files, 1 It double-counts many lines, since often the same li- pipes, ttys, directories, links, etc.) containing the con- brary function is called by many applications. crete values used in the test case. It then executes the un- 2 It unfairly under-counts coverage. Often, the bulk of modified program using the concrete command-line ar- a library function called by an application is effec- guments from the test case. Our biggest challenge was tively dead code since the library code is general but making system calls fail outside of KLEE — we built a call sites are not. For example, printf is excep- simple utility that uses the ptrace debugging interface tionally complex, but the call printf("hello") to skip the system calls that were supposed to fail and can only hit a small a fraction (missing the code to instead return an error. print integers, floating point, formatting, etc.). 5 Evaluation However, we do include library code when measuring This section describes our in-depth coverage experi- the raw size of the application: KLEE must successfully ments for COREUTILS (§ 5.2) and BUSYBOX (§ 5.3) handle this library code (and gets no credit for doing so) as well as errors found during quick bug-finding runs in order to exercise the code in the tool itself. We mea- (§ 5.4). We use KLEE to find deep correctness errors by sure size in terms of executable lines of code (ELOC) cross-checking purportedly equivalent tool implementa- by counting the total number of executable lines in the tions (§ 5.5) and close with results for HISTAR (§5.6). final executable after global optimization, which elimi- nates uncalled functions and other dead code. This mea- 5.1 Coverage methodology sure is usually a factor of three smaller than a simple line We use line coverage as a conservative measure of KLEE- count. produced test case effectiveness. We chose line coverage In our experiments KLEE minimizes the test cases it
8 COREUTILS BUSYBOX 100% Base + Fail Coverage KLEE Devel. KLEE Devel. Base (w/o lib) tests tests tests tests 80% 100% 16 1 31 4 90-100% 40 6 24 3 60% 80-90% 21 20 10 15 70-80% 7 23 5 6 40% 60-70% 5 15 2 7 50-60% - 10 - 4 Coverage (ELOC %) 20% 40-50% - 6 - - 0% 30-40% - 3 - 2 1 25 50 75 20-30% - 1 - 1 10-20% - 3 - - Figure 5: Line coverage for each application with and without 0-10% - 1 - 30 failing system calls. Overall cov. 84.5% 67.7% 90.5% 44.8% Med cov/App 94.7% 72.5% 97.5% 58.9% Ave cov/App 90.9% 68.4% 93.5% 43.7% size of arguments and files. We found this easy to do, so presumably a tool implementer or user would as well. Table 2: Number of COREUTILS tools which achieve line After these runs completed, we improved them by failing coverage in the given ranges for KLEE and developers’ tests system calls (see § 4.2). (library code not included). The last rows shows the aggre- 5.2.1 Line coverage results gate coverage achieved by each method and the average and median coverage per application. The first two columns in Table 2 give aggregate line coverage results. On average our tests cover 90.9% of the lines in each tool (median: 94.7%), with an overall generates by only emitting tests cases for paths that hit a (aggregate) coverage across all tools of 84.5%. We get new statement or branch in the main utility code. A user 100% line coverage on 16 tools, over 90% on 56 tools, that wants high library coverage can change this setting. and over 80% on 77 tools (86.5% of all tools). The min- 5.2 GNUCOREUTILS imum coverage achieved on any tool is 62.6%. We believe such high coverage on a broad swath of ap- We now give KLEE coverage results for all 89 GNU plications “out of the box” convincinglyshows the power COREUTILS utilities. of the approach, especially since it is across the entire Figure 4 breaks down the tools by executable lines tool suite rather than focusing on a few particular appli- of code (ELOC), including library code the tool calls. cations. While relatively small, the tools are not toys — includ- Importantly, KLEE generates high coverage with few ing library code called, the smallest five have between test cases: for our non-failing runs, it needs a total of 2K and 3K ELOC, over half (52) have between 3K and 3,321 tests, with a per-tool average of 37 (median: 33). 4K, and ten have over 6K. The maximum numberneeded was 129 (for the “[” tool) Previous work, ours included, has evaluated and six needed 5. As a crude measure of path complexity, constraint-based execution on a small number of we countedthe numberof static branchesrun by each test hand-selected benchmarks. Reporting results for the case using gcov6 (i.e., an executed branch counts once entire COREUTILS suite, the worst along with the best, no matter how many times the branch ran dynamically). prevents us from hand-picking results or unintentionally The average path length was 76 (median: 53), the maxi- cheating through the use of fragile optimizations. mum was 512 and (to pick a random number) 160 were Almost all tools were tested using the same command at least 250 branches long. (command arguments explained in § 2.1): Figure 5 shows the coverage KLEE achieved on each ./run
9 100% paste -d\\ abcdefghijklmnopqrstuvwxyz pr -e t2.txt tac -r t3.txt t3.txt 50% mkdir -Z a b mkfifo -Z a b 0% mknod -Z a b p md5sum -c t1.txt ptx -F abcdefghijklmnopqrstuvwxyz
vs. Manual (ELOC %) \\ −50% ptx x t4.txt
klee seq -f %0 1 −100% 1 10 25 50 75 t1.txt: "\t \tMD5(" t2.txt: "\b\b\b\b\b\b\b\t" Figure 6: Relative coverage difference between KLEE and t3.txt: "\n" the COREUTILS manual test suite, computed by subtracting t4.txt: "a" the executable lines of code covered by manual tests (Lman) from KLEE tests (Lklee) and dividing by the total possible: Figure 7: KLEE-generated command lines and inputs (modi- (Lklee − Lman)/Ltotal. Higher bars are better for KLEE, fied for readability) that cause program crashes in C OREUTILS which beats manual testing on all but 9 applications, often version 6.10 when run on Fedora Core 7 with SELinux on a significantly. Pentium machine.
is a more modest 13.1% extra coverage on the df tool. relative improvement over the developers’ tests remains: KLEE achieves 76.9% overall branch coverage, while the 5.2.2 Comparison against developer test suites developers’ tests get only 56.5%. Each utility in COREUTILS comes with an extensive Finally, it is important to note that although KLEE’s manually-written test suite extended each time a new bug runs significantly beat the developers’ tests in terms of fix or extra feature is added. 7 As Table 2 shows, KLEE coverage, KLEE only checks for low-level errors and vi- beats developer tests handily on all aggregate measures: olations of user-level asserts. In contrast, developer tests overall total line coverage (84.5% versus 67.7%), aver- typically validate that the application output matches the age coverage per tool (90.9% versus 68.4%) and median expected one. We partially address this limitation by val- coverage per tool (94.7% versus 72.5%). At a more de- idating the output of these utilities against the output pro- tailed level, KLEE gets 100% coverage on 16 tools and duces by a different implementation (see § 5.5). over 90% coverage on 56 while the developer tests get 5.2.3 Bugs found 100% on a single utility (true) and reach over 90% on only 7. Finally, the developers tests get below 60% cov- KLEE found ten unique bugs in COREUTILS (usually erage on 24 tools while KLEE always achieves over 60%. memory error crashes). Figure 7 gives the command In total, an 89 hour run of KLEE (about one hour per ap- lines used to trigger them. The first three errors ex- plication) exceeds the coverage of a test suite built over isted since at least 1992, so should theoretically crash any a period of fifteen years by 16.8%! COREUTILS distribution up to 6.10. The others are more Figure 6 gives a relative view of KLEE versus devel- recent, and do not crash older COREUTILS distributions. oper tests by subtracting the lines hit by manual testing While one bug (in seq) had been fixed in the develop- from those hit by KLEE and dividing this by the total pos- ers’ unreleased version, the other bugs were confirmed sible. A bar above zero indicates that KLEE beat the man- and fixed within two days of our report. In addition, ver- ual test (and by how much); a bar below shows the oppo- sions of the KLEE-generated test cases for the new bugs site. KLEE beats manual testing, often significantly, on were added to the official COREUTILS test suite. the vast majority of the applications. As an illustrative example, we discuss the bug in pr To guard against hidden bias in line coverage, we (used to paginate files before printing) hit by the invoca- also compared the taken branch coverage (as reported by tion “pr -e t2.txt” in Figure 7. The code contain- gcov) of the manual and KLEE test suites. While the ing the bug is shown in Figure 8. On the path that hits absolute coverage for both test suites decreases, KLEE’s the bug, both chars per input tab and chars per c equal tab width (let’s call it T ). Line 2665 computes 7We ran the test suite using the commands: env RUN EXPENSIVE width = (T − input position mod T ) using the TESTS=YES RUN VERY EXPENSIVE TESTS=YES make macro on line 602. The root cause of the bug is the in- check and make check-root (as root). A small number of tests correct assumption that 0 ≤ x mod y 10 Random Devel klee 602: #define TAB WIDTH(c , h ) ((c ) − ((h ) % (c ))) 100 . . . 1322: clump buff = xmalloc(MAX(8,chars per input tab)); . . . // (set s to clump buff) 80 2665: width = TAB WIDTH(chars per c, input position); 2666: 2667: if (untabify input) 60 2668: { 2669: for (i = width; i; −−i) 40 2670: *s++ = ' '; 2671: chars = width; 2672: } 20 Figure 8: Code snippet from pr where a memory 0 df overflow of clump buff via pointer s is possible if seq tee stat fmt join expr sum pinky csplit tsort chars per input tab == chars per c and nohup ginstall base64 input position < 0. unexpand Figure 9: Coverage of random vs. manual vs. KLEE testing for 15 randomly-chosen COREUTILS utilities. Manual testing input position mod T 11 overhead made paths ran 7x to 220x more slowly than date -I ls --co cut -f t3.txt native execution), the performance tradeoff for the oth- install --m ers is more nuanced. Assume we have a branch deep in chown a.a - kill -l a nmeter - the program. Covering both true and false directions us- setuidgid a "" envdir ing traditional testing requires running the program from setuidgid printf "% *" B start to finish twice: once for the true path and again od t1.txt envuidgid for the false. In contrast, while KLEE runs each instruc- od t2.txt envdir - tion more slowly than native execution, it only needs to printf % arp -Ainet run the instruction path before the branch once, since it printf %Lo tar tf / forks execution at the branch point (a fast operation given tr [ top d setarch "" "" its object-level copy-on-write implementation). As path tr [= tr [a-z 12 1 : unsigned mod opt(unsigned x, unsigned y) { trived example in Figure 11, which crosschecks a triv- 2 : if((y & −y) == y) // power of two? ial modulo implementation (mod) against one that opti- 3 : return x & (y−1); mizes for modulo by powers of two (mod opt). It first 4 : else makes the inputs x and y symbolic and then uses the 5 : return x % y; 6 : } assert (line 14) to check for differences. Two code 7 : unsigned mod(unsigned x, unsigned y) { paths reach this assert, depending on whether the 8 : return x % y; test for power-of-two (line 2) succeeds or fails. (Along 9 : } the way, KLEE generates a division-by-zero test case for 10: int main() { 11: unsigned x,y; when y =0.) The true path uses the solver to check that 12: make symbolic(&x, sizeof(x)); the constraint (y& − y) == y implies (x&(y − 1)) == 13: make symbolic(&y, sizeof(y)); x%y holds for all values. This query succeeds. The 14: assert(mod(x,y) == mod opt(x,y)); 15: return 0; false path checks the vacuous tautology that the con- 16: } straint (y& − y) 6= y implies that x%y == x%y also holds. The KLEE checking run then terminates, which Figure 11: Trivial program illustrating equivalence checking. means that KLEE has proved equivalence for all non-zero KLEE proves total equivalence when y 6= 0. values using only a few lines of code. 9 modulo bugs in KLEE or non-determinism in the code. This methodology is useful in a broad range of con- Importantly, KLEE will do such proofs for any condition texts. Most standardized interfaces — such as libraries, the programmerexpresses as C code, from a simple non- networking servers, or compilers — have multiple im- null pointer check, to one verifying the correctness of a plementations (a partial motivation for and consequence program’s output. of standardization). In addition, there are other common This property can be leveraged to perform deeper cases where multiple implementations exist: checking as follows. Assume we have two procedures 1 f is a simple reference implementation and f' a real- f and f' that take a single argument and purport to im- world optimized version. plement the same interface. We can verify functional 2 f' is a patched version of f that purports only to equivalence on a per-path basis by simply feeding them remove bugs (so should have strictly fewer crashes) the same symbolic argument and asserting they return or refactor code without changing functionality. the same value: assert(f(x) == f'(x)). Each 3 f has an inverse, which means we can change our time KLEE follows a path that reaches this assertion, it −1 equivalence check to verify f (f(x)) ≡ x, such as: checks if any value exists on that path that violates it. If assert(uncompress(compress(x))==x). it finds none exists, then it has proven functional equiv- alence on that path. By implication, if one function is Experimental results. We show that this technique correct along the path, then equivalence proves the other can find deep correctness errors and scale to real pro- one is as well. Conversely, if the functions compute dif- grams by crosschecking 67 COREUTILS tools against ferent values along the path and the assert fires, then their allegedly equivalent BUSYBOX implementations. KLEE will produce a test case demonstrating this differ- For example, given the same input, the BUSYBOX and ence. These are both powerful results, completely be- COREUTILS versions of wc should output the same num- yond the reach of traditional testing. One way to look at ber of lines, words and bytes. In fact, both the BUSYBOX KLEE is that it automatically translates a path through a and COREUTILS tools intend to conform to IEEE Stan- C program into a form that a theorem prover can reason dard 1003.1 [3] which specifies their behavior. about. As a result, proving path equivalence just takes a We built a simple infrastructure to make crosschecking few lines of C code (the assertion above), rather than an automatic. Given two tools, it renames all their global enormous manual exercise in theorem proving. symbols and then links them together. It then runs both Note that equivalence results only hold on the finite set with the same symbolic environment (same symbolic ar- of paths that KLEE explores. Like traditional testing, it guments, files, etc.) and compares the data printed to cannot make statements about paths it misses. However, stdout. When it detects a mismatch, it generates a test if KLEE is able to exhaust all paths then it has shown total case that can be run to natively to confirm the difference. equivalence of the functions. Although not tractable in Table 3 shows a subset of the mismatches found by general, many isolated algorithms can be tested this way, KLEE. The first three lines show hard correctness er- at least up to some input size. rors (which were promptly fixed by developers), while We help make these points concrete using the con- the others mostly reveal missing functionality. As an ex- ample of a serious correctness bug, the first line gives the 9Code that depends on the values of memory addresses will not satisfy determinism since KLEE will almost certainly allocate memory inputs that when run on BUSYBOX’s comm causes it to objects at different addresses than native runs. behave as if two non-identical files were identical. 13 Input BUSYBOX COREUTILS comm t1.txt t2.txt [does not show difference] [shows difference] tee - [does not copy twice to stdout] [does] tee "" Table 3: Very small subset of the mismatches KLEE found between the BUSYBOX and COREUTILS versions of equivalent utili- ties. The first three are serious correctness errors; most of the others are revealing missing functionality. KLEE Test Random ELOC 1 : static void test(void *upage, unsigned num calls) { With Disk 50.1% 67.1% 4617 2 : make symbolic(upage, PGSIZE); No Disk 48.0% 76.4% 2662 3 : for (int i=0; i 5.6 The HiStar OS kernel Figure 12: Test driver for HISTAR: it makes a single page of We have also applied KLEE to checking non-application user memory symbolic and executes a user-specified number code by using it to check the HiStar [39] kernel. We used of system calls with entirely symbolic arguments. a simple test driver based on a user-mode HISTAR ker- nel. The driver creates the core kernel data structures and initializes a single process with access to a single page of KLEE’s constraint-based reasoning allowed it to find a user memory. It then calls the test function in Figure 12, tricky, critical security bug in the 32-bit version of HIS- which makes the user memory symbolic and executes a TAR. Figure 13 shows the code for the function contain- predefined number of system calls using entirely sym- ing the bug. The function safe addptr is supposed bolic arguments. As the system call number is encoded to set *of to true if the addition overflows. However, in the first argument, this simple driver effectively tests because the inputs are 64 bit long, the test used is insuf- all (sequences of) system calls in the kernel. ficient (it should be (r < a) || (r < b)) and the Although the setup is restrictive, in practice we have function can fail to indicate overflow for large values of found that it can quickly generate test cases — sequences b. of system call vectors and memory contents — which The safe addptr function validates user memory cover a large portion of the kernel code and uncover addresses prior to copying data to or from user space. A interesting behaviors. Table 4 shows the coverage ob- kernel routine takes a user address and a size and com- tained for the core kernel for runs with and without a putes if the user is allowed to access the memory in that disk. When configured with a disk, a majority of the un- range; this routine uses the overflow to prevent access coveredcode can only be triggeredwhen there are a large when a computation could overflow. This bug in com- number of kernel objects. This currently does not happen puting overflow therefore allows a malicious process to in our testing environment; we are investigating ways to gain access to memory regions outside its control. exercise this code adequately during testing. As a quick 6 Related Work comparison, we ran one million random tests through the same driver (similar to § 5.2.4). As Table 4 shows, Many recent tools are based on symbolic execution [11, KLEE’s tests achieve significantly more coverage than 14–16,20–22,24,26,27,36]. We contrast how KLEE random testing both for runs with (+17.0%) and without deals with the environment and path explosion problems. (+28.4%) a disk. To the best of our knowledge, traditional symbolic ex- 14 1 : uintptr t safe addptr(int *of, uint64 t a, uint64 t b) { 7 Conclusion 2 : uintptr t r = a + b; Our long-term goal is to take an arbitrary program and 3 : if (r < a) 4: *of = 1; routinely get 90%+ code coverage, crushing it under test 5 : return r; cases for all interesting inputs. While there is still a long 6 : } way to go to reach this goal, our results show that the ap- proach works well across a broad range of real code. Our Figure 13: HISTAR function containing an important security system KLEE, automatically generated tests that, on av- vulnerability. The function is supposed to set *of to true erage, covered over 90% of the lines (in aggregate over if the addition overflows but can fail to do so in the 32-bit 80%) in roughly 160 complex, system-intensive appli- version for very large values of b. cations ”out of the box.” This coverage significantly exceeded that of their corresponding hand-written test suites, including one built over a period of 15 years. ecution systems [17, 18, 32] are static in a strict sense and In total, we used KLEE to check 452 applications (with do not interact with the running environmentat all. They over 430K lines of code), where it found 56 serious bugs, either cannot handle programs that make use of the en- including ten in COREUTILS, arguably the most heavily- vironment or require a complete working model. More tested collection of open-source applications. To the best recent work in test generation [16,26,36] does allow ex- of our knowledge, this represents an order of magnitude ternal interactions, but forces them to use entirely con- more code and distinct programs than checked by prior crete procedure call arguments, which limits the behav- symbolic test generation work. Further, because KLEE’s iors they can explore: a concrete external call will do ex- constraints have no approximations, its reasoning allow actly what it did, ratherthan all things it could potentially it to prove properties of paths (or find counter-examples do. In KLEE, we strive for a functional balance between without false positives). We used this ability both to these two alternatives; we allow both interaction with the prove path equivalence across many real, purportedly outside environment and supply a model to simulate in- identical applications, and to find functional correctness teraction with a symbolic one. errors in them. The path explosion problem has instead received more The techniques we describe should work well with attention [11,22,24,27,34]. Similarly to the search other tools and provide similar help in handling a broad heuristics presented in Section 3, search strategies pro- class of applications. posed in the past include Best First Search [16], Gener- ational Search [27], and Hybrid Concolic Testing [34]. 8 Acknowledgements Orthogonal to search heuristics, researchers have ad- We thankthe GNU COREUTILS developers, particularly dressed the path explosion problem by testing paths com- the COREUTILS maintainer Jim Meyering for promptly positionally [8,24], and by tracking the values read and confirming our reported bugs and answering many ques- written by the program [11]. tions. We similarly thank the developers of BUSYBOX, Like KLEE, other symbolic execution systems imple- particularly the BUSYBOX maintainer, Denys Vlasenko. ment their own optimizations before sending the queries We also thank Nickolai Zeldovich, the designer of HIS- to the underlying constraint solver, such as the simple TAR, for his great help in checking HISTAR, including syntactic transformations presented in [36], and the con- writing a user-level driver for us. We thank our shepard straint subsumption optimization discussed in [27]. Terence Kelly, the helpful OSDI reviewers, and Philip Similar to symbolic execution systems, model check- Guo for valuable comments on the text. This research ers have been used to find bugs in both the design and was supported by DHS grant FA8750-05-2-0142, NSF the implementation of software [10,12,19,25,29,30]. TRUST grant CCF-0424422, and NSF CAREER award These approaches often require a lot of manual effort to CNS-0238570-001. A Junglee Graduate Fellowship par- build test harnesses. However, the approaches are some- tially supported Cristian Cadar. what complementary to KLEE: the tests KLEE generates References can be used to drive the model checked code, similar to the approach embraced by Java PathFinder [31,37]. [1] Busybox. www.busybox.net, August 2008. Previously, we showed that symbolic execution can [2] Coreutils. www.gnu.org/software/coreutils, August find correctness errors by crosschecking various imple- 2008. mentations of the same library function [15]; this paper [3] IEEE Std 1003.1, 2004 edition. www.unix.org/version3/ shows that the technique scales to real programs. Subse- ieee std.html, May 2008. quent to our initial work, others applied similar ideas to [4] MINIX 3. www.minix3.org, August 2008. finding correctness errors in applications such as network [5] SecurityFocus, www.securityfocus.com, March 2008. protocol implementations [13] and PHP scripts [9]. [6] uCLibc. www.uclibc.org, May 2008. 15 [7] United States National Vulnerability Database, nvd.nist. [24] GODEFROID, P. Compositional dynamic test generation. In Pro- gov, March 2008. ceedings of the 34th Symposium on Principles of Programming Languages (POPL 2007). [8] ANAND, S., GODEFROID, P., AND TILLMANN, N. Demand- driven compositional symbolic execution. In Proceedings of [25] GODEFROID, P. Model Checking for Programming Languages Tools and Algorithms for the Construction and Analysis of Sys- using VeriSoft. In Proceedings of the 24th ACM Symposium on tems (TACAS 2008). Principles of Programming Languages (POPL 1997). [9] ARTZI,S.,KIEZUN˙ ,A., DOLBY,J.,TIP, F., DIG,D.,PARAD- [26] GODEFROID, P., KLARLUND,N., AND SEN, K. DART: Di- KAR,A., AND ERNST, M. D. Finding bugs in dynamic web rected automated random testing. In Proceedings of the Con- applications. In Proceedings of the International Symposium on ference on Programming Language Design and Implementation Software Testing and Analysis (ISSTA 2008). (PLDI 2005). [10] BALL, T., AND RAJAMANI, S. Automatically validating tem- [27] GODEFROID,P., LEVIN, M. Y., AND MOLNAR, D. Automated poral safety properties of interfaces. In SPIN 2001 Workshop on whitebox fuzz testing. In Proceedings of Network and Distributed Model Checking of Software (May 2001). Systems Security (NDSS 2008). [11] BOONSTOPPEL, P., CADAR,C., AND ENGLER, D. RWset: At- [28] HOFFMANN,J., AND KOEHLER, J. A new method to index and tacking path explosion in constraint-based test generation. In query sets. In Proceedings of the Sixteenth International Joint Proceedings of Tools and Algorithms for the Construction and Conference on Artificial Intelligence (IJCAI 1999). Analysis of Systems (TACAS 2008). [29] HOLZMANN, G. J. From code to models. In Proceedings of [12] BRAT,G.,HAVELUND,K.,PARK,S., AND VISSER, W. Model 2nd International Conference on Applications of Concurrency to checking programs. In IEEE International Conference on Auto- System Design (ACSD 2001). mated Software Engineering (ASE 2000). [30] HOLZMANN, G. J. The model checker SPIN. Software Engi- [13] BRUMLEY, D., CABALLERO,J.,LIANG, Z., NEWSOME,J., neering 23, 5 (1997), 279–295. AND SONG, D. Towards automatic discovery of deviations in [31] KHURSHID,S.,PASAREANU,C.S., AND VISSER, W. Gen- binary implementations with applications to error detection and eralized symbolic execution for model checking and testing. In fingerprint generation. In Proceedings of USENIX Security Sym- Proceedings of Tools and Algorithms for the Construction and posium (USENIX Security 2007). Analysis of Systems (TACAS 2003). [14] BRUMLEY, D., NEWSOME, J., SONG, D., WANG,H., AND [32] KROENING,D.,CLARKE,E., AND YORAV, K. Behavioral con- JHA, S. Towards automatic generation of vulnerability-based sig- sistency of C and Verilog programs using bounded model check- natures. In Proceedings of the 2006 IEEE Symposium on Security ing. In Proceedings of the 40th Design Automation Conference and Privacy (IEEE S&P 2006). (DAC 2003). [15] CADAR,C., AND ENGLER, D. Execution generated test cases: [33] LATTNER,C., AND ADVE, V. LLVM: A compilation framework How to make systems code crash itself. In Proceedings of the for lifelong program analysis & transformation. In Proceedings 12th International SPIN Workshop on Model Checking of Soft- of the international symposium on Code generation and optimiza- ware (SPIN 2005). tion (CGO 2004). [16] CADAR, C., GANESH, V., PAWLOWSKI, P., DILL,D., AND [34] MAJUMDAR,R., AND SEN, K. Hybrid concolic testing. In Pro- ENGLER, D. EXE: Automatically generating inputs of death. ceedings of the 29th International Conference on Software Engi- In Proceedings of the 13th ACM Conference on Computer and neering (ICSE 2007). Communications Security (CCS 2006). [35] MILLER,B.,KOSKI,D.,LEE,C. P.,MAGANTY, V., MURTHY, [17] CLARKE,E., AND KROENING, D. Hardware verification using R., NATARAJAN,A., AND STEIDL, J. Fuzz revisited: A re- ANSI-C programs as a reference. In Proceedings of the Asia and examination of the reliability of UNIX utilities and services. South Pacific Design Automation Conference (ASP-DAC 2003). Tech. rep., University of Wisconsin - Madison, 1995. [18] CLARKE,E.,KROENING,D., AND LERDA, F. A tool for check- [36] SEN,K.,MARINOV,D., AND AGHA, G. CUTE: A concolic ing ANSI-C programs. In Proceedings of Tools and Algorithms unit testing engine for C. In In 5th joint meeting of the European for the Construction and Analysis of Systems (TACAS 2004). Software Engineering Conference and ACM Symposium on the Foundations of Software Engineering (ESEC/FSE 2005). [19] CORBETT, J., DWYER,M.,HATCLIFF,J.,LAUBACH,S., PASAREANU,C.,ROBBY, AND ZHENG, H. Bandera: Extracting [37] VISSER, W., PASAREANU,C.S., AND KHURSHID, S. Test finite-state models from Java source code. In Proceedings of the input generation with Java PathFinder. In ISSTA ’04: Proceedings International Conference on Software Engineering (ICSE 2000). of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis (New York, NY, USA, 2004), ACM Press, [20] COSTA, M., CASTRO,M.,ZHOU,L.,ZHANG,L., AND pp. 97–107. PEINADO, M. Bouncer: Securing software by blocking bad in- put. In Proceedings of the 21th ACM Symposium on Operating [38] YANG, J., SAR,C., AND ENGLER, D. eXplode: a lightweight, Systems Principles (SOSP 2007). general system for finding serious storage system errors. In Pro- ceedings of the 7th Symposium on Operating Systems Design and [21] COSTA, M., CROWCROFT, J., CASTRO, M., ROWSTRON,A., Implementation (OSDI 2006). ZHOU,L.,ZHANG,L., AND BARHAM, P. Vigilante: end-to-end containment of Internet worms. In Proceedings of the 20th ACM [39] ZELDOVICH, N., BOYD-WICKIZER,S.,KOHLER,E., AND Symposium on Operating Systems Principles (SOSP 2005). MAZIERES` , D. Making information flow explicit in HiStar. In Proceedings of the 7th Symposium on Operating Systems Design [22] EMMI,M.,MAJUMDAR,R., AND SEN, K. Dynamic test input and Implementation (OSDI 2006). generation for database applications. In International Symposium on Software Testing and Analysis (ISSTA 2007). [23] GANESH, V., AND DILL, D. L. A decision procedure for bit- vectors and arrays. In Proceedings of the 19th International Con- ference on Computer Aided Verification (CAV 2007). 16