DeepState: Symbolic Unit Testing for C and C++ Peter Goodman Alex Groce Trail of Bits, Inc. School of Informatics, Computing & Cyber Systems [email protected] Northern Arizona University [email protected] Abstract—Unit testing is a popular software development larger audience of developers. DeepState makes it possible methodology that can help developers detect functional regres- to write parameterized unit tests [38] in a Google Test- sions, explore boundary conditions, and document expected be- like framework, and automatically produce tests using angr, havior. However, writing comprehensive unit tests is challenging and time-consuming, and developers seldom explore the obscure Manticore, or Dr. Memory’s fuzzer, Dr. Fuzz [2]. (and bug-hiding) corners of software behavior without assistance. DeepState also targets the same space as property-based DeepState is a tool that provides a Google Test-like API to give testing tools such as QuickCheck [10], ScalaCheck [28], C and C++ developers push-button access to symbolic execution Hypothesis [26], and TSTL [17], [20], but DeepState’s test engines, such as Manticore and angr, and fuzzers, such as Dr. harnesses look like C/C++ unit tests. The major difference Fuzz. Rather than learning multiple complex tools, users learn one interface for defining a test harness, and can use various from previous tools is that DeepState aims to provide a methods to automatically generate tests for software. In addition front-end that can make use of a growing variety of back- to providing a familiar interface to binary analysis and fuzzing end methods for test generation, including (already) multiple for parameterized unit testing, DeepState also provides constructs binary analysis engines and a non-symbolic fuzzer. Developers that aid in the construction of API-sequence tests, where the who write tests using DeepState can expect that DeepState tool chooses the functions or methods to call, allowing for even more diverse and powerful tests. By serving as a front-end to will let them, without rewriting their tests, make use of new multiple tools, DeepState additionally provides a way to apply binary analysis or fuzzing advances. The harness/test definition (novel) high-level strategies to test generation, and to compare remains the same, but the method(s) used to generate tests effectiveness and efficiency of testing back-ends, including binary may change over time. In contrast, most property-based tools analysis tools. only provide random testing, and symbolic execution based approaches such as Pex [37], [39] or KLEE [8], while similar I. INTRODUCTION on the surface in some ways, always have a single back-end for A key limitation in the advancement of binary analysis and test generation. Even a system such as TSTL, which aims to other approaches to improving software security and reliability provide a common interface [14] to different testing methods is that there is very little overlap between security experts assumes that all of those methods will be written using the familiar with tools such as angr [34], [35], [33], Manticore TSTL API and (concrete) notion of state. [27], or S2E [9], and the developers who produce most code In addition to letting developers find the best tool for that needs to be secure or highly reliable. testing their system (or find different bugs with different Developers, as a class, do not know how to use binary tools), the ability to shift back-ends effortlessly addresses a analysis tools; developers, as a class, seldom even know how core problem of practical automated test generation. Most of to use less challenging tools such as fuzzers, even relatively the tools in wide use are research prototypes with numerous push-button ones such as AFL [41]. Even those developers bugs. In fact, in our own early efforts to test a file system whose primary code focus is critical security infrastructure using DeepState, we discovered a show-stopping fault in such as OpenSSL are often not users, much less expert users, angr [3]. Without DeepState, encountering such a bug means of such tools. stopping testing efforts until the bug can be fixed (which Developers do, however, often know how to use unit testing can be difficult, especially if reporting a bug arising from frameworks, such as JUnit [12] or Google Test [1]. DeepState sensitive code is at issue), or re-writing your tests (possibly aims to bring some of the power (in particular, high quality after learning a new tool). With DeepState, one simply types automated test generation) of binary analysis frameworks to a deepstate-manticore instead of deepstate-angr to switch from angr to Manticore. Moreover, sharing a single notion of test harness makes it possible to implement high-level test generation strategies once and thus avoid the effort of re-implementing such ap- proaches for every tool (or, worse yet, every individual testing effort, as happens with many strategies). In this paper we describe the design and implementation of Network and Distributed Systems Security (NDSS) Symposium 2018 18-21 February 2018, San Diego, CA, USA DeepState (Section II) and show how DeepState’s approach ISBN 1-1891562-49-5 enables API-call sequence testing of a simple user mode http://dx.doi.org/10.14722/ndss.2018.23xxx www.ndss-symposium.org DeepState.hpp libdeepstate.a 1 bool IsPrime(const unsigned p) { #define TEST(... DeepState_Assume: for (unsigned i = 2; i <= (p/2); ++i) { #define TEST_F(... ... 2 #define ASSUME(... DeepState_Pass: 3 if (!(p % i)) { #define EXPECT(... ... #define ASSERT(... DeepState_SoftFail: 4 return false; #define LOG(... ... DeepState_ 5 } 6 } 7 return true; 8 } Tests.cpp Tests.o Tests 9 #include <DeepState Unit_Name1_Test: 0101010 call 1010101 10 TEST(PrimePolynomial, OnlyGeneratesPrimes) { TEST(Unit, Name1) { DeepState_Int 0010101 11 symbolic_unsigned x, y, z; symbolic_int x; cmp eax, 0 0110101 ASSUME(x > 0); jg .L0 0100101 12 ASSUME_GT(x, 0); ... call DeepState_.. 0101010 } .L0: 1010101 13 unsigned poly = (x * x) + x + 41; ... 0101010 ASSUME_GT(y, 1); TEST(Unit, 1010101 14 0101010 15 ASSUME_GT(z, 1); 16 ASSUME_LT(y, poly); 17 ASSUME_LT(z, poly); 18 ASSERT_NE(poly, y * z) Fig. 1. The DeepState workflow mirrors that of Google Test. The developer 19 << x << "ˆ2 + " << x << " + 41 is not prime"; includes the DeepState API header file (DeepState.hpp) into their testing 20 ASSERT(IsPrime(Pump(poly))) code (Tests.cpp), and uses the various macros, such as TEST, to organize 21 << x << "ˆ2 + " << x << " + 41 is not prime"; the code to be tested. The compiled tests (Tests.o) are linked against the 22 } DeepState runtime (libdeepstate.a), producing a test suite binary (Tests). This 23 binary can be run independently, or under the control of programs like 24 int main(int argc, char *argv[]) { DeepState_InitOptions(argc, argv); deepstate-angr. 25 26 return DeepState_Run(); 27 } filesystem (Section III). Finally, we show how the DeepState Fig. 2. A simple test case that asserts that every number generated by the approach enables novel solutions to scalability challenges in polynomial x2 + x + 41 is prime. The Pump function (described in Section symbolic execution (Section IV). IV-A) “pre-forks” the executor on possible assignments to poly, thereby enabling many state forks to simultaneously execute IsPrime at native speeds on many different concrete assignments to poly. II. IMPLEMENTATION A. Architecture by the DeepState library to define and run tests. The key DeepState is made of two components: i) a static library elements of a DeepState-based test suite are the definition of linked into every test suite (Figure 1), and ii) the executor, tests and test fixtures, the sources of symbolic values, and the which is a lightweight orchestrator around symbolic execution pre- and post-conditions that constrain the properties to be engines like Manticore and angr, or fuzzing engines like tested. Figure 2 shows how these basic components combine Dr. Fuzz. Adding new back-ends involves producing a new into a unit test. executor, but does not require changes to the static library. 1) Library: The DeepState library is approximately 1200 B. Tests lines of C, with an additional 600 lines of C++ which wrap Unit tests are functions that are defined using the core C APIs and provide the conveniences of a Google the special TEST macro. This macro denotes the Test-like system. The majority of the DeepState library code is unit name (PrimePolynomial) and the test name unrelated to the core task of testing. In fact, a lot of complexity (OnlyGeneratesPrimes) accordingly. Contained within in DeepState relates to ensuring that the common practice of the typical body of a TEST-defined function is code that sets printf-like logging inside of unit tests does not degrade up the environment, executes the functions to be tested, and performance. asserts any pre- or post-conditions that must hold. 2) Executor: The orchestration engine is approximately The TEST macro wraps the test code in a function that calls 350 lines of Python, with an additional 250 lines each for the DeepState_Pass function as its last action. The macro enabling Manticore and angr support. DeepState uses APIs also ensures that the wrapped function is registered at runtime already provided by angr and Manticore to locate and hook into a global linked list of all test cases. At runtime, the key APIs and find all test cases to run. In the case of executor simulates the program’s normal execution (including Dr. Fuzz, a special test case harness function is defined, test registration) up until the call to DeepState_Run. At DrMemFuzzFunc, which is Dr. Fuzz’s standard “entry-point” this point, the executor takes control, and forks execution for into a code-base and the source of mutated input bytes. each registered test. The executor will exhaustively explore Eventually we plan to integrate libFuzzer [32] (and perhaps the state space of each test case, up until the execution other fuzzers) in a similar way.