Achieving High Coverage for Floating-point Code via Unconstrained Programming (Extended Version) Zhoulai Fu Zhendong Su University of California, Davis, USA [email protected] [email protected] Abstract execution and its variants can perform systematic path explo- Achieving high code coverage is essential in testing, which ration, but suffer from path explosion and are weak in dealing gives us confidence in code quality. Testing floating-point with complex program logic involving numerical constraints. code usually requires painstaking efforts in handling floating- Our Work This paper considers the problem of coverage- point constraints, e.g., in symbolic execution. This paper turns based testing for floating-point code and focuses on the cov- the challenge of testing floating-point code into the oppor- erage of program branches. We turn the challenge of test- tunity of applying unconstrained programming — the math- ing floating-point programs into the opportunity of apply- ematical solution for calculating function minimum points ing unconstrained programming — the mathematical solu- over the entire search space. Our core insight is to derive a tion for calculating function minima over the entire search representing function from the floating-point program, any of space [38, 51]. whose minimum points is a test input guaranteed to exercise Our approach has two unique features. First, it introduces a new branch of the tested program. This guarantee allows the concept of representing function, which reduces the us to achieve high coverage of the floating-point program by branch coverage based testing problem to the unconstrained repeatedly minimizing the representing function. programming problem. Second, the representing function We have realized this approach in a tool called CoverMe is specially designed to achieve the following theoretical and conducted an extensive evaluation of it on Sun’s C math guarantee: Each minimum point of the representing function library. Our evaluation results show that CoverMe achieves, is an input to the tested floating-point program, and the input on average, 90.8% branch coverage in 6.9 seconds, drastically necessarily triggers a new branch unless all branches have outperforming our compared tools: (1) Random testing, (2) been covered. This guarantee is critical not only for the AFL, a highly optimized, robust fuzzer released by Google, soundness of our approach, but also for its efficiency — the and (3) Austin, a state-of-the-art coverage-based testing tool unconstrained programming process is designed to cover only designed to support floating-point code. new branches; it does not waste efforts on covering already covered branches. 1. Introduction We have implemented our approach into a tool called Test coverage criteria attempt to quantify the quality of test CoverMe. CoverMe first derives the representing function data. Coverage-based testing [39] has become the state-of- from the program under test. Then, it uses an existing uncon- the-practice in the software industry. The higher expectation strained programming algorithm to compute the minimum arXiv:1704.03394v1 [cs.PL] 11 Apr 2017 for software quality and the shrinking development cycle points. Note that the theoretical guarantee mentioned above have driven the research community to develop a spectrum allows us to apply any unconstrained programming algorithm of automated testing techniques for achieving high code as a black box. Our implementation uses an off-the-shelf coverage. Monte Carlo Markov Chain (MCMC) [10] tool. A significant challenge in coverage-based testing lies in CoverMe has achieved high or full branch coverage for the testing of numerical code, e.g., programs with floating- the tested floating-point programs. Fig.1 lists the program point arithmetic, non-linear variable relations, or external s_tanh.c from our benchmark suite Fdlibm [5]. The program function calls, such as logarithmic and trigonometric func- takes a double input. In Line 3, variable jx is assigned tions. Existing solutions include random testing [13, 26], with the high word of x according to the comment given symbolic execution [14, 17, 20, 27], and various search-based in the source code; the right-hand-side expression in the strategies [11, 29, 32, 35], which have found their way into assignment takes the address of x (&x), cast it as a pointer- many mature implementations [15, 16, 46]. Random testing to-int (int*), add 1, and dereference the resulting pointer. is easy to employ and fast, but ineffective in finding deep In Line 4, variable ix is assigned with jx whose sign bit is semantic issues and handling large input spaces; symbolic masked off. Lines 5-15 are two nested conditional statements 2. Background 1 double tanh(double x){ 2 int jx, ix; This section presents the definition and algorithms of un- 3 jx = *(1+(int*)&x); // High word of x constrained programming that will be used in this paper. As 4 ix = jx&0x7fffffff; 5 if(ix>=0x7ff00000) { mentioned in Sect.1, we will treat the unconstrained pro- 6 if (jx>=0) ...; gramming algorithms as black boxes. 7 else ...; 8 } Unconstrained Programming We formalize unconstrained 9 if (ix < 0x40360000) { programming as the problem below [22]: 10 if (ix<0x3c800000) ...; Given f : Rn ! R 12 if (ix>=0x3ff00000) ...; ∗ n ∗ n 13 else ...; Find x 2 R for which f (x ) ≤ f (x) for all x 2 R 14 } 15 else ...; where f is the objective function; x∗, if found, is called a 16 return ...; ( ∗) 17 } minimum point; and f x is the minimum. An example is 2 2 f (x1;x2) = (x1 − 3) + (x2 − 5) ; (1) Figure 1: Benchmark program s_tanh.c taken from Fdlibm. which has the minimum point x∗ = (3;5). on ix and jx, which contain 16 branches in total according Unconstrained Programming Algorithms We consider to Gcov [6]. Testing this type of programs is beyond the two kinds of algorithms, known as local optimization and capabilities of traditional symbolic execution tools such global optimization. Local optimization focuses on how func- as Klee [16]. CoverMe achieves full coverage within 0.7 tions are shaped near a given input and where a minimum can seconds, dramatically outperforming our compared tools, be found at local regions. It usually involves standard tech- including random testing, Google’s AFL, and Austin (a niques such as Newton’s or the steepest descent methods [40]. tool that combines symbolic execution and search-based Fig.2(a) shows a common local optimization method with heuristics). See details in Sect.6. the objective function f (x) that equals 0 if x ≤ 1, or (x − 1)2 otherwise. The algorithm uses tangents of f to converge to a Contributions This work introduces a promising auto- minimum point quickly. In general, local optimization is usu- mated testing solution for programs that are heavy on floating- ally fast. If the objective function is smooth to some degree, point computation. Our approach designs the representing the local optimization can deduce function behavior in the function whose minimum points are guaranteed to exercise neighborhood of a particular point x by using information at new branches of the floating-point program. This guarantee x only (the tangent here). allows us to apply any unconstrained programming solution Global optimization for unconstrained programming as a black box, and to efficiently generate test inputs for searches for minimum points over Rn. Many global opti- covering program branches. mization algorithms have been developed. This work uses Our implementation, CoverMe, proves to be highly effi- Monte Carlo Markov Chain (MCMC) [10]. MCMC is a cient and effective. It achieves 90.8% branch coverage on sampling method that targets (usually unknown) probability average, which is substantially higher than those obtained by distributions. A fundamental fact is that MCMC sampling random testing (38.0%), AFL [1] (72.9%), and Austin [30] follows the target distributions asymptotically, which is for- (42.8%). malized by the lemma below. For simplicity, we present the Paper Outline We structure the rest of the paper as follows. lemma in the form of discrete-valued probabilities [10]. Sect.2 presents background material on unconstrained pro- Lemma 2.1. Let x be a random variable, A be an enumerable gramming. Sect.3 gives an overview of our approach, and set of the possible values of x, f be a target probability Sect.4 presents the algorithm. Sect.5 describes our imple- distribution for x, i.e., the probability of x taking value a 2 A mentation CoverMe, and Sect.6 describes our evaluation. is f (a). Then, an MCMC sampling sequence x1;:::;xn;::: Sect.7 surveys related work and Sect.8 concludes the paper. satisfies the property that Prob(xn = a) ! f (a). For completeness, Sect.A-D provide additional details on our approach. For example, consider the target distribution of coin tossing with 0:5 probability of getting a head. An MCMC Notation We write F for floating-point numbers, Z for sampling is a sequence of random variables x1,. , xn;:::, integers, Z>0 for strictly positive integers. we use the ternary such that the probability of xn being head converges to 0:5. operation B ? a : a0 to denote an evaluation to a if B Using MCMC to solve unconstrained programming prob- holds, or a0 otherwise. The lambda terms in the form of lems provides multiple advantages in practice. First, Lem. 2.1 lx: f (x) may denote mathematical function f or its machine ensures that MCMC sampling can be configured to attain the implementation according to the given context. minimum points with higher probability than the other points. FOO:: ProgramProgram under under test test in any LLVM-supported language p4 type_tFOO(doublex1,doublex2,...) p2 p0 pen (.cpp) double pen (int i, int op, double lhs, double rhs) p1 p3 p5 FOO_I: Instrumented program (.bc) xn type_tFOO_I(doublex1,doublex2,...) (a) (b) FOOloader: Program(.cpp) under test in any LLVM-supported language Figure 2: (a) Local optimization example with objective function void LOADER (double* P) l : ≤ ( − )2 type_tFOO(doublex1,doublex2,...) x x 1 ? 0 : x 1 .