Semantic Fuzzing with Zest Rohan Padhye Caroline Lemieux Koushik Sen University of California, Berkeley University of California, Berkeley University of California, Berkeley USA USA USA
[email protected] [email protected] [email protected] Mike Papadakis Yves Le Traon University of Luxembourg University of Luxembourg Luxembourg Luxembourg
[email protected] [email protected] ABSTRACT Syntactic Semantic Valid Input Output Programs expecting structured inputs often consist of both a syntac- Stage Stage tic analysis stage, which parses raw input, and a semantic analysis Syntactically Semantically stage, which conducts checks on the parsed input and executes Invalid Invalid the core logic of the program. Generator-based testing tools in the Syntactic Semantic lineage of QuickCheck are a promising way to generate random Error Error syntactically valid test inputs for these programs. We present Zest, a technique which automatically guides QuickCheck-like random- input generators to better explore the semantic analysis stage of test Figure 1: Inputs to a program taking structured inputs can programs. Zest converts random-input generators into determinis- be either syntactically or semantically invalid or just valid. tic parametric generators. We present the key insight that mutations in the untyped parameter domain map to structural mutations in the input domain. Zest leverages program feedback in the form 1 INTRODUCTION of code coverage and input validity to perform feedback-directed parameter search. We evaluate Zest against AFL and QuickCheck Programs expecting complex structured inputs often process their on five Java programs: Maven, Ant, BCEL, Closure, and Rhino. Zest inputs and convert them into suitable data structures before in- covers 1:03×–2:81× as many branches within the benchmarks’ se- voking the actual functionality of the program.