DOCSLIB.ORG

Symbolic Execution for Javascript

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Symbolic Execution for JavaScript José Fragoso Santos Petar Maksimović Théotime Grohens Imperial College London, UK Imperial College London, UK ENS Paris, France [email protected] Mathematical Institute SASA, Serbia [email protected] [email protected] Julian Dolby Philippa Gardner IBM Research, New York, USA Imperial College London, UK [email protected] [email protected] ABSTRACT Whole-program analysis is not sufficient for JavaScript appli- We present a framework for trustworthy symbolic execution of cations, which are commonly run in highly dynamic execution JavaScripts programs, whose aim is to assist developers in the test- environments: for example, client-side programs dynamically load ing of their code: the developer writes symbolic tests for which and execute third-party code as a matter of course. One also needs the framework provides concrete counter-models. We create the to be able to analyse incomplete code in a compositional manner. framework following a new, general methodology for designing Designing compositional analyses for dynamic languages such as compositional program analyses for dynamic languages. We prove JavaScript, however, is non-trivial. This is because, unlike static lan- that the underlying symbolic execution is sound and does not gen- guages such as C, C++ and Java, dynamic languages do not observe erate false positives. We establish additional trust by using the the frame property [39], essential for compositionality. Intuitively, theory to precisely guide the implementation and by thorough test- the frame property means that the output of a program cannot ing. We apply our framework to whole-program symbolic testing change if the state in which it is run is extended. In JavaScript, of real-world JavaScript libraries and compositional debugging of it is possible to introduce bugs by extending the state in which a separation logic specifications of JavaScript programs. program is run. To our knowledge, there has been no previous work on compositional symbolic execution for JavaScript. CCS CONCEPTS We introduce Cosette, a framework for trusted symbolic execu- tion of JavaScript (ECMAScript 5 Strict [18]). Its aim is to provide • Theory of computation → Separation logic; Program anal- general-purpose symbolic analysis and assist developers in the test- ysis; Logic and verification; • Software and its engineering → ing of their code. We present a new, general methodology for deve- Automated static analysis; loping compositional analyses for languages that do not satisfy the KEYWORDS frame property, and apply it to the design of the symbolic execution of Cosette. In contrast to existing tools, the symbolic analysis of Symbolic execution, Separation logic, Formal semantics, JavaScript Cosette is fully formalised in a way that guides the implementation ACM Reference Format: and is also trusted, in that it follows the JavaScript semantics and José Fragoso Santos, Petar Maksimović, Théotime Grohens, Julian Dolby, does not produce false positive bug reports. We apply Cosette to and Philippa Gardner. 2018. Symbolic Execution for JavaScript. In Pro- whole-program symbolic testing of real-world JavaScript libraries ceedings of The 20th International Symposium on Principles and Practice of and compositional debugging of separation logic specifications of Editor E. Editor (Ed.). ACM, New York, Declarative Programming (PPDP’18), JavaScript programs. We illustrate these use cases in §2. NY, USA, Article -, 14 pages. https://doi.org/sampleDOI We show the architec- 1 INTRODUCTION ture of Cosette on the right. We extend JS with con- JavaScript (JS) is the most widespread dynamic language, used by structs for creating and 95.1% of websites [51]. Due to its dynamic, complex nature, it is a reasoning about symbolic difficult target for symbolic analysis. Recent symbolic execution values. Using JS-2-JSIL [22], tools for JS [30, 42, 52] have made significant progress: they are fully a trusted compiler from JS automatic, aim at code in the large, and primarily focus on scalabil- to the JSIL intermediate ity and coverage issues. However, they do have some limitations. language, we compile the extended JS program to an extended They only target specific bug patterns rather than general-purpose JSIL program. The core of Cosette is a new symbolic interpreter for symbolic execution, and depend on whole-program analysis. JSIL, written in Rosette [48, 49], a framework for creating symbolic Permission to make digital or hard copies of part or all of this work for personal or interpreters. The JSIL symbolic interpreter either outputs a concrete classroom use is granted without fee provided that copies are not made or distributed counter-model for the given assertions, or guarantees correctness for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. up to a given bound for unfolding loops and recursive predicates. For all other uses, contact the owner/author(s). Our new, general methodology underpinning the compositional PPDP’18, September 2018, Frankfurt, Germany JSIL symbolic interpreter consists of three stages: (1) we design a © 2018 Copyright held by the owner/author(s). ACM ISBN sampleISBN. JSIL instrumented semantics that exhibits the frame property by https://doi.org/sampleDOI explicitly keeping track of object properties that we know are not PPDP’18, September 2018, Frankfurt, Germany J. Fragoso Santos, P. Maksimović, T. Grohens, J. Dolby, P. Gardner present; (2) we define the JSIL symbolic semantics by lifting theJSIL Our running example is a key-value map implementation, given instrumented semantics; and (3) we link the JSIL symbolic semantics in Figure 1a. It contains four functions: Map, for constructing an to the JSIL concrete semantics by describing the frames that can empty map; get, for retrieving the value associated with a given be safely added to the initial state. The key innovation is to have key; put, for inserting/updating key-value pairs; and validKey, for the instrumented semantics as a proper interim stage in the design deciding whether or not a key is valid. The library implements a of the symbolic execution, obtaining more modular reasoning and key-value map as an object with property _contents, denoting the substantially simpler proofs. We present this methodology in §3. object storing the map contents. The named properties of _contents An essential goal for us was to establish trust in Cosette. In §3.5, and their value attributes correspond to the map keys and values. we give a bounded soundness result for the JSIL symbolic semantics The functions get, put, and validKey are shared between all map and prove that Cosette never produces false positive bug reports. objects, as they are defined in Map.prototype, the prototype1 of These results, combined with the correctness of JS-2-JSIL and the objects created using Map as a constructor. The get function returns fact that the memory models of JavaScript and JSIL are the same the value associated with a given key, or null if the key is not in by design, enable us to lift the results of analyses done on com- the map. To check that the given key is in the map, get uses the piled JSIL code back to JS (cf. §3.7, §4.4). We implement the JSIL built-in function hasOwnProperty, which lives in Object.prototype, instrumented interpreter following the instrumented semantics of the default prototype of all objects. The put function updates the §3.3 to the letter. We test the combination of JS-2-JSIL and the JSIL map if the supplied key is valid, and otherwise throws an error. instrumented interpreter using Test262, the official ECMAScript In Figure 1b, we show a general heap of key-value maps created test suite [19]. Out of the 10469 tests for ES5 Strict, we identify by the library. The map object, whose prototype is Map.prototype, 8330 tests appropriate for our coverage, of which we pass 100%. has the property _contents, pointing to the contents object. The Finally, we ensure that the JSIL symbolic interpreter obtained by contents object holds the key-value pairs, and its prototype is the Rosette lifting of the instrumented interpreter is consistent with Object.prototype. The Map.prototype object holds the get, put, and the symbolic semantics of §3.4 by constructing symbolic unit tests validKey functions,2 and its prototype is also Object.prototype. Fi- for each JSIL command, assuming the premises and asserting the nally, the Object.prototype holds the hasOwnProperty function that conclusion of the appropriate rule of the symbolic semantics. is called by Map.prototype.get. We apply Cosette to whole-program symbolic testing. A gen- 2.1 Whole-program Symbolic Testing eral developer can use Cosette for symbolic testing of their code by having symbolic inputs instead of concrete inputs and stating Developers are used to writing unit tests—verifying that, given the constraints that the output needs to satisfy as simple, intuitive some concrete inputs, their code produces the expected outputs. first-order assertions over these inputs. If a test fails, Cosette pro- Using Cosette, they can write unit tests with symbolic inputs and vides the concrete input that causes it to fail, exposing bugs in the outputs, testing a broad range of behaviours with a single symbolic tested code. In §5, we evaluate Cosette on two real-world JavaScript test. For example, one unit test for the put function consists of data structure libraries, where, using fewer tests, we achieve better inserting a valid key-value pair (k, v) into a map and then verifying coverage (100%) than the concrete unit test suites shipped with the that it has been inserted correctly. In Cosette, this test can be written libraries, and discover unexpected bugs in both libraries. as in Figure 1c. First, we declare k to be a symbolic string and v We also apply Cosette to specification-driven bug-finding.
Recommended publications
  • Chopped Symbolic Execution

    Chopped Symbolic Execution

    Chopped Symbolic Execution David Trabish Andrea Mattavelli Noam Rinetzky Cristian Cadar Tel Aviv University Imperial College London Tel Aviv University Imperial College London Israel United Kingdom Israel United Kingdom [email protected] [email protected] [email protected] [email protected] ABSTRACT the code with symbolic values instead of concrete ones. Symbolic Symbolic execution is a powerful program analysis technique that execution engines thus replace concrete program operations with systematically explores multiple program paths. However, despite ones that manipulate symbols, and add appropriate constraints on important technical advances, symbolic execution often struggles to the symbolic values. In particular, whenever the symbolic executor reach deep parts of the code due to the well-known path explosion reaches a branch condition that depends on the symbolic inputs, it problem and constraint solving limitations. determines the feasibility of both sides of the branch, and creates In this paper, we propose chopped symbolic execution, a novel two new independent symbolic states which are added to a worklist form of symbolic execution that allows users to specify uninter- to follow each feasible side separately. This process, referred to as esting parts of the code to exclude during the analysis, thus only forking, refines the conditions on the symbolic values by adding targeting the exploration to paths of importance. However, the appropriate constraints on each path according to the conditions excluded parts are not summarily ignored, as this may lead to on the branch. Test cases are generated by finding concrete values both false positives and false negatives. Instead, they are executed for the symbolic inputs that satisfy the path conditions.
  • Processing an Intermediate Representation Written in Lisp

    Processing an Intermediate Representation Written in Lisp

    Processing an Intermediate Representation Written in Lisp Ricardo Pena,˜ Santiago Saavedra, Jaime Sanchez-Hern´ andez´ Complutense University of Madrid, Spain∗ [email protected],[email protected],[email protected] In the context of designing the verification platform CAVI-ART, we arrived to the need of deciding a textual format for our intermediate representation of programs. After considering several options, we finally decided to use S-expressions for that textual representation, and Common Lisp for processing it in order to obtain the verification conditions. In this paper, we discuss the benefits of this decision. S-expressions are homoiconic, i.e. they can be both considered as data and as code. We exploit this duality, and extensively use the facilities of the Common Lisp environment to make different processing with these textual representations. In particular, using a common compilation scheme we show that program execution, and verification condition generation, can be seen as two instantiations of the same generic process. 1 Introduction Building a verification platform such as CAVI-ART [7] is a challenging endeavour from many points of view. Some of the tasks require intensive research which can make advance the state of the art, as it is for instance the case of automatic invariant synthesis. Some others are not so challenging, but require a lot of pondering and discussion before arriving at a decision, in order to ensure that all the requirements have been taken into account. A bad or a too quick decision may have a profound influence in the rest of the activities, by doing them either more difficult, or longer, or more inefficient.
  • Rubicon: Bounded Verification of Web Applications

    Rubicon: Bounded Verification of Web Applications

    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by DSpace@MIT Rubicon: Bounded Verification of Web Applications Joseph P. Near, Daniel Jackson Computer Science and Artificial Intelligence Lab Massachusetts Institute of Technology Cambridge, MA, USA {jnear,dnj}@csail.mit.edu ABSTRACT ification language is an extension of the Ruby-based RSpec We present Rubicon, an application of lightweight formal domain-specific language for testing [7]; Rubicon adds the methods to web programming. Rubicon provides an embed- quantifiers of first-order logic, allowing programmers to re- ded domain-specific language for writing formal specifica- place RSpec tests over a set of mock objects with general tions of web applications and performs automatic bounded specifications over all objects. This compatibility with the checking that those specifications hold. Rubicon's language existing RSpec language makes it easy for programmers al- is based on the RSpec testing framework, and is designed to ready familiar with testing to write specifications, and to be both powerful and familiar to programmers experienced convert existing RSpec tests into specifications. in testing. Rubicon's analysis leverages the standard Ruby interpreter to perform symbolic execution, generating veri- Rubicon's automated analysis comprises two parts: first, fication conditions that Rubicon discharges using the Alloy Rubicon generates verification conditions based on specifica- Analyzer. We have tested Rubicon's scalability on five real- tions; second, Rubicon invokes a constraint solver to check world applications, and found a previously unknown secu- those conditions. The Rubicon library modifies the envi- rity bug in Fat Free CRM, a popular customer relationship ronment so that executing a specification performs symbolic management system.
  • Advanced Systems Security: Symbolic Execution

    Advanced Systems Security: Symbolic Execution

    Systems and Internet Infrastructure Security Network and Security Research Center Department of Computer Science and Engineering Pennsylvania State University, University Park PA Advanced Systems Security:! Symbolic Execution Trent Jaeger Systems and Internet Infrastructure Security (SIIS) Lab Computer Science and Engineering Department Pennsylvania State University Systems and Internet Infrastructure Security (SIIS) Laboratory Page 1 Problem • Programs have flaws ‣ Can we find (and fix) them before adversaries can reach and exploit them? • Then, they become “vulnerabilities” Systems and Internet Infrastructure Security (SIIS) Laboratory Page 2 Vulnerabilities • A program vulnerability consists of three elements: ‣ A flaw ‣ Accessible to an adversary ‣ Adversary has the capability to exploit the flaw • Which one should we focus on to find and fix vulnerabilities? ‣ Different methods are used for each Systems and Internet Infrastructure Security (SIIS) Laboratory Page 3 Is It a Flaw? • Problem: Are these flaws? ‣ Failing to check the bounds on a buffer write? ‣ printf(char *n); ‣ strcpy(char *ptr, char *user_data); ‣ gets(char *ptr) • From man page: “Never use gets().” ‣ sprintf(char *s, const char *template, ...) • From gnu.org: “Warning: The sprintf function can be dangerous because it can potentially output more characters than can fit in the allocation size of the string s.” • Should you fix all of these? Systems and Internet Infrastructure Security (SIIS) Laboratory Page 4 Is It a Flaw? • Problem: Are these flaws? ‣ open( filename, O_CREAT );
  • Test Generation Using Symbolic Execution

    Test Generation Using Symbolic Execution

    Test Generation Using Symbolic Execution Patrice Godefroid Microsoft Research [email protected] Abstract This paper presents a short introduction to automatic code-driven test generation using symbolic execution. It discusses some key technical challenges, solutions and milestones, but is not an exhaustive survey of this research area. 1998 ACM Subject Classification D.2.5 Testing and Debugging, D.2.4 Software/Program Veri- fication Keywords and phrases Testing, Symbolic Execution, Verification, Test Generation Digital Object Identifier 10.4230/LIPIcs.FSTTCS.2012.24 1 Automatic Code-Driven Test Generation In this paper, we discuss the problem of automatic code-driven test generation: Given a program with a known set of input parameters, automatically generate a set of input values that will exercise as many program statements as possible. Variants of this problem definition can be obtained using other code coverage criteria [48]. An optimal solution to this problem is theoretically impossible since this problem is undecidable in general (for infinite-state programs written in Turing-expressive programming languages). In practice, approximate solutions are sufficient. Although automating test generation using program analysis is an old idea (e.g., [41]), practical tools have only started to emerge during the last few years. Indeed, the expensive sophisticated program-analysis techniques required to tackle the problem, such as symbolic execution engines and constraint solvers, have only become computationally affordable in recent years thanks to the increasing computational power available on modern computers. Moreover, this steady increase in computational power has in turn enabled recent progress in the engineering of more practical software analysis techniques. Specifically, this recent progress was enabled by new advances in dynamic test generation [29], which generalizes and is more powerful than static test generation, as explained later in this paper.
  • HOL-Testgen Version 1.8 USER GUIDE

    HOL-Testgen Version 1.8 USER GUIDE

    HOL-TestGen Version 1.8 USER GUIDE Achim Brucker, Lukas Brügger, Abderrahmane Feliachi, Chantal Keller, Matthias Krieger, Delphine Longuet, Yakoub Nemouchi, Frédéric Tuong, Burkhart Wolff To cite this version: Achim Brucker, Lukas Brügger, Abderrahmane Feliachi, Chantal Keller, Matthias Krieger, et al.. HOL-TestGen Version 1.8 USER GUIDE. [Technical Report] Univeristé Paris-Saclay; LRI - CNRS, University Paris-Sud. 2016. hal-01765526 HAL Id: hal-01765526 https://hal.inria.fr/hal-01765526 Submitted on 12 Apr 2018 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. R L R I A P P O R HOL-TestGen Version 1.8 USER GUIDE T BRUCKER A D / BRUGGER L / FELIACHI A / KELLER C / KRIEGER M P / LONGUET D / NEMOUCHI Y / TUONG F / WOLFF B D Unité Mixte de Recherche 8623 E CNRS-Université Paris Sud-LRI 04/2016 R Rapport de Recherche N° 1586 E C H E R C CNRS – Université de Paris Sud H Centre d’Orsay LABORATOIRE DE RECHERCHE EN INFORMATIQUE E Bâtiment 650 91405 ORSAY Cedex (France) HOL-TestGen 1.8.0 User Guide http://www.brucker.ch/projects/hol-testgen/ Achim D. Brucker Lukas Brügger Abderrahmane Feliachi Chantal Keller Matthias P.
  • Exploring Languages with Interpreters and Functional Programming Chapter 11

    Exploring Languages with Interpreters and Functional Programming Chapter 11

    Exploring Languages with Interpreters and Functional Programming Chapter 11 H. Conrad Cunningham 24 January 2019 Contents 11 Software Testing Concepts 2 11.1 Chapter Introduction . .2 11.2 Software Requirements Specification . .2 11.3 What is Software Testing? . .3 11.4 Goals of Testing . .3 11.5 Dimensions of Testing . .3 11.5.1 Testing levels . .4 11.5.2 Testing methods . .6 11.5.2.1 Black-box testing . .6 11.5.2.2 White-box testing . .8 11.5.2.3 Gray-box testing . .9 11.5.2.4 Ad hoc testing . .9 11.5.3 Testing types . .9 11.5.4 Combining levels, methods, and types . 10 11.6 Aside: Test-Driven Development . 10 11.7 Principles for Test Automation . 12 11.8 What Next? . 15 11.9 Exercises . 15 11.10Acknowledgements . 15 11.11References . 16 11.12Terms and Concepts . 17 Copyright (C) 2018, H. Conrad Cunningham Professor of Computer and Information Science University of Mississippi 211 Weir Hall P.O. Box 1848 University, MS 38677 1 (662) 915-5358 Browser Advisory: The HTML version of this textbook requires a browser that supports the display of MathML. A good choice as of October 2018 is a recent version of Firefox from Mozilla. 2 11 Software Testing Concepts 11.1 Chapter Introduction The goal of this chapter is to survey the important concepts, terminology, and techniques of software testing in general. The next chapter illustrates these techniques by manually constructing test scripts for Haskell functions and modules. 11.2 Software Requirements Specification The purpose of a software development project is to meet particular needs and expectations of the project’s stakeholders.
  • Beginner's Luck

    Beginner's Luck

    Beginner’s Luck A Language for Property-Based Generators Leonidas Lampropoulos1 Diane Gallois-Wong2,3 Cat˘ alin˘ Hrit¸cu2 John Hughes4 Benjamin C. Pierce1 Li-yao Xia2,3 1University of Pennsylvania, USA 2INRIA Paris, France 3ENS Paris, France 4Chalmers University, Sweden Abstract An appealing feature of QuickCheck is that it offers a library Property-based random testing a` la QuickCheck requires build- of property combinators resembling standard logical operators. For ing efficient generators for well-distributed random data satisfying example, a property of the form p ==> q, built using the impli- complex logical predicates, but writing these generators can be dif- cation combinator ==>, will be tested automatically by generating ficult and error prone. We propose a domain-specific language in valuations (assignments of random values, of appropriate type, to which generators are conveniently expressed by decorating pred- the free variables of p and q), discarding those valuations that fail icates with lightweight annotations to control both the distribu- to satisfy p, and checking whether any of the ones that remain are tion of generated values and the amount of constraint solving that counterexamples to q. happens before each variable is instantiated. This language, called QuickCheck users soon learn that this default generate-and-test Luck, makes generators easier to write, read, and maintain. approach sometimes does not give satisfactory results. In particular, We give Luck a formal semantics and prove several fundamen- if the precondition p is satisfied by relatively few values of the ap- tal properties, including the soundness and completeness of random propriate type, then most of the random inputs that QuickCheck generation with respect to a standard predicate semantics.
  • Symstra: a Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution

    Symstra: a Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution

    Symstra: A Framework for Generating Object-Oriented Unit Tests using Symbolic Execution Tao Xie1, Darko Marinov2, Wolfram Schulte3, David Notkin1 1 Dept. of Computer Science & Engineering, Univ. of Washington, Seattle, WA 98195, USA 2 Department of Computer Science, University of Illinois, Urbana-Champaign, IL 61801, USA 3 Microsoft Research, One Microsoft Way, Redmond, WA 98052, USA {taoxie,notkin}@cs.washington.edu, [email protected], [email protected] Abstract. Object-oriented unit tests consist of sequences of method invocations. Behavior of an invocation depends on the method’s arguments and the state of the receiver at the beginning of the invocation. Correspondingly, generating unit tests involves two tasks: generating method sequences that build relevant receiver- object states and generating relevant method arguments. This paper proposes Symstra, a framework that achieves both test generation tasks using symbolic execution of method sequences with symbolic arguments. The paper defines sym- bolic states of object-oriented programs and novel comparisons of states. Given a set of methods from the class under test and a bound on the length of sequences, Symstra systematically explores the object-state space of the class and prunes this exploration based on the state comparisons. Experimental results show that Symstra generates unit tests that achieve higher branch coverage faster than the existing test-generation techniques based on concrete method arguments. 1 Introduction Object-oriented unit tests are programs that test classes. Each test case consists of a fixed sequence of method invocations with fixed arguments that explores a particular aspect of the behavior of the class under test. Unit tests are becoming an important component of software development.
  • Quickcheck Tutorial

    Quickcheck Tutorial

    QuickCheck Tutorial Thomas Arts John Hughes Quviq AB Queues Erlang contains a queue data structure (see stdlib documentation) We want to test that these queues behave as expected What is “expected” behaviour? We have a mental model of queues that the software should conform to. Erlang Exchange 2008 © Quviq AB 2 Queue Mental model of a fifo queue …… first last …… Remove from head Insert at tail / rear Erlang Exchange 2008 © Quviq AB 3 Queue Traditional test cases could look like: We want to check for arbitrary elements that Q0 = queue:new(), if we add an element, Q1 = queue:cons(1,Q0), it's there. 1 = queue:last(Q1). Q0 = queue:new(), We want to check for Q1 = queue:cons(8,Q0), arbitrary queues that last added element is Q2 = queue:cons(0,Q1), "last" 0 = queue:last(Q2), Property is like an abstraction of a test case Erlang Exchange 2008 © Quviq AB 4 QuickCheck property We want to know that for any element, when we add it, it's there prop_itsthere() -> ?FORALL(I,int(), I == queue:last( queue:cons(I, queue:new()))). Erlang Exchange 2008 © Quviq AB 5 QuickCheck property We want to know that for any element, when we add it, it's there prop_itsthere() -> ?FORALL(I,int(), I == queue:last( queue:cons(I, queue:new()))). This is a property Test cases are generasted from such properties Erlang Exchange 2008 © Quviq AB 6 QuickCheck property We want to know that for any element, when we add it, it's there int() is a generator for random integers prop_itsthere() -> ?FORALL(I,int(), I == queue:last( queue:cons(I, I represents one queue:new()))).
  • Branching Processes for Quickcheck Generators (Extended Version)

    Branching Processes for Quickcheck Generators (Extended Version)

    Branching Processes for QuickCheck Generators (extended version) Agustín Mista Alejandro Russo John Hughes Universidad Nacional de Rosario Chalmers University of Technology Chalmers University of Technology Rosario, Argentina Gothenburg, Sweden Gothenburg, Sweden [email protected] [email protected] [email protected] Abstract random test data generators or QuickCheck generators for short. In QuickCheck (or, more generally, random testing), it is chal- The generation of test cases is guided by the types involved in lenging to control random data generators’ distributions— the testing properties. It defines default generators for many specially when it comes to user-defined algebraic data types built-in types like booleans, integers, and lists. However, when (ADT). In this paper, we adapt results from an area of mathe- it comes to user-defined ADTs, developers are usually required matics known as branching processes, and show how they help to specify the generation process. The difficulty is, however, to analytically predict (at compile-time) the expected number that it might become intricate to define generators so that they of generated constructors, even in the presence of mutually result in a suitable distribution or enforce data invariants. recursive or composite ADTs. Using our probabilistic formu- The state-of-the-art tools to derive generators for user- las, we design heuristics capable of automatically adjusting defined ADTs can be classified based on the automation level probabilities in order to synthesize generators which distribu- as well as the sort of invariants enforced at the data genera- tions are aligned with users’ demands. We provide a Haskell tion phase. QuickCheck and SmallCheck [27] (a tool for writing implementation of our mechanism in a tool called generators that synthesize small test cases) use type-driven and perform case studies with real-world applications.
  • Foundational Property-Based Testing

    Foundational Property-Based Testing

    Foundational Property-Based Testing Zoe Paraskevopoulou1,2 Cat˘ alin˘ Hrit¸cu1 Maxime Den´ es` 1 Leonidas Lampropoulos3 Benjamin C. Pierce3 1Inria Paris-Rocquencourt 2ENS Cachan 3University of Pennsylvania Abstract Integrating property-based testing with a proof assistant creates an interesting opportunity: reusable or tricky testing code can be formally verified using the proof assistant itself. In this work we introduce a novel methodology for formally verified property-based testing and implement it as a foundational verification framework for QuickChick, a port of QuickCheck to Coq. Our frame- work enables one to verify that the executable testing code is testing the right Coq property. To make verification tractable, we provide a systematic way for reasoning about the set of outcomes a random data generator can produce with non-zero probability, while abstracting away from the actual probabilities. Our framework is firmly grounded in a fully verified implementation of QuickChick itself, using the same underlying verification methodology. We also apply this methodology to a complex case study on testing an information-flow control abstract machine, demonstrating that our verification methodology is modular and scalable and that it requires minimal changes to existing code. 1 Introduction Property-based testing (PBT) allows programmers to capture informal conjectures about their code as executable specifications and to thoroughly test these conjectures on a large number of inputs, usually randomly generated. When a counterexample is found it is shrunk to a minimal one, which is displayed to the programmer. The original Haskell QuickCheck [19], the first popular PBT tool, has inspired ports to every mainstream programming language and led to a growing body of continuing research [6,18,26,35,39] and industrial interest [2,33].