ARTICLE IN PRESS JID: JSS [m5G; June 16, 2017;14:57 ] The Journal of Systems and Software 0 0 0 (2017) 1–14 Contents lists available at ScienceDirect The Journal of Systems and Software journal homepage: www.elsevier.com/locate/jss Contract-based testing for PHP with Praspel ∗ Frederic Dadeau a, , Alain Giorgetti a, Fabrice Bouquet a, Ivan Enderlin b a FEMTO-ST Institute, Univ. Bourgogne Franche-Comté, CNRS, Besançon, France b Liip.ch, rue Etraz 4, Lausanne, Switzerland a r t i c l e i n f o a b s t r a c t Article history: We summarize several contributions related to the PHP Realistic Annotation and SPEcification Language Received 21 July 2016 (Praspel). This language extends PHP programs with annotations for the formal specification of the be- Revised 19 May 2017 havior of their functions and for the declaration of types for their data. These contracts are used to au- Accepted 8 June 2017 tomate test generation, by deriving test cases and test data, and test execution, by checking assertions Available online xxx at run-time in order to establish the test verdict. Our approach to contract-based testing for PHP is fully Keywords: implemented into a PHP framework currently in use by several web companies. PHP ©2017 Published by Elsevier Inc. Design by Contract Annotations Test generation 1. Introduction from the model) into executable test scripts (using the API pro- vided by the SUT and the concrete data on which it operates). Each Model-Based Testing (MBT for short) ( Beizer, 1995 ) is a tech- evolution of the API or model can also impact this concretization nique in which a model of the System Under Test (SUT) is em- step. ployed in order to validate it by using tests. In this context, the Annotation languages provide an interesting solution to address model can be of two uses. First, it can be used to compute the these issues. Their underlying paradigm consists of inserting the test cases by providing an exploitable abstraction of the SUT from model as annotations (i.e. comments) in the code, expressing for- which test data or even test sequences can be computed. Second, mal assertions about it, namely: (i ) invariants , which are properties the model can provide the test oracle, namely the expected result that should hold at each execution step, and (ii ) pre- and postcon- against which the execution of the SUT is compared. MBT makes ditions , which are conditions that respectively have to hold for a it possible to automate the test generation, and, with an appropri- method to be invoked and after its execution. These assertions es- ate concretization layer, the test execution and verdict assignment tablish a contract between code developers and their users. There- phases. It is implemented in many tools, based on various model- fore, behavioral interface specification languages for these annota- ing languages or notations ( Utting et al., 2012 ). tions are also called contract languages. Design by Contract (DbC for Although MBT is a good way to promote formal methods among short) ( Meyer, 1992 ) was first introduced by B. Meyer with the Eif- developers, its application is often restricted to critical and/or em- fel language ( Meyer, 1987 ). Various contract languages extend pop- bedded software that requires a strong level of validation ( Zander ular programming languages, such as JML for Java ( Leavens et al., et al., 2011 ). Three main causes can be identified. First, the de- 1999 ) or ACSL for C ( Baudin et al., 2013 ). Annotation-based DbC is sign of the formal model represents an additional cost, sometimes thus a way to introduce lightweight formal methods in the process much higher than the manual validation cost. Second, the model of development of an application, and the syntactic proximity be- design is a complex task that requires modeling skills and under- tween the code and its model makes its writing easier to learn and standing of formal methods, thus requiring skilled validation en- to practice by developers, even if they are not familiar with formal gineers to be put into practice. Finally, the production of the test methods. scripts adds the additional effort and cost of the test concretiza- One of the main interests and uses of contracts is for testing. In- tion. Since the model is an abstraction of the SUT, a test concretiza- deed, Contract-Based Testing (CBT for short) ( Aichernig, 2003 ) has tion layer is required to translate abstract test cases (computed been introduced to exploit annotation languages for program test- ing: ∗ Corresponding author. E-mail addresses: [email protected] , [email protected]. (i) The information contained in invariants and preconditions fr (F. Dadeau). can be used to generate test cases. http://dx.doi.org/10.1016/j.jss.2017.06.017 0164-1212/© 2017 Published by Elsevier Inc. Please cite this article as: F. Dadeau et al., Contract-based testing for PHP with Praspel, The Journal of Systems and Software (2017), http://dx.doi.org/10.1016/j.jss.2017.06.017 ARTICLE IN PRESS JID: JSS [m5G; June 16, 2017;14:57 ] 2 F. Dadeau et al. / The Journal of Systems and Software 0 0 0 (2017) 1–14 (ii) Postconditions provide a test oracle for free , by assertion checking at run-time: the test succeeds if no assertion is vi- olated, and fails otherwise. (iii) Due to the proximity between the code and the model, there is no abstraction gap to bridge, contrary to other MBT approaches which require concretization of the ab- stract test cases, and establish the test verdict on the implementation. We present a contract-based testing approach for PHP, which has been fully integrated in a tool-set and made available to the PHP community. This approach uses Praspel, a specification lan- guage for PHP ( Enderlin et al., 2011 ) that relies on realistic domains . These domains make it possible to assign a domain of values to data appearing either as class attributes or method parameters. Realistic domains present two useful features for testing: predica- bility , which is the possibility to check whether a datum belongs to its associated domain, and samplability , which is the possibil- ity to automatically generate a datum from a realistic domain. A standard library of predefined realistic domains (mainly scalar do- mains, strings and arrays) is available along with an integrated test environment. Our approach aims at automating test generation, test execution Fig. 1. Praspel contract clauses. and test verdict assignment, especially in the domain of web appli- cations. Automated test generation is guided by a dedicated set of coverage criteria. They are based on the contracts’ structure, in or- der to activate the behaviors described by the contract. Test data are produced by various effective data generators, dedicated to the kinds of data most frequently used in web applications. Finally, test verdict assignment is ensured by an automated runtime assertion , ∗, checking mechanism that is used to detect non-conformances be- by s. r can be ? + or respectively for 0 or 1 time, 1 or more tween the code and the specification. times and 0 or more times. If s is not empty, then s must be a This article focuses on the following contributions: token. The rules for attribute-clauses and method-clauses in Fig. 1 re- • The definition of test selection criteria, that can be used either spectively define clauses annotating a class and a method. (The to generate test cases, even parameterized test cases, or to as- syntactic entity method-clauses shows the order of clauses in nor- sess the relevance of a test suite, mal form, but the actual Praspel grammar accepts clauses in • The use of contracts as a test oracle, and any order.) Invariants, behaviors, preconditions, normal and ex- • An experiment with independent software testing engineers, to ceptional postconditions are respectively declared by the key- identify the benefits of contract-based testing versus manual word @invariant , @behavior , @requires , @ensures and testing. @throwable . The syntax and semantics of these clauses are stan- This article is organized as follows. Section 2 introduces the dard (see Enderlin et al., 2011 for details and short examples). We syntax and semantics of contracts in Praspel, to specify class in- emphasize here only those language features added to the version variants and method behaviors. Section 3 presents the contract- of Praspel presented in Enderlin et al. (2011) . The first two are the based testing approach we propose, based on contract coverage @description clause to write an informal description of a be- criteria, and the runtime assertion checking used to provide the havior, and the @default behavior, whose implicit precondition test oracle and automatically assign a test verdict. Experimental is the conjunction of negations of all other behavior preconditions. results of real-world usage of Praspel are reported in Section 4 . Fig. 2 describes the syntax of Praspel expressions. They are Section 5 presents related work. Finally, Section 6 concludes and composed of declarations, predicates, and array conditions, respec- presents some future work. tively detailed in Sections 2.2, 2.3 and 2.5 . The simplest declaration is of the form v : r, where v is a variable and r is a realistic domain , 2. Praspel contracts a key notion for testing summarized in Section 2.1 and exemplified in Section 2.4 . This section summarizes the syntax and semantics of Praspel Fig. 3 shows the running example of a contract on a method ( PHP Realistic Annotation and SPEcification Language ). that manipulates an IRC stream. The method compute has two As an annotation language, Praspel is written in the source code arguments: server which is a class of kind \ Irc \ Server , and of the software, more precisely in PHP comments.