Cross-Checking Oracles from Intrinsic Software Redundancy Antonio Carzaniga Alberto Goffi University of Lugano University of Lugano Switzerland Switzerland [email protected] alberto.goffi@usi.ch Alessandra Gorla Andrea Mattavelli Mauro Pezzè Saarland University University of Lugano University of Lugano Germany Switzerland Switzerland [email protected] [email protected] University of Milano-Bicocca saarland.de Italy [email protected] ABSTRACT 1. INTRODUCTION Despite the recent advances in automatic test generation, Test oracles discriminate successful from failing executions testers must still write test oracles manually. If formal speci- of test cases. Good oracles combine simplicity, generality, fications are available, it might be possible to use decision and accuracy. Oracles should be simple to write and straight- procedures derived from those specifications. We present a forward to check, otherwise we would transform the problem technique that is based on a form of specification but also of testing the software system into the problem of testing leverages more information from the system under test. We the oracles. They should also be generally applicable to the assume that the system under test is somewhat redundant, widest possible range of test cases, in particular so that they in the sense that some operations are designed to behave can be used within automatically generated test suites. And like others but their executions are different. Our experience crucially, they should be accurate in revealing all the faulty in this and previous work indicates that this redundancy behaviors (completeness, no false negatives) and only the exists and is easily documented. We then generate oracles by faulty ones (soundness, no false positives). cross-checking the execution of a test with the same test in Test oracles are often written manually on a case-by-case which we replace some operations with redundant ones. We basis, commonly in the form of assertions, for example JUnit develop this notion of cross-checking oracles into a generic assertions.1 Such input-specific oracles are usually simple technique to automatically insert oracles into unit tests. An and effective but they lack generality. Writing such oracles for experimental evaluation shows that cross-checking oracles, large test suites and maintaining them through the evolution used in combination with automatic test generation tech- of the system can be expensive. Writing and maintaining niques, can be very effective in revealing faults, and that such oracles for large automatically generated test suites may they can even improve good hand-written test suites. be practically impossible. It is possible to also generate oracles automatically, even Categories and Subject Descriptors though research on test automation has focused mostly on supporting the testing process, creating scaffolding, managing D.2.4 [Software Engineering]: Software/Program Verifi- regression test suites, and generating and executing test cation; D.2.5 [Software Engineering]: Testing and Debug- cases, but much less on generating oracles [7, 27]. Most of ging the work on the automatic generation of oracles is based on some form of specification or model. Such oracles are very General Terms generic, since they simply check that the behavior of the system is consistent with the prescribed model. However, Verification their applicability and quality depend on the availability and completeness of the models. For example, specification- Keywords based oracles are extremely effective in the presence of precise Redundancy, test oracles, oracle generation specifications, such as protocol specifications [21], but they are not easily applicable to many other systems that come with informal and often incomplete specifications. Permission to make digital or hard copies of all or part of this work for personal or Another classic approach to obtain generic oracles is to use classroomPermission use to ismake granted digital without or fee hard provided copies that of copies all or are part not madeof this or workdistributed for what Weyuker calls a pseudo-oracle [46, 17], that is, another for profit or commercial advantage and that copies bear this notice and the full citation onpersonal the first or page. classroom Copyrights use for is components granted without of this work fee provided owned by thatothers copies than ACM are program intended to behave exactly as the original. The mustnot made be honored. or distributed Abstracting for with profit credit or is commercial permitted. To advantage copy otherwise, and that or republish, copies actual oracle requires the execution of the two programs to post on servers or to redistribute to lists, requires prior specific permission and/or a bear this notice and the full citation on the first page. To copy otherwise, to with the same input, followed by a comparison between fee.republish, Request to permissions post on servers from [email protected]. or to redistribute to lists, requires prior specific ICSE’14permission, May and/or 31 – a June fee. 7, 2014, Hyderabad, India the results of the two executions. The production of an CopyrightICSE ’14, May2014 31 ACM – June 978-1-4503-2756-5/14/05...$15.00 7, 2014, Hyderabad, India 1 http://dx.doi.org/10.1145/2568225.2568287Copyright 14 ACM 978-1-4503-2756-5/14/05 ...$15.00. http://junit.org 931 alternative version of the program makes this technique experimental evaluation that demonstrates that this method completely generic, but it also makes it quite expensive. can indeed lead to efficient and effective oracles. In summary, There are also interesting solutions somewhere in between we make the following contributions: model-based oracles and pseudo-oracles. For example, AS- TOOT [19], symmetric testing [25] and metamorphic test- • We introduce the notion of cross-checking oracles ob- ing [12] do not require complete behavioral models, and tained by exploiting the intrinsic redundancy of soft- instead exploit symmetries and equivalences in the specifi- ware systems and realized through the embedding and cation. ASTOOT derives tests and corresponding pseudo- coordinated execution of fragments of cross-checking oracles from algebraic specifications in the form of pairs of code. equivalent and non-equivalent sequences of operations. In • We develop a specific concrete technique that makes a similar way, metamorphic and symmetric testing use the cross-checking oracles practical and accurate. commutativity of some operations, or the symmetric behav- ior of an operation with different inputs, to identify different • We report the results of an experimental evaluation sets of inputs that should produce the same result. that shows that cross-checking oracles can be effective The testing methods proposed with these techniques are especially in conjunction with automatic test genera- interesting alternatives to both input-specific and completely tion. generic oracles. Differently from input-specific oracles, they are not limited to a fixed set of inputs. And differently from The paper is organized as follows. Section 2 introduces generic oracles, they do not attempt to compute or verify the concept of intrinsic redundancy for software, summarizes a result using an alternative program but instead use the the origins and scope of the phenomenon, and shows some system as both the original and the alternative program, cases of intrinsic redundancy in libraries at the method level. effectively making the system test itself. Section 3 details a technique to generate and deploy cross- We propose a similar method that is also rooted in the checking oracles obtained from specifications of redundancies idea of a pseudo-oracle. Specifically, we exploit the intrinsic in the system under test. Section 4 presents the experimental redundancy of software systems to generate what we call evaluation of this technique, and in particular shows that cross-checking oracles. Unlike other approaches, we use spec- cross-checking oracles complement state-of-the-art test case ifications that are closer to the code and therefore arguably generation tools such as Randoop. Section 5 surveys the simpler to obtain. We also decouple the generation of the main approaches to the generation of oracles with a special oracle from the test input, obtaining a more versatile method focus on those that are most related to this work. Section 6 to generate oracles and therefore a more effective testing summarizes the contributions of the paper and describes process. ongoing research work. In very simple terms, we build oracles by having the sys- tem under test cross-check itself. These cross-checks exploit 2. SOFTWARE REDUNDANCY the intrinsic redundancy of the system under test. This is a We obtain cross-checking oracles by exploiting the intrinsic property of various systems that we have studied and charac- redundancy of software systems. In this section we introduce terized [10] and that we have used to implement a self-healing the concept of intrinsic redundancy informally, and discuss mechanism for Web applications [9] and for general purpose some manifestations of this redundancy. The interested systems [8]. In this paper we describe a method to exploit reader can find more details about software redundancy and this intrinsic redundancy to generate oracles and enhance some of its uses elsewhere [10, 11, 24]. the efficacy of test suites. Informally, a system is redundant when it can perform As summarized in the next section, the intrinsic redun- the same actions through different executions,