Model-Based Mutation Testing of an Industrial Measurement Device Bernhard K. Aichernig1, Jakob Auer2, Elisabeth Jöbstl1, Robert Korošec2, Willibald Krenn3, Rupert Schlick3, and Birgit Vera Schmidt2 1 Institute for Software Technology, Graz University of Technology, Austria {aichernig,joebstl}@ist.tugraz.at 2 AVL List GmbH, Graz, Austria {jakob.auer,robert.korosec,birgitvera.schmidt}@avl.com 3 AIT Austrian Institute of Technology, Vienna, Austria {willibald.krenn,rupert.schlick}@ait.ac.at Abstract. MoMuT::UML is a model-based mutation testing tool for UML models. It maps UML state machines to a formal semantics and performs a conformance check between an original and a set of mutated models to automatically generate test cases. The resulting test suite is able to detect whether a system under test implements one of the faulty models instead of the correct, original model. In this work, we illus- trate the whole model-based mutation testing process by means of an industrial case study. We test the control logic of a device that counts the particles in exhaust gases. First, we model the system under test in UML. Then, MoMuT::UML is used to automatically generate three test suites from the UML test model: one mutation-based test suite, one set of random test cases, and a third test suite combining random and mutation-based test case generation. The test cases are executed on the system under test and effectively reveal several errors. Finally, we com- pare the fault detection capabilities of the three test suites on a set of faulty systems, which were created by intentionally injecting faults into the implementation. Keywords: test case generation, model-based testing, mutation testing, automotive industry, UML 1 Introduction Testing of complex systems is a challenging and labour-intensive task. Approx- imately 50% of the elapsed time and costs of a software project are spent on testing [24]. Furthermore, the later a software error is detected, the higher are the costs for fixing it [18]. Hence, tools and techniques to assist testers are de- manded by industry. In this work, we present a formal approach to software testing and demonstrate its applicability in an industrial setting. Figure 1 gives an overview of our approach, which we refer to as model-based mutation testing. Yellow parts highlight the aspects of mutation testing that we integrate into model-based testing, which is depicted in grey. Model-based test- ing (MBT) is a black-box testing technique requiring no knowledge about the 2 B.K. Aichernig et al. mutation operators 1) model creation 1a) model requirements 1) model creation 1a) model and validation mutation and validation test mutation mutated model models SUT concrete abstract 2)2) testtest casecase generationgeneration 3)3) concretionconcretion (black-box) test cases test cases byby conformanceconformance checkcheck 4)4) testtest casecase test 5)5) analysisanalysis executionexecution results Fig. 1. Overview of model-based mutation testing. source code of the system under test (SUT). Only the interface to the SUT has to be known. A test engineer creates a formal model that describes the expected be- haviour of the SUT (Step 1). Test cases are then automatically derived from this test model. A crucial matter in MBT is the choice of the test criterion. It specifies which test cases shall be generated and hence, has a great influence on the qual- ity of the resulting test suite. Exhaustive testing, i.e., using all of the test cases that can possibly be created from the test model, is impractical. Examples for commonly used test criteria are coverage criteria, random traversals, equivalence classes, or specified testing scenarios (test purposes). We follow a fault-centred approach, i.e., use mutations for test case generation (TCG). We syntactically alter the original test model producing a set of mutated models (Step 1a). We then automatically generate test cases that kill the model mutants, i.e., reveal their non-conforming behaviour. This is accomplished by a conformance check between the original and the mutated models (Step 2). As the test model is an abstraction of the SUT, also the derived test cases are abstract. Hence, they have to be concretised, i.e., mapped to the level of detail of the SUT (Step 3). Finally, the concrete test cases can be executed on the SUT (Step 4) and the test results can be analysed (Step 5). A particular feature of the generated test suites is their fault coverage. The generated tests will detect whether a faulty model has been implemented instead of the correct, original model. Hence, the generated test suite covers all of the modelled faults expressed by the model mutation operators and has a high chance of covering many additional similar faults (cf. coupling effect [14]). Tool support for our model-based mutation testing approach is provided by the MoMuT::UML test case generator. It takes a UML model of the SUT, automatically creates mutated models, and subsequently uses these models for the automatic generation of abstract test cases. For model creation, we rely on external UML modelling tools like Visual Paradigm. The concretion and execution of the abstract test cases has also not been integrated in MoMuT::UML as these tasks highly depend on the SUT. We already presented and applied the model-based mutation testing approach previously [2]. However, this earlier work relied on an enumerative TCG engine. One contribution of this work is the application of a new and more efficient Model-Based Mutation Testing of an Industrial Measurement Device 3 TCG engine based on SMT solving techniques. Underlying research has already been presented in [3–5]. However, these earlier tool versions did not yet support the full language required for the UML approach and additionally used Prolog’s constraint solver instead of the SMT solver Z3. Hence, this is the first time that we apply our SMT-based TCG tool to a UML model. The main contribution is a comprehensive case study: we demonstrate the whole model-based mutation testing process on an industrial use case from the automotive domain incl. modelling, test generation, concretion, as well as exe- cution and analysis of the test results. Moreover, we evaluate the fault detection capability of the mutation-based tests on a set of faulty SUTs and compare it with random tests and a test suite combining random and mutation-based tests. The rest of this paper is structured as follows. Section 2 presents the SUT and describes how it has been modelled in UML. Section 3 deals with test case generation with MoMuT::UML and Section 4 reports on test execution. Finally, we discuss related work in Section 5 and conclude the paper in Section 6. 2 System under Test: a Particle Counter The SUT is a measurement device for the automotive domain produced by AVL1 which is used to measure particle number concentrations of diluted exhaust gas in compliance with UNECE-83 and PMP2. The particle counter consists of a conditioning component (volatile particle remover VPR) and the actual particle number counter (PNC). The VPR consists of the first dilution step, an evaporation tube, and a secondary dilution step. In order to count non-volatile particles, a pump draws the exhaust gas into a sampling probe which eliminates all particles >2.5 µm. The sampled exhaust gas is then diluted with cleaned hot air to stabilize the particle number concentra- tion. After the hot primary dilution, the diluted exhaust gas is further heated up in the evaporation tube in order to convert all volatile particles into the gaseous phase. Afterwards, a secondary dilution is performed to prevent further conden- sation or adsorption of volatile substances and to ensure that the maximum inlet temperature of the particle number counter (PNC) is not exceeded. Within the PNC the particles are enlarged due to the condensation of butanol and detected and counted using the light-scattering method. In this paper we are concerned with testing the control logic of the particle counter, which offers several different operation modes to the user. For exam- ple, the user can choose between continuously measuring the current particle concentration or accumulating the number of particles counted over a period of time. During the measurement, the ratio by which the exhaust gas is mixed with particle-free dilution air can be adjusted. Additionally, there is a command to measure pure, particle-free air to check whether the sensors are calibrated correctly. Other commands are provided for necessary maintenance tasks like a leakage test, a response check, or for purging the sampling line. 1 https://www.avl.com/particle-counter, 18.03.2014 2 http://www.unece.org/, 14.05.2014 State Pause_0 Active entry / SetPause / set not Manual body = send SPAU_state() to testEnvironment; exit / Timed / send StatusReady Activity2 Purging_Pause_12 body = 20 entry / send StatusBusy() to testEnvironment; SetPurge [not Busy and not Manual] body = send SPUL_state() to testEnvironment; Initial_top Busy = true; SetStandby / set not Manual Leakage_11 LeakageTest [not Busy and not Manual] entry / Pause_internal body = send SLEC_state() to testEnvironment; Initial_pause Purging_Standby_12 SetPause / set not Manual entry / body = send SPUL_state() to testEnvironment; Response_14 SetStandby [not Busy] / set not Manual SetPause / set not Manual entry / body = send SEGA_state() to testEnvironment; Standby_1 entry / body = send STBY_state() to testEnvironment; Timeless exit / 20 Activity3 Integral_9 body = entry / send StatusBusy() to testEnvironment;