A Method for Testing and Validating Executable Statechart Models
Total Page:16
File Type:pdf, Size:1020Kb
Software & Systems Modeling https://doi.org/10.1007/s10270-018-0676-3 THEME SECTION PAPER A method for testing and validating executable statechart models Tom Mens1 · Alexandre Decan1 · Nikolaos I. Spanoudakis2 Received: 29 August 2016 / Accepted: 11 April 2018 © Springer-Verlag GmbH Germany, part of Springer Nature 2018 Abstract Statecharts constitute an executable language for modelling event-based reactive systems. The essential complexity of statechart models solicits the need for advanced model testing and validation techniques. In this article, we propose a method aimed at enhancing statechart design with a range of techniques that have proven their usefulness to increase the quality and reliability of source code. The method is accompanied by a process that flexibly accommodates testing and validation techniques such as test-driven development, behaviour-driven development, design by contract, and property statecharts that check for violations of behavioural properties during statechart execution. The method is supported by the Sismic tool, an open-source statechart interpreter library in Python, which supports all the aforementioned techniques. Based on this tool- ing, we carry out a controlled user study to evaluate the feasibility, usefulness and adequacy of the proposed techniques for statechart testing and validation. Keywords Statechart · Executable modeling · Behaviour-driven development · Design by contract · Runtime verification 1 Introduction cial tools such as IBM Rational’s StateMate and Rhapsody, the Mathworks’ Stateflow, itemis’ Yakindu Statechart Tools Statecharts were introduced nearly three decades ago by and IAR Systems’ visualSTATE. There are also multiple Harel [27,28] as a visual executable modelling language. open-source frameworks defining domain-specific languages From a formal point of view, they can be considered as (DSLs) based on statecharts, such as the ATOMPM [47] web- an extension of hierarchical finite-state machines with char- based domain-specific modelling environment with support acteristics of both Mealy and Moore automata. Statecharts for model transformation, and the AMOLA [46] language (sometimes referred to as behavioural state machines)are targeting the multi-agent systems community. Most of these part of the UML standard [41] and constitute a popular mod- tools support visualisation, modification and simulation of elling notation for representing the executable behaviour statecharts, as well as code generation from statechart mod- of complex reactive event-based systems. They are fre- els. The more advanced tools also provide support for model quently used in industry for the development of real-time debugging and model verification. systems and embedded systems [13,44], relying on commer- At the level of source code development, a variety of test- ing techniques have found widespread use. These include Communicated by Dr. M. Papadakis, S. Ali, and G. Perrouin. test-driven development (TDD)[3], behaviour-driven devel- opment (BDD)[39] and design by contract (DbC)[38]. B Tom Mens Most of these techniques have not been taken up at design [email protected] level, however, for increasing the quality and reliability of Alexandre Decan behavioural software models. In particular, it is poorly under- [email protected] stood how such techniques can be used for testing and Nikolaos I. Spanoudakis validating executable statechart models. Indeed, designing [email protected] statecharts and their interaction with the environment can be 1 Software Engineering Lab, Faculty of Sciences, University of quite complex and error-prone, partly because of the stat- Mons, 7000 Mons, Belgium echart formalism itself, and partly because of the complex 2 Applied Mathematics and Computers Laboratory, School of behaviour that these statecharts are modelling [11]. Production Engineering and Management, Technical To address this problem, we propose a method aimed University of Crete, 73100 Chania, Greece to enhance statechart design with such techniques. The 123 T. Mens et al. method comes with dedicated tooling, in the form of Sismic User stories and BDD would provide a very good basis (a recursive acronym for “Sismic Interactive Statechart for coming up with relevant functional tests for executable Model Interpreter and Checker”), a modular Python library statecharts, assuming that developers are available to write composed of a statechart interpreter and associated libraries the necessary mapping code to execute the scenarios as func- for testing and validation of executable statecharts based on tional tests over the statechart. the techniques of TDD, BDD, DbC, and property statecharts that allow to monitor for violations of behavioural proper- ties during statechart execution. The library is provided as an 2.2 DbC open-source solution, in order to facilitate its extension by other researchers. It comes with support for importing and Design by contract (DbC) was introduced more than two exporting statechart models to and from external statechart decades ago by Bertrand Meyer and popularised through visualisation and editing tools. the object-oriented programming language Eiffel [19,38]. The remainder of this article is structured as follows. Sec- It is often used to support the interaction and composition tion 2 explains and motivates the techniques of TDD, BDD, of software components (e.g. methods, functions, classes or DbC and run-time monitoring. Section 3 presents the pro- packages) in complex systems based on the interface speci- posed method and process for statechart design and testing. fications of these components. DbC prescribes that software Section 4 explains the tooling that we have developed to sup- components should have formal, precise and verifiable inter- port all the techniques supported by the method. Sections 5 face specifications, which extend the ordinary definition of and 6 illustrate how to use the method and its associated tools abstract data types with preconditions, postconditions and by means of an example. Section 7 evaluates the method and invariants. The term contract stresses the obligation of the tools through a controlled user study. Section 8 compares our programmer to respect the conditions of the code she is work with other approaches, and Sect. 9 concludes. using. The use of DbC has been shown to increase the reliability 2 Background of executable code. It has been used with success to enable automated debugging with AutoFix, a tool that uses con- 2.1 TDD and BDD tracts to fix faults in general-purpose software [43]. Other programming languages also provide support for DbC. For Test-driven development (TDD)[3] has become accepted for Java programs for example, DbC is supported by JML, a source code development thanks to the many available frame- formal behavioural interface specification language [34]. works for automated unit testing. TDD is often accompanied OCL, the constraint language that is part of the UML by so-called user stories, representing informal, natural lan- standard, offers the possibility to specify preconditions, guage descriptions of one or more features of the software postconditions and invariants on UML models [40]. It system, written from an end user perspective. has been used to specify contracts on class diagrams The technique of behaviour-driven development (BDD) [6] supported by the USE tool [24] and has recently allows to bridge the gap between user stories and executable been extended to support behavioural models such as pro- functional tests. BDD extends TDD with acceptance test tocol state machines [26] and sequence diagrams [23]. or customer test-driven development practices as found in We are not aware, however, of any DbC tool that pro- extreme programming [39,50]. BDD allows users to spec- vides integrated support for specifying and checking struc- ify representative scenarios and their expected outcomes tural properties during the run-time execution of state- using Gherkin,1 a domain-specific natural language that has charts. been created specifically to support BDD. It enables users Just like contracts can be defined on source code com- to describe the intended software behaviour without need- ponents at different levels of granularity, it is useful and ing to detail how that behaviour is or will be implemented. desirable to express contracts at different levels on an exe- Scenarios and their outcomes are described as sequences cutable statechart. Contracts could be defined on individual of steps expressed as domain-specific natural language sen- states (either basic or composite states) to express how these tences using Gherkin-specific keywords such as Given, states are supposed to interact with other states. Contracts When, Then, And and But. Developers can then define could be expressed on concurrent regions of a state to spec- mapping code for these steps, in order to enable the automatic ify how these regions are supposed to synchronise with execution of each of these scenarios as executable functional each other. Finally, contracts could be specified on tran- tests. As such, the technical gap between developers and users sitions to verify that any action executed by a transition is reduced. satisfies some invariants. Once defined, contracts can be monitored continuously while the statechart is being exe- 1 See http://docs.behat.org/en/v2.5/guides/1.gherkin.html. cuted. 123 A method for testing and validating executable statechart models 2.3 Run-time verification Any mechanism for monitoring observable behaviours or properties