Systematic Review of State Based Model Based Testing Tools
Total Page:16
File Type:pdf, Size:1020Kb
Systematic Review of State Based Model Based Testing Tools By Muhammad Shafique A thesis submitted to the Faculty of Graduate Studies and Research in partial fulfillment of the requirements for the degree of Master of Applied Science in Electrical and Computer Engineering Ottawa-Carleton Institute of Electrical and Computer Engineering (OCIECE) Department of Systems and Computer Engineering Carleton University Ottawa, Ontario, Canada June 2010 Copyright © 2010 by Muhammad Shafique Library and Archives BibliothSque et 1*1 Canada Archives Canada Published Heritage Direction du Branch Patrimoine de I'gdition 395 Wellington Street 395, rue Wellington OttawaONK1A0N4 OttawaONK1A0N4 Canada Canada Your file Votre reference ISBN: 978-0-494-71563-5 Our file Notre reference ISBN: 978-0-494-71563-5 NOTICE: AVIS: The author has granted a non L'auteur a accorde une licence non exclusive exclusive license allowing Library and permettant a la Biblioth&que et Archives Archives Canada to reproduce, Canada de reproduire, publier, archiver, publish, archive, preserve, conserve, sauvegarder, conserves transmettre au public communicate to the public by par telecommunication ou par Tlnternet, preter, telecommunication or on the Internet, distribuer et vendre des theses partout dans le loan, distribute and sell theses monde, a des fins commerciales ou autres, sur worldwide, for commercial or non support microforme, papier, electronique et/ou commercial purposes, in microform, autres formats. paper, electronic and/or any other formats. The author retains copyright L'auteur conserve la propriete du droit d'auteur ownership and moral rights in this et des droits moraux qui protege cette these. Ni thesis. Neither the thesis nor la these ni des extraits substantiels de celle-ci substantial extracts from it may be ne doivent etre imprimes ou autrement printed or otherwise reproduced reproduits sans son autorisation. without the author's permission. In compliance with the Canadian Conformement a la loi canadienne sur la Privacy Act some supporting forms protection de la vie privee, quelques may have been removed from this formulaires secondaires ont ete enleves de thesis. cette these. While these forms may be included Bien que ces formulaires aient inclus dans in the document page count, their la pagination, il n'y aura aucun contenu removal does not represent any loss manquant. of content from the thesis. •+• Canada CuJ iii The undersigned recommend to the Faculty of Graduate Studies and Research acceptance of the thesis Systematic Review of State Based Model Based Testing Tools submitted by Muhammad Shafique in partial fulfillment of the requirements for the degree of Master of Applied Science Professor Y. Labiche (Thesis Supervisor) Professor H. M. Schwartz (Department Chair) Carleton University June 2010 iv ACKNOWLEDGEMENTS First and foremost, I offer sincere thanks to my supervisor Dr. Yvan Labiche for all his guidance, patience and knowledge throughout my research. This thesis would not have been completed without his encouragement. I gained a lot of confidence from scheduled as well as casual research and technical discussions with him. His timely feedbacks kept me on right track throughout my research work at SQUALL. I also thank to my family and friends for their help and encouragement. Their constant moral support and kindness proved a vital catalyst. Last but not least, I would like to thank all my colleagues at SQUALL for their cooperation and support. I also learnt from them. Thank you all for cheering and caring me. V ABSTRACT Model-based testing (MBT) is about testing a software system by using a test model of its behaviour. MBT helps in early fault detection and facilitates the automation of many testing activities such as regression testing. To get proper benefits from MBT, (ideally full) automation support is required. This thesis presents a systematic review of prominent MBT tool support where mainly we focus on tools from industry and research which use different state/transition based models. The systematic review protocol describes the scope of this work and the steps involved in tool selection. Evaluation criteria are used to compare selected tool supports and comprise support for test coverage criteria, level of automation for various testing activities, and support for the construction of test scaffolding. The results of this systematic review should be of interest to a wide range of stakeholders. First, results should help software companies in selecting the most appropriate model based testing tool for their needs. Second, results should help organizations willing to invest into creating MBT tool support by identifying how to compete with existing support. Third, this study should also be helpful in setting directions of research in MBT since researchers should easily identify gaps between research activities and existing commercial tool support. VI Table of Contents Abstract v 1 INTRODUCTION 1 2 MODEL BASED TESTING 4 2.1 Software Model 4 2.2 Model-Based Testing Process 5 2.2.1 Build the mental model 5 2.2.2 Build the model 7 2.2.3 Generate abstract test cases 8 2.2.4 Execute test cases 8 2.2.5 Analyse test results 9 2.3 Benefits of Model-Based Testing 9 2.4 Limitations of Model-Based Testing 12 2.5 Issues and Future Challenges 13 3 SYSTEMATIC REVIEW PROTOCOL 16 3.1 Scope, Objectives, and Research Questions 16 3.2 Search Process 17 3.3 Inclusion and Exclusion Criteria 18 3.4 Search Strings 18 3.4.1 Population Terms 19 3.4.2 Intervention Terms 1 19 3.4.3 Intervention Terms2 20 3.5 Statistics gained from Search 20 4 SYSTEMATIC REVIEW CRITERIA 22 4.1 Types of Notations 22 4.2 Test Criteria 23 4.2.1 Model Flow Coverage Criteria 24 4.2.2 Script Flow Coverage Criteria 25 4.2.3 Data and Requirements Coverage Criteria 27 4.3 Tool Automation Criteria 28 4.3.1 Model Creation 28 4.3.2 Model Verification 29 4.3.3 Sub-models 29 4.3.4 Test Case Generation 29 4.3.5 Test Case Execution 29 4.3.6 Test Case Debugging 30 4.3.7 Requirements Traceability 30 4.3.8 Regression Testing 30 4.4 Test Scaffolding Criteria 31 4.4.1 Test Adapter Creation 31 4.4.2 Test Oracle Automation 32 vii 4.4.3 Stub creation 32 4.4.4 Offline Testing 32 4.4.5 Online Testing 33 4.5 Miscellaneous Criteria 33 4.5.1 Modeling Notations 33 4.5.2 Tool Category 33 4.5.3 Software Domain 34 4.5.4 Target Platform 34 5 EVALUATION 35 5.1 Tools 35 5.1.1 GOTCHA-TCBeans 35 5.1.2 mbt 37 5.1.3 MOTES 40 5.1.4 NModel 41 5.1.5 TestOptimal 43 5.1.6 Spec Explorer 2010 45 5.1.7 AGEDIS 47 5.1.8 ParTeG 50 5.1.9 Qtronic 51 5.1.10 Test Designer 52 5.1.11 Rhapsody Automatic Test Generation 54 5.1.12 Rational SDL and TTCN Suite 55 5.2 Results 57 5.2.1 Comparison 1: Test Criteria 57 5.2.2 Comparison 2: Tool Automation Criteria 62 5.2.3 Comparison 3: Test Scaffolding Criteria 64 5.2.4 Comparison 4: Miscellaneous Criteria 65 6 RELATED WORK 67 7 VALIDITY OF RESULTS 68 7.1 Conclusion validity 68 7.2 Construct validity 69 7.3 External validity 69 8 CONCLUSION AND FUTURE WORK 70 REFERENCES 72 Appendix A Models for MBT 79 A.l Finite State Machine 79 A.2 Extended Finite State Machine 80 A.3 Abstract State Machine 81 A.4 Simulink/Stateflow Chart Machine 82 A.5 UML State Machine 85 A.6 Harel Statecharts 89 A.7 Labeled Transition System 89 viii Appendix B Search Strings 92 B.l ACM Digital Library 92 B.2 IEEE Xplore Digital Library and SpringerLink 93 B.3 ELSEVIER 94 B.4 Wiley InterScience 94 B.5 EI Engineering Village 94 B.6 CiteseerX 94 B.7 Google and Google Scholar 94 IX List of Figures Figure 1 - Finite State Machine [79] 80 Figure 2 - Stateflow chart [17] 83 Figure 3 - A Simple UML state machine with start and end states 86 Figure 4 - A UML state machine with nested, history states and parallel regions 88 Figure 5 - Labeled Transition Systems of a sample Candy Machine, from [93] 90 Figure 6 - Labeled Transition Systems of sample Candy Machine with inputs and outputs 91 X List of Tables Table 1 - State-based AC (adequacy criteria) comparison 59 Table 2 - Script flow (adequacy and coverage) comparison 61 Table 3 - Data and requirements coverage comparisons 62 Table 4 - Automation coverage comparison 64 Table 5 - Test scaffolding criteria comparison 65 Table 6 - Miscellaneous criteria comparison 66 1 1 INTRODUCTION Software testing, that is evaluating software by observing its executions on actual valued inputs [46], is probably the most widely used verification and validation technique. Software testing techniques are usually characterized as being black-box (or functional) or white-box (or structural) depending on whether they rely solely on a specification of the software under test (SUT) or solely on its implementation [48]. Authors also introduce the notion of grey-box testing that mixes information from the specification and the implementation to derive tests [89]. Another term that is more and more used in textbooks and articles is model-based testing (MBT) [5, 7, 17]. Utting and Legeard recognize that this term has different acceptations and they focus on one which is the automation of the design of black-box tests [99]. In this thesis we consider a slightly broader definition and consider model-based testing as the support of one or more (the more the merrier) software testing activities from a model of the SUT behaviour.