Automatic Test Generation Based on Formal Speci¯cations Practical Procedures for E±cient State Space Exploration and Improved Representation of Test Cases Dissertation zur Erlangung des Doktorgrades der Mathematisch-Naturwissenschaftlichen FakultÄaten der Georg-August-UniversitÄat zu GÄottingen vorgelegt von Michael Schmitt aus Orsoy GÄottingen 2003 D 7 Referent: Prof. Dr. Dieter Hogrefe Korreferent: Prof. Dr. Jens Grabowski Tag der mundlichenÄ Prufung:Ä 3. April 2003 Acknowledgments I would like to thank my supervisor Prof. Dr. Hogrefe for his assistance and the pos- sibility to research under perfect conditions. The work at the Institute for Telematics in LubÄ eck has been a very positive experience. I appreciate that I was allowed to act independently and to take on responsibility in a trustful atmosphere. I would also like to thank my former colleagues in LubÄ eck who have accompanied me through the years. Special thanks go to Jens Grabowski and Beat Koch with whom I worked together in the Autolink project and who shared endless discussions and debates on testing and test generation with me. This thesis would not be in its current state without the numerous proof-readers who spent their spare time on ¯nding typing errors, missing indexes, etc. and who gave helpful hints at how to improve my dissertation. I thank Zhen Ru Dai, Michael Ebner, Andreas Heuer, Maeve HÄolscher, Helmut Neukirchen, Cornelia Rieckho®, and Uwe Roth for their e®orts. Another major contribution to this dissertation was made by my parents who not only conceived me but also encouraged me to study computer science. This thesis is dedicated to Susanne. Thanks to her support and care, this dissertation has become reality. I hope that I will ever be able to return what she has done for me. Contents 1 Introduction 1 2 Foundations of Testing 5 2.1 Classi¯cation of Tests . 5 2.2 Conformance Testing Concepts . 9 2.2.1 Conformance Requirements . 9 2.2.2 Test Cases . 10 2.2.3 Test Verdicts . 11 2.2.4 Test Suites . 11 2.3 The Conformance Testing Process . 12 2.3.1 Test Suite Development . 12 2.3.2 Test Preparation . 12 2.3.3 Test Operation . 13 2.3.4 Test Evaluation . 13 2.4 Test Methods and Con¯gurations . 14 3 Test Languages 17 3.1 The Tree and Tabular Combined Notation . 19 3.1.1 Test Suite Overview and Import Part . 20 3.1.2 Declarations Part . 21 3.1.3 Constraints Part . 24 3.1.4 Dynamic Part . 26 3.2 The Testing and Test Control Notation . 30 3.2.1 Modules and Groups . 30 3.2.2 Data Model . 31 3.2.3 Communication . 32 3.2.4 Test Con¯gurations . 34 3.2.5 Templates . 36 3.2.6 Behavior Descriptions . 37 3.2.7 Development Tools . 40 3.3 Discussion . 40 4 Test Generation Based on Formal Speci¯cations 45 4.1 Formal Methods in Conformance Testing . 46 4.1.1 Speci¯cation and Implementation . 46 4.1.2 Static and Dynamic Conformance . 46 i 4.1.3 Testing Concepts . 47 4.1.4 Test Generation . 49 4.2 Test Generation Methods . 49 4.2.1 Fault Models . 50 4.2.2 Test Coverage Criteria . 53 4.2.3 Scenario-Based Requirements . 59 4.3 Test Generation, Veri¯cation, and Validation . 60 5 High-Level Speci¯cation Languages 63 5.1 Message Sequence Chart . 63 5.1.1 Basic Message Sequence Charts . 64 5.1.2 Data Model . 67 5.1.3 Structural Concepts . 67 5.1.4 High-Level Message Sequence Charts . 68 5.2 Speci¯cation and Description Language . 70 5.2.1 Agents and Structuring . 70 5.2.2 Communication . 72 5.2.3 Behavior . 72 5.2.4 Object Orientation . 75 5.2.5 SDL Data Model and ASN.1 . 75 5.2.6 Further Language Constructs . 76 5.3 Tool Support . 76 6 The Autolink Tool 79 6.1 The Autolink Test Generation Process . 80 6.2 Test Purpose Speci¯cation . 80 6.2.1 Manual Speci¯cation . 82 6.2.2 Interactive Simulation . 82 6.2.3 Observer Processes . 83 6.2.4 Automatic Computation . 84 6.3 Test Case Generation . 85 6.3.1 State Space Exploration . 85 6.3.2 Direct Translation . 87 6.4 Test Suite Production . 87 6.5 Interpretation of MSC Test Purposes . 89 6.5.1 Partial Order Semantics . 89 6.5.2 Structuring Concepts . 90 6.6 Case Studies . 94 6.6.1 Core INAP CS-2 . 94 6.6.2 VB5.1 and VB5.2 . 97 6.7 Comparison with Other SDL-based Test Tools . 98 6.7.1 SaMsTaG . 98 6.7.2 TestComposer . 99 6.8 Discussion . 100 ii 7 Test Generation for Distributed Test Architectures 103 7.1 Concurrency in TTCN-2 . 103 7.2 De¯nition of Test Component Con¯gurations . 105 7.3 Synchronization of Test Components . 106 7.3.1 Implicit Synchronization . 106 7.3.2 Explicit Synchronization . 106 7.4 A Test Generation Procedure . 110 7.4.1 Simulator Requirements . 111 7.4.2 Test Generation for MTC and PTCs . 112 7.5 Case Studies . 115 8 The Tree Walk Search Strategy 119 8.1 Classical Search Strategies . 120 8.2 Labeled Transition Systems . 121 8.3 Main Concepts . 122 8.3.1 Root States . 123 8.3.2 Algorithmic Description . 124 8.4 Detection of Identical States . 128 8.4.1 Example . 131 8.4.2 Algorithmic Description . 131 8.5 Case Studies . 134 8.6 Discussion . 138 9 Test Suite Representation 141 9.1 The Autolink Script Language . 142 9.1.1 General Language Concepts . 143 9.1.2 Constraint Rules . 143 9.1.3 Test Suite Structure Rules . 149 9.2 Automatic Structuring of Constraint Descriptions . 153 9.2.1 Constraint Factorization . 156 9.2.2 Constraint Merging . 157 9.2.3 Constraint Parameterization . 158 9.2.4 Constraint Derivation . 161 9.2.5 Constraint Defactorization . 162 9.2.6 Case Studies . 163 9.3 A List Pattern Matching and Manipulation Language . 164 9.3.1 General Language Concepts . 167 9.3.2 Basic Patterns . 168 9.3.3 Variables and Variable Operators . 170 9.3.4 Manipulation Operators . 171 9.3.5 Extension Operator . 172 9.3.6 Search Operators . 172 9.4 Discussion . 173 iii 10 Advanced Test Generation by Symbolic Execution 177 10.1 Motivation . 178 10.2 Checking the Feasibility of Path Conditions . 180 10.3 The ValiBOSE Tool . 184 10.3.1 Navigation . ..