An Open Framework for the Specification and Execution of a Testable Requirements Model
Total Page：16
File Type：pdf, Size：1020Kb
An Open Framework for the Specification and Execution of a Testable Requirements Model by David Arnold B.C.S., M.C.S. A Thesis submitted to the Faculty of Graduate Studies and Research in partial fulfillment of the requirements for the degree of Doctor of Philosophy Ottawa-Carleton Institute for Computer Science School of Computer Science Carleton University Ottawa, Ontario April, 2009 © Copyright 2009, David Arnold Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Library and Bibliotheque et 1*1 Archives Canada Archives Canada Published Heritage Direction du Branch Patrimoine de I'edition 395 Wellington Street 395, rue Wellington Ottawa ON K1A0N4 Ottawa ON K1A0N4 Canada Canada Your file Votre reference ISBN: 978-0-494-52057-4 Our file Notre reference ISBN: 978-0-494-52057-4 NOTICE: AVIS: The author has granted a non L'auteur a accorde une licence non exclusive exclusive license allowing Library permettant a la Bibliotheque et Archives and Archives Canada to reproduce, Canada de reproduire, publier, archiver, publish, archive, preserve, conserve, sauvegarder, conserver, transmettre au public communicate to the public by par telecommunication ou par Nntemet, preter, telecommunication or on the Internet, distribuer et vendre des theses partout dans loan, distribute and sell theses le monde, a des fins commerciales ou autres, worldwide, for commercial or non sur support microforme, papier, electronique commercial purposes, in microform, et/ou autres formats. paper, electronic and/or any other formats. The author retains copyright L'auteur conserve la propriete du droit d'auteur ownership and moral rights in et des droits moraux qui protege cette these. this thesis. Neither the thesis Ni la these ni des extraits substantiels de nor substantial extracts from it celle-ci ne doivent etre imprimes ou autrement may be printed or otherwise reproduits sans son autorisation. reproduced without the author's permission. In compliance with the Canadian Conformement a la loi canadienne Privacy Act some supporting sur la protection de la vie privee, forms may have been removed quelques formulaires secondaires from this thesis. ont ete enleves de cette these. While these forms may be included Bien que ces formulaires in the document page count, aient inclus dans la pagination, their removal does not represent il n'y aura aucun contenu manquant. any loss of content from the thesis. i*I Canada Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Abstract Bertolino [34] remarks: "The leading idea [in model-based testing] is to use models defined in software construction to drive the testing process, in particular to automatically generate the test cases." In practice, however, we observe that Model- Based Testing (MBT) typically suffers from a gap between such models and the Implementation Under Test (IUT). More precisely, a model contains a specification that must be transformed into test cases that can be executed by a specific IUT. Such transformation is seldom automated and thus the resulting executable test cases may no longer correspond to the ones obtained from the model. Furthermore, current MBT approaches generally compound this problem by not addressing the problem of test case instrumentation, in particular of the ones pertaining to scenario testing. In this work, we present a solution that eliminates both of these problems by automatically bridging from a Testable Requirements Model (TRM) to a concrete implementation. Through the course of our research we have designed and implemented a Validation Framework (VF) supporting the specification of a TRM using a text-based contract specification language grounded in the notion of responsibilities and scenarios. The VF compiles this requirements model along with a set of bindings that bridge the gap between model and implementation. The result is a model that is directly executed against the IUT, resulting in the production of a contract evaluation report. It is the operation of this framework and its application to the MBT domain that we address in this dissertation: specifically how our VF bridges the gap between a TRM and an IUT allowing for automated validation of the implementation against the model. ~ iii ~ Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Acknowledgements I am grateful for the encouragement, enthusiasm, and inspiration provided by my thesis supervisor: Jean-Pierre Corriveau. Without your help and support, the thesis and implementation would never have left the whiteboard in your office. Special thanks go to Alain Forget, my roommate these past five years: you served as a sounding board, helped me procrastinate, and made countless trips to Tim Hortons even though you don't even drink coffee. I am indebted to Carleton University, the Government of Ontario, and the Government of Canada for the monetary support provided during the course of my research. I definitely would not have been able to focus on my research without their contributions. My mother, my father, and my sister have always been a source of encouragement: they kept me focused on my work during tough times at home. I know they will never understand exactly what I do, but they will always be there to encourage me. Finally, during the course of this endeavor I met someone who has become very special to me. My thesis has indirectly allowed me to fall in love. I love you Mary, and appreciate all of the caring, compassion, and support you have provided. This thesis is dedicated to you. Thank you all. IV Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. Table of Contents LIST OF ACRONYMS VIII LIST OF FIGURES X LIST OF TABLES XII 1 INTRODUCTION 1 1.1 OVERVIEW OF THE PROBLEM 1 1.2 OVERVIEW OF OUR SOLUTION 6 1.2.1 Capturing a Requirements Model 6 1.2.2 Testable Models 7 1.2.3 Candidate Implementations 9 1.3 TESTABLE REQUIREMENTS MODELS 12 1.3.1 Requirements for a Testable Model 12 1.3.2 Scenarios and Responsibilities 14 1.3.3 Metric Capture and Analysis 19 1.3.4 Discussion 20 1.4 SCOPE OF OUR WORK 20 1.4.1 Validation Framework Input 23 1.4.2 Plug-ins 25 1.4.3 Compilation 27 1.4.4 Execution 28 1.4.5 Summary 28 1.5 VALIDATION OF OUR SOLUTION 29 1.5.1 Another Contract Language Compiler Validation 29 1.5.2 Contract Intermediate Language Compiler Validation 30 1.5.3 Validation through Case Studies 30 1.6 CONTRIBUTIONS 32 1.7 ORGANIZATION OF THE THESIS 33 2 RELATED WORK 34 2.1 CODE-BASED TESTING 34 2.1.1 JUnit 35 2.1.2 FindBugs, FxCop, and PREFast 35 2.1.3 Splint 36 2.1.4 Discussion 37 2.2 MODEL-BASED TESTING 37 2.2.1 The Unified Modeling Language 39 2.2.2 Model-Based Tools 39 2.2.3 Model-Based Testing and Our Approach 41 2.2.4 Beyond State-Of-The-Art 63 2.3 CONTRACT-BASED TESTING 67 2.3.1 Basic Contracts 69 2.3.2 Behavioural Contracts 70 2.3.3 Behavioural Contract Implementation 74 2.3.4 Synchronization Contracts 82 2.3.5 Quality of Service Contracts 83 2.3.6 Summary 84 2.4 A SUMMARY OVERVIEW OF EXISTING WORK 85 3 EXAMPLE WALKTHROUGH 89 3.1 THE CONTAINER EXAMPLE VERSION 1: No MULTIPLE OCCURRENCES 89 ~ v ~ Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 3.1.1 The Testable Requirements Model 90 3.1.2 The Implementation Under Test 122 3.1.3 Bindings 129 3.1.4 Execution 131 3.1.5 The Contract Evaluation Report 132 3.2 THE CONTAINER EXAMPLE VERSION 1.1: MULTIPLE OCCURRENCES 135 3.2.1 The Testable Requirements Model 136 3.2.2 The Implementation Under Test 139 3.2.3 Bindings 143 3.3 ADDITIONAL CASE STUDIES 146 3.3.1 A Simple Container. 146 3.3.2 Web-Based lUTs 147 3.3.3 The Grocery Store 147 3.3.4 The University 148 4 THE VALIDATION FRAMEWORK 149 4.1 OVERVIEW 150 4.2 IMPLEMENTATION TECHNOLOGY 154 4.2.1 Java and the Eclipse Platform 155 4.2.2 Microsoft Phoenix 155 4.2.3 Side-Effects 158 4.2.4 Microsoft Visual Studio Integration 162 4.3 ANOTHER CONTRACT LANGUAGE 163 4.4 IMPLEMENTATION UNDER TEST 165 4.4.1 .NET Executables 165 4.4.2 C++ Executables 166 4.4.3 ASP.NET Web Applications 166 4.5 BINDINGS 167 4.5.1 The Binding Tree 168 4.5.2 Binding Status 169 4.5.3 The Binding Viewer 169 4.6 CONTRACT INTERMEDIATE LANGUAGE 183 4.7 STATIC CHECKS 185 4.7.1 Creation of a Static Check 186 4.8 EXECUTING THE IMPLEMENTATION UNDER TEST 191 4.8.1 Execution of a Bound Responsibility 192 4.8.2 Dynamic Checks 195 4.8.3 Scenario Execution 198 4.9 METRICS 200 4.9.1 Creation of a Metric Evaluator. 201 4.10 CONTRACT EVALUATION REPORT 203 4.10.1 The Report Tree 204 4.10.2 The Main View 206 4.11 SUMMARY 216 4.12 MORE ON VALIDATION 216 5 CONCLUSION AND FUTURE WORK 218 5.1 SUMMARY OF OUR APPROACH 218 5.1.1 Framework Input 218 5.1.2 Plug-ins 219 5.1.3 Compilation 221 5.1.4 Execution 221 5.1.5 Contributions 222 ~ vi ~ Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 5.2 DISCUSSION OF TECHNICAL LIMITATIONS 223 5.2.1 Inputs 223 5.2.2 Distribution 223 5.2.3 Scalability 224 5.2.4 Performance 225 5.3 FUTURE WORK 226 5.3.1 The Bigger Picture 226 5.3.2 Extension through Openness 228 5.3.3 Internal Framework Enhancement 228 5.3.4 Use of Graphical Testable Requirements Models 229 5.3.5 Interpretation of the Contract Evaluation Report 229 5.4 CONCLUSIONS 230 REFERENCES 231 ~ vii ~ Reproduced with permission of the copyright owner.