Scalable Model-based Robustness Testing: Novel Methodologies and Industrial Application Shaukat Ali Thesis submitted for the degree of Ph.D. Department of Informatics Faculty of Mathematics and Natural Sciences University of Oslo September 2011 © Shaukat Ali, 2011 Series of dissertations submitted to the Faculty of Mathematics and Natural Sciences, University of Oslo No. 1154 ISSN 1501-7710 All rights reserved. No part of this publication may be reproduced or transmitted, in any form or by any means, without permission. Cover: Inger Sandved Anfinsen. Printed in Norway: AIT Oslo AS. Produced in co-operation with Unipub. The thesis is produced by Unipub merely in connection with the thesis defence. Kindly direct all inquiries regarding the thesis to the copyright holder or the unit which grants the doctorate. Abstract Embedded systems, as for example communication and control systems, are being increasingly used in our daily lives and hence require thorough and systematic testing before their actual use. Many of these systems interact with their environment and, therefore, their functionality is largely dependent on this environment whose behavior can be unpredictable. Robustness testing aims at testing the behavior of a system in the presence of faulty situations in its operating environment (e.g., sensors and actuators). In such situations, the system should gracefully degrade its performance instead of abruptly stopping execution. To systematically perform robustness testing, one option is to resort to Model-Based Robustness Testing (MBRT), which is a systematic, rigorous, and automated way of conducting robustness testing. However, to successfully apply MBRT in industrial contexts, new technologies need to be developed to scale to the complexity of real industrial systems. This thesis presents a solution for MBRT on industrial systems, including scalable robustness modeling and executable test case generation. One important contribution of this thesis is a scalable RobUstness Modeling Methodology (RUMM), which is achieved using Aspect-Oriented Modeling (AOM). It is a complete, automated, and practical methodology that covers all features of state machines and aspect concepts necessary for MBRT. Such methodology, relying on a standard (Unified Modeling Language or UML) and using the target notation as the basis to model the aspects themselves, is expected to make the practical adoption of robustness modeling easier in industrial contexts. The applicability of the methodology is demonstrated using an industrial case study. Results showed that the approach significantly reduced modeling effort (98% on average), improved separation of concerns, and eased model evolution. The approach is further empirically evaluated using two controlled experiments involving human subjects and results showed that the proposed methodology significantly improves the readability of models as compared to modeling using standard UML notations. Another important contribution of this thesis is an efficient approach for solving constraints (written in Objects Constraint Language (OCL)) on the operating environment of a system, which is mandatory for emulating faulty situation in the environment for the purpose of MBRT. A set of novel heuristics is devised for various OCL constructs, which are required for the application of search algorithms. The heuristics have been empirically evaluated on an industrial case study for robustness testing and the results showed to be i very promising and significantly better than the existing works in the literature on OCL constraint solvers. A final contribution of the thesis is robustness test case generation from the models developed using RUMM. Test case generation also includes scripts generation for environment emulation, which is mandatory for automated robustness testing again using an industrial case study. In preliminary experiments, the execution of test cases found one critical, robustness fault in a deployed industrial system. ii Acknowledgements First of all, I would like to thank my supervisors Lionel Briand and Andrea Arcuri. Without their deep knowledge in software engineering and constant guidance throughout my PhD, this work couldn’t have been possible. Lionel is an excellent supervisor and his profound knowledge and passion for research has instilled in me a passion to conduct quality research. Andrea is an outstanding supervisor and his immense knowledge about search algorithms and statistics have provided me an opportunity to grasp these topics very quickly. I would also like to thank Erik Arisholm, who was my supervisor during earlier part of the PhD. I always had good discussions with him. I would also like to thank Simula Research Laboratory (SRL) and Simula School of Research and Innovation (SSRI) for funding my PhD and providing an excellent multicultural research environment. I would like to thank Marius Liaaen from Cisco, whose intense passion for model-based testing enabled me to successfully apply and evaluate my model-based methodologies in Cisco. I would also like to thank Suneth Walawege from Cisco, who is also a friend, for helping in modeling Cisco's case study and running experiments. I would like to specially thank my best friend Tao Yue, for her unconditional support in everything. Particularly, during my PhD she helped in conducting controlled experiments in Pakistan and China. Special thanks to my best friend and office mate, Aiko Yamashita for providing an excellent company in the office and outside. She has been a wonderful friend in many different ways. I would also like to thank my best friends in Simula, Rajwinder Panesar-Walawege, Zohaib Zafar, and Hadi Hemmati for all the good discussions at work and fun times in general. Last but not least, I would also like to thank my parents, brother, and sisters for their love, support, and care. iii List of papers The following papers are included in this thesis: Paper 1. A Systematic Review of the Application and Empirical Investigation of Search-based Test-Case Generation S. Ali, L. Briand, H. Hemmati, and R. K. Panesar-Walawege Published in the IEEE Transactions on Software Engineering (TSE), vol 36, no 6, pp. 742-762, 2010 Paper 2. Modeling Robustness Behavior Using Aspect-Oriented Modeling to Support Robustness Testing of Industrial Systems S. Ali, L. Briand, and H. Hemmati Accepted for a publication in the Journal of Software and Systems Modeling (SOSYM), Springer, 2011. Paper 3. Does Aspect-Oriented Modeling Help Improve the Readability of UML State Machines? S. Ali, T. Yue, and L. Briand Submitted to the the Journal of Software and Systems Modeling (SOSYM), Springer, 2011. Paper 4. Solving OCL Constraints for Test Data Generation in Industrial Systems with Search Techniques S. Ali, M. Z. Iqbal, A. Arcuri, and L. Briand. Submitted to ACM Transactions on Software Engineering and Methodology (TOSEM), 2011 Paper 5. An Industrial Application of Robustness Testing using Aspect-Oriented Modeling, UML/MARTE, and Search Algorithms S. Ali, L. Briand, A. Arcuri, and S. Walawege. In: ACM/IEEE 14th International Conference on Model Driven Engineering Languages and Systems (Models 2011), ACM/IEEE, 2011. The five papers are self-contained and thus some information might be redundant across the papers. Different abbreviations may have been used in the papers. iv My contributions For all papers except the first paper, I was the main contributor. My supervisors contributed in all phases of the work. For Paper 1, all authors equally contributed to the paper. In case of Paper 2, I was main contributor for the idea, implementation, and case study design and application. The controlled experiments reported in Paper 3 are conducted in collaboration with Tao Yue and Lionel Briand. I was responsible for creating the experiment material, experiment execution, data collection and analysis, and writing of the paper. Tao Yue and Lionel Briand were involved throughout the process. In Paper 4, I was the main contributor, but got help from supervisors (Andrea Arcuri and Lionel Briand) and collaborator Mohammad Zohaib Zafar. In the last paper, I was the main contributor for the idea, implementation, and case study design and application. In addition, during my PhD study, I also contributed in other papers which are not included in this thesis. Paper 6 is not included since the journal version of the paper (Paper 4) is included in this thesis. Paper 7 is excluded since it is a result of an equal contribution from all authors and is a basic framework which was needed to conduct the research reported in this thesis. Paper 8 and Paper 9 are not directly related to this PhD thesis, but are preliminary extensions to the work presented in the thesis. Paper 10 is an additional controlled experiment, which was conducted to evaluate the modeling methodology presented in Paper 2. The paper was submitted to a conference at the time of submission of this thesis for evaluation. Paper 6. A Search-based OCL Constraint Solver for Model-based Test Data Generation S. Ali, M. Z. Iqbal, A. Arcuri, L. Briand In: Proceedings of the 11th International Conference on Quality Software (QSIC 2011), pp. 41-50, IEEE, 2011. Paper 7. Model Transformations as a Strategy to Automate Model-Based Testing - A Tool and Industrial Case Studies. S. Ali, H. Hemmati, N. E. Holt, E. Arisholm, and L. Briand v Technical Report 2010-01, Simula Research Laboratory Paper 8. Automated Transition from Use Cases to UML State Machines to Support State-based Testing. T. Yue, S. Ali, and L. Briand. In: