Toward Improved Traceability of Safety Requirements and State-Based Design Models A Dissertation submitted to the Graduate School of the University of Cincinnati in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY at the UNIVERSITY OF CINCINNATI COLLEGE OF ENGINEERING AND APPLIED SCIENCE March. 2021 by Mounifah Alenazi M.Sc. Kennesaw State University May. 2016 Thesis advisor and Committee chair: Nan Niu, Ph.D Abstract Traceability has long been recognized as an important component in building safety critical systems. Traceability therefore is often required by many government regulations. For example, the Federal Aviation Administrations (FAA) standard DO-178B specifies that software developers must be able to demonstrate traceability of designs against requirements. In systems engineering projects, the development of complex and dependable systems like autonomous vehicles relies increasingly on the use of the Systems Modeling Language (SysML). In fact, SysML has become a de facto standard for systems engineering. Effective traceability in such systems can be very costly and difficult. Researchers have therefore proposed many techniques to automatically establish and evolve trace links for high assurance projects. Various research approaches use information retrieval-based tracing methods to automatically recover trace links between modeling artifacts. For example, to verify a safety requirement, a query is used to retrieve the related elements in the design models. Our ability to trace is therefore anchored to the ability to retrieve. While trace retrieval has been the predominant way of automatically creating links, the performance is yet to be satisfactory for broad industrial adaption, and many false positives remain a significant challenge. In this thesis, we present a novel approach that overcomes this challenge. In particular, the work in this thesis has three main objectives. The first is to identify and address the research challenges of identifying trace links in the context of SysML models. For this objective, we empirically investigate if traditional traceability approaches using textual information could yield promising results in our context. We also conduct a comprehensive investigation of traceability features within state-of-the- practice SysML modeling tools to understand how the traceability information is iii supported and managed in these tools. The second objective is to leverage mutation analysis and process mining to verify safety requirements. For this objective, we first carry out a systematic mapping study to identify the common modeling mistakes in SysML. Our goal is to understand the scope of these mistakes (the incorrect links), their types, the implications of those mistakes in model-driven requirements engineering, and then use these mistakes as a basis to identify mutation operators. Once the mutants are created, they undergo model checking so as to automatically verify the safety requirements. Building this foundation is a necessary step that facilitates the third objective which is to tackle false positives that have plagued automated requirements traceability. Rather than striving for defining an accurate tracing mechanism which often ends up with many imperfect links, our core idea is to exploit the mutants (imperfect tracing targets) and then take full advantage of them to discover the traceability links. Checking the requirements over the mutants leads to the distinction between killed and survived mutants. We leverage the underlying killed-survived distinction and develop a correlation analysis procedure to identify the traceability links. The results show considerable precision improvements compared with the state-of-the-art. v Acknowledgements First and foremost, I would like to express my deepest appreciation to my advisor Dr. Nan Niu for his support and enthusiastic encouragement throughout my graduate studies. I could not have finished this dissertation without his continuous guidance. Working under his supervision has been an unforgettable learning experience for me. I am grateful for the tremendous amount of time and effort he devoted to not only discussing my ideas, providing feedback and suggestions, collaborating with me, and celebrating our achievements, but also allowing to present our work at top-tier conferences and meet very well-known researchers in our field. Despite his busy schedule, he was always available and generous in sharing his experiences on academic life and beyond. Dr. Niu set a great example for me as a great mentor and research supervisor. I am very grateful to my committee members Raj Bhatnagar, Chia Yung Han, Carla Purdy, as well as my external examiner Gunter Mussbacher for serving in my committee and giving valuable and constructive comments. I thank Professors Dan Lo, Michael Franklin, Frank Tsui from KSU and George Purdy From UC for their impact on my academic life. It was an honor to be one of their students. Their impact will last forever. I would like to thank all the members of our lab, especially, Wentao, Hemanth, Rue, Zedong, and Xuanyi for collaborations, discussions, and friendships. I have always enjoyed our conversations. Special thanks to Abhijith for the good discussions and feedback. I also thank my best friends Asma, Khitam, Mona, and Fatma for the great times we spent together. I thank my country, the Kingdom of Saudi Arabia, for supporting me and my family throughout my graduate studies. I would like also to thank the University vi of Cincinnati for the UGS Award and for providing me the opportunity to pursue my doctoral studies. Finally, I would like to thank my parents, my brothers and sisters for their support and love and for always praying for me. My special thanks go to my dear husband Fahad for all his support and encouragement throughout these years. Thanks for your understanding and sacrifice. Thanks for helping me achieve my dream. Thanks to my kids, Faisal, Osama, and Raed. You have been my motivation, my inspiration and drive. This dissertation is dedicated to you. vii Contents Abstract.................................... ii List of Figures................................. viii List of Tables................................. xi 1 Introduction1 1.1 Motivation................................1 1.2 Scope..................................2 1.3 Thesis Contribution...........................7 1.4 Thesis Organization...........................8 2 Background and Related Work 10 2.1 Systems Modeling Language (SysML)................. 10 2.2 Traceability............................... 13 2.3 Mutation Analysis........................... 20 2.4 Summary................................ 23 3 Assuring Safety Requirements Using Textual Information 24 3.1 Introduction............................... 25 3.2 Experimental Setup........................... 26 3.3 Subject System............................. 28 viii 3.4 Results and Analysis.......................... 35 3.5 Discussion................................ 38 3.6 Summary................................ 40 4 SysML Modeling Mistakes: A Systematic Literature Mapping 42 4.1 Introduction............................... 42 4.2 Related Work.............................. 45 4.3 Mapping Study Design......................... 47 4.4 Results and Analysis.......................... 52 4.5 Concluding Remarks.......................... 65 4.6 Summary................................ 67 5 Tracing Safety Requirements and State-Based Design Models 68 5.1 Introduction............................... 68 5.2 Running Example............................ 71 5.3 Traceability Information Model.................... 75 5.4 Mutation-Driven Traceability..................... 78 5.5 Experimental Evaluation........................ 88 5.6 Summary................................ 96 6 Conclusions and Future Work 97 6.1 Thesis Summary............................ 97 6.2 Limitations............................... 99 6.3 Future Directions............................ 100 Bibliography 101 ix List of Figures 2.1 SysML diagrams and their relationships with UML 2 (adapted from [52])................................. 11 2.2 Example of a simple traceability tree.................. 14 2.3 Integration analysis of SysML and model checking (adapted from [149]). ...................................... 23 3.1 Transmission Control Module (TCM) [138].............. 29 3.2 Fault Tree Analysis........................... 30 3.3 User Interface of the V-PLC [138]................... 30 3.4 F2 metric for similarity measures................... 37 3.5 Integrating a virtual PLC in SysML models adapted from [21].... 38 3.6 Fault Tree Analysis Example..................... 39 3.7 Tree-based coverage for similarity measure S1 (left) and S2 (right) 40 4.1 SysML activity diagram reviewed in our study (Figure 13 in PS2 ). 51 4.2 Distribution of the 42 SysML mistake types.............. 53 4.3 Observability of the 42 mistakes in SysML models.......... 55 4.4 SysML diagrams and mistake types.................. 57 4.5 SysML diagrams and mistake observability.............. 58 x 4.6 SysML mistakes' impacts on requirements............... 62 4.7 Illustration of SysML mistakes' impacts on requirements....... 63 5.1 State machine diagram (SMD) of the water distiller example (adapted from [51])................................. 72 5.2 Traceability information contextualizing the artifacts and relations relevant to our approach......................... 75 5.3 Overview of our mutation-driven traceability approach where mu- tants are created by modifying the tracing