Fault Injection Framework for Time-Triggered Systems DISSERTATION zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften (Dr.-Ing.) Dissertation vorgelegt von: Onwuchekwa, Daniel Lucky eingereicht bei der Naturwissenschaftlich-Technischen Fakultät der Universität Siegen Date of Oral Examination: 09. October 2020 Betreuer und erster Gutachter: Prof. Dr. Roman Obermaisser, Universität Siegen Zweiter Gutachter Prof. Dr. Kristof Van Laerhoven, Universität Siegen Prüfungskommission: Prof. Dr. Roman Obermaisser Prof. Dr. Kristof Van Laerhoven Prof. Frank Gronwald Prof. Malte Lochau Acknowledgement I wrote this thesis during my employment at the department of Embedded Systems, Uni- versity of Siegen. I thank Prof. Dr Roman Obermaisser for providing the opportunity for me to be a member of his team, and also for introducing me into the field of fault-tolerant and safety-critical systems. His valuable remarks and criticism advanced my scientific experience. I would also like to thank my colleague, friend, and football teammate, Tobias Pieper. He played a considerable role that enabled my integration in Germany. He supported my settlement, taught me a lot about the German culture and introduced me to TSG Adler Dielfen football team, a common interest for us. I would also like to thank all my colleagues at the institute. Tobias, Hongie Fang and Maryam Palehvan have been an immense source of inspiration and a pool of knowledge which was beneficial during my work. I thank my colleague also Stefan Otterbach for ensuring that I get all the necessary tools required for the success of this work. I also thank Simon Meckel, Veit Wiese, Michael Schmidt, Hamidreza Ahmadian and Setareh Majidi for the pleasant atmosphere and harmony at work. Special thanks go to my dearest wife Jennifer, who has been very helpful in providing constructive criticisms and exhibited patience on the way to reach my goal. She also provided the required emotional support to carry out my work. I also thank my kids, Giovanni and Gianna, for providing me with the necessary daily smiles and emotional support. I also thank my sisters Eberechi and Ezinne for their encouragement. I would also like to thank my in-laws for their presence and support in much needed time while I carried out my work. Finally, I would like to thank my friends Ugochukwu Osabiku, Oghneneochuko Obie, and Raymond Webilor for their support. I Zusammenfassung In dieser Dissertation wird eine Methodik zur Verifizierung und Validierung des Ver- haltens von integrierten System vorgestellt, die auf zeitgesteuerten Ethernet-Netzwerken basieren. Der Determinismus und die ausreichende Bandbreite, die durch ein zeitgesteu- ertes Ethernet-Netzwerk bereitgestellt werden, ermöglichen die Konstruktion sicherheits- kritischer Systeme in verschiedenen Bereichen wie Eisenbahn, Luftfahrt, Gesundheit und Automobil. Viele Anwendungen in diesen Bereichen stellen hohe Anforderungen an die Zuverlässigkeit. Deshalb sind Verifizierung und Validierung in den meisten Phasen des Entwicklungsprozesses sicherheitskritischer Systeme erforderlich. Aufgrund der Komplexität von zeitgesteuerten Netzwerkprotokollen verwenden Entwick- ler meist formale Methoden und Simulationen als Verifikations- und Validierungstechni- ken. Allerdings verifizieren und validieren diese Methoden hauptsächlich bestimmte Funk- tionen der zeitgesteuerten Protokolle und nicht das Verhalten des integrierten Systems. Die Gründe dafür liegen in den Nachteilen dieser Ansätze. Die Modellierung komplexer Systeme führt bei der Benutzung formaler Methoden zu einer Explosion des Zustands- raums und Simulatoren modellieren bestimmte komplexe Funktionen nicht ausreichend. Des Weiteren erfordern Simulatoren eine zusätzliche Verifizierung durch ein physikalisches Netzwerk, um die Aussagekräftigkeit zu verbessern. Da die Evaluierung der physikalischen Realisierung von zeitgesteuerten Ethernet-Netzwerken zu den besten Ergebnissen führt, konzentriert sich diese Arbeit auf die Anwendung der Fehlerinjektion auf physikalische Geräte. Diese Dissertation schlägt ein neuartiges und topologieunabhängiges Cut-Through-Fehler- injektions-Framework vor, welches das integrierte Systemverhalten von zeitgesteuerten Ethernet-Netzwerken auswerten kann. Sie bietet zudem eine Lösung für die Fehlerer- kennung in zeitgesteuerten Netzwerken während des Synchronisationsstarts, bevor eine globale Zeit festgelegt wird. Darüber hinaus werden experimentelle Verfahren und Ergeb- nisse diskutiert, die die Verwendung des Fehlerinjektions-Frameworks für die Bewertung einer Auswahl verschiedener Anwendungsfälle demonstrieren. Die hier durchgeführten Experimente bestätigen, wie das neuartige Framework andere zeitgesteuerte Ethernet- II 0.0 Inhaltsverzeichnis Frameworks übertrifft, indem es die kollektiven Anforderungen erfüllt. Hierzu gehören geringe Störanfälligkeit, Portabilität und die Abstraktion der Fehlerinjektionskomponen- te aus dem zu testenden Netzwerk. Page III Abstract This thesis presents a methodology and tool for verifying and validating the integrated system behaviour of time-triggered Ethernet networks. The determinism and sufficient bandwidth provided by time-triggered Ethernet network make it appealing for building safety-critical systems in different domains such as railway, aviation, health, and auto- mobile. Many applications in these domains impose stringent dependability requirements. Therefore, verification and validation are often required at most stages of the development process when designing these systems. Due to the complexity of time-triggered network protocols, design engineers mostly employ formal methods and simulations as the verification and validation techniques. However, these methods mainly verify and validate only certain functions of the time-triggered protocol and not the integrated system behaviour. The reasons stem from the downsides of these approaches. The formal method suffers from a state-space explosion when modelling complex systems, and simulators do not sufficiently model certain complex functionality. Simulators also require cross-verification from a physical network to gain better confidence. Since evaluating the physical realisation of time-triggered Ethernet networks results in the best confidence levels, this work then focuses on the use of fault injection on physical devices for this purpose. This work proposes a novel and topology independent cut-through fault injection frame- work that can be used to evaluate the integrated system behaviour of time-triggered Eth- ernet networks. This work also describes a technique that can be used for failure detection in time-triggered networks during the synchronisation startup before the establishment of global time. It furthermore presents a discussion of experimental procedure(s) and results that demonstrate the use of the fault injection framework for the evaluation of a selection of different use cases. The Experiments carried out herein confirms how the novel fault injection framework surpasses other time-triggered Ethernet frameworks by satisfying a set of collective requirements which mainly include low-intrusiveness, portability, and the abstraction of fault injection component from the network under test. IV Contents Acknowledgement I Zusammenfassung III Abstract IV Table of Contents V 1 Introduction 1 1.1 Context and motivation . 1 1.2 Objectives and contribution . 3 1.3 Thesis structure . 5 2 Background Theory 7 2.1 Real-time systems . 7 2.2 dependability of a system . 8 2.2.1 Means to attain dependability in a system . 8 2.2.2 Threats to the dependability of a system . 9 2.2.3 Attributes of the dependability of a system . 10 2.2.4 Redundancy . 11 2.2.5 Methods of dependability evaluation . 12 2.2.6 Safety . 13 2.2.7 Safety-criticality system . 13 2.3 Verification and validation . 14 2.4 Fault injection . 14 2.4.1 Fault injection categorisation . 15 2.4.2 Fault injection environment . 16 2.4.3 Modelling a fault injection framework . 17 2.4.4 Types of fault injection . 19 2.5 Concept of deep learning . 21 2.5.1 Machine learning . 21 2.5.2 Deep learning . 22 3 Time-Triggered Ethernet communication 26 3.1 Ethernet . 27 3.1.1 Open system interconnection layers . 27 3.1.2 Key characteristics of Ethernet . 28 V TABLE OF CONTENTS 3.1.3 Components of Ethernet . 30 3.1.4 Absence of determinism in Ethernet . 31 3.2 Time-triggered control . 31 3.2.1 Clock and global time . 32 3.2.2 Clock offset . 33 3.2.3 Time-Triggered system . 34 3.3 TTEthernet system . 34 3.3.1 TTEthernet frame structure . 36 3.3.2 Fault tolerant clock synchronization . 37 3.3.3 TTEthernet startup and restart service . 40 3.4 Time sensitive networking . 43 3.4.1 Background on precision time protocol and profile . 45 3.4.2 Generalized precision time protocol . 49 3.4.3 Start-up time . 50 4 Related Work 55 4.1 Requirement . 55 4.2 Fault injection tools . 58 4.3 Network verification methods . 61 4.4 Related works on the verification of network protocols . 63 4.4.1 Verification and validation of TTEthernet . 65 4.4.2 Verification and validation of TSN . 68 4.5 Summary of related works . 69 5 System Model of Fault Injection Framework 72 5.1 System model . 72 5.1.1 Fault hypothesis . 73 5.2 TRAITOR in TTEthernet . 74 5.2.1 Architectural overview of TRAITOR . 75 5.2.2 Fault injector component . 78 5.2.3 Observation probe . 80 5.2.4 Data collector/analyser . 81 5.2.5 Controller . 81 5.2.6 Monitor . 82 5.3 TSN fault injection framework . 83 5.4 FPGA block diagram design . 83 5.5 Software design . 84 5.6 TRAITOR operation summary . 86 6 Implementation