Universität Bremen Fachbereich 3 Hard Real-Time Linux for Off-The-Shelf Multicore Architectures Dissertation zur Erlangung des Grades eines Doktors der Ingenieurwissenschaften – Dr.-Ing. – vorgelegt von Dirk Radder im Fachbereich 3 der Universität Bremen am 19. Mai 2015 Datum des Promotionskolloquiums: 10.11.2015 Gutachter: Prof. Dr. Jan Peleska (Universität Bremen) Prof. Dr.-Ing. Görschwin Fey (Universität Bremen) Zusammenfassung In dieser Dissertation werden die Forschungsergebnisse bezüglich der Entwicklung einer Echtzeiterweiterung für das Linux Betriebssystem vorgestellt. Die Arbeit beschreibt eine vollständige Erweiterung das Kernels, welche hartes Echtzeitverhalten auf einer 64 Bit x86 Architektur ermöglicht. Im ersten Teil dieser Arbeit werden Echtzeit-Systeme kategorisiert und Konzepte von Echtzeit-Betriebssystemen eingeführt. Im Weiteren werden zahlreiche bekannte Echtzeit-Betriebssysteme betrachtet. QNX Neutrino, RT_PREEMPT Linux Patch sowie HLRT Linux Patch werden detailiert analysiert und deren Kernkonzepte ausführlich diskutiert. Darüber hinaus wird eine Test-Suite erarbeitet, mit der aussagekrätige Benchmarks der analysierten Systeme erstellt werden. Die Systeme werden anhand dieser Benchmarks evaluiert und mit der in dieser Arbeit entwickelten Echtzeit- Erweiterung verglichen. Anhand der vorausgegangenen Analysen der genannten Systeme wird ein Katalog von Anforderungen definiert, den die entwickelte Echtzeit-Erweiterung umsetzen wird. Basierend auf diesem Anforderungs-Katalog und den identifizierten Kernkonzepten der analysierten Systeme wird der Entwurf der Echtzeit-Erweiterung erarbeitet und deren konkrete Implementierung dargestellt. Abschließend werden die Benchmarks aller analysierten Systeme, einschließlich der erarbeiteten Echtzeit-Erweiterung, miteinander verglichen und bewertet. iii Abstract This document describes the research results that were obtained from the development of a real-time extension for the Linux operating system. The paper describes a full extension of the kernel, which enables hard real-time performance on a 64-bit x86 architecture. In the first part of this study, real-time systems are categorized and concepts of real-time operating systems are introduced to the reader. In addition, numerous well-known real-time operating systems are considered. QNX Neutrino, RT_PREEMPT Linux Patch and HLRT Linux Patch are analyzed in detail. The core concepts of these systems are shown and discussed. Furthermore, a test suite is developed, which is used to obtain expressive benchmarks from the systems that were analyzed before. The systems are evaluated on the basis of these benchmarks and compared to the real-time extension which is developed in this work. A requirements catalogue is defined based on the analysis of the stated operating systems. The design of a real-time extension is developed based on the specification catalogue and the identified core concepts. Furthermore, the concrete implementation of the developed real-time extension is presented in detail. Finally, the benchmarks of all analyzed systems, including the developed real-time extension, are compared to each other and evaluated. v Contents List of Figures xiii List of Tables xv 1 Introduction 1 1.1 Objectives .................................. 2 1.2 Main Contributions ............................. 2 1.3 Related Work ................................ 3 1.4 Structure of this Document ........................ 6 I Background and Components 9 2 Concepts of Real-Time Systems 11 2.1 Definition and Classification of Real-Time Systems ........... 11 2.1.1 Proprietary versus Open ...................... 12 2.1.2 Centralized versus Distributed .................. 13 2.1.3 Fail-Safe versus Fail-Operational ................. 13 2.1.4 Hard versus Soft Real-Time .................... 14 2.1.5 Event-Triggered versus Time-Triggered .............. 15 2.2 Requirements of Real-Time Systems ................... 16 2.2.1 Functional Requirements ...................... 17 2.2.2 Temporal Requirements ...................... 17 2.2.3 Dependability Requirements .................... 18 2.2.4 Architectural Requirements .................... 18 2.3 Hardware for Real-Time Systems ..................... 19 2.4 x86 Instruction Set Architecture ..................... 20 2.4.1 General Remarks and Design ................... 20 2.4.2 Hyper-Threading Technology ................... 21 2.4.3 APIC Architecture ......................... 21 2.4.4 Time Stamp Counter ........................ 22 2.4.5 Problems with Hard Real-Time on the x86 Architecture .... 22 3 Real-Time Operating Systems 23 3.1 Task Management ............................. 23 vii Contents 3.1.1 Task States ............................. 24 3.1.2 Task Hierarchy ........................... 24 3.1.3 Task Types ............................. 25 3.2 Scheduling .................................. 25 3.2.1 Clock Driven Scheduling Strategies ................ 26 3.2.2 Event Driven Scheduling Strategies ................ 26 3.2.2.1 Earliest-Deadline-First Scheduling ........... 26 3.2.2.2 Rate-Monotonic Scheduling ............... 26 3.2.3 Hybrid Scheduling Strategies ................... 27 3.2.3.1 First-In-First-Out Scheduling .............. 27 3.2.3.2 Round-Robin Scheduling ................ 28 3.3 Partitioning ................................. 28 3.4 Main System Services ........................... 29 3.5 POSIX Standard .............................. 29 3.5.1 Real-Time System Profiles ..................... 30 3.6 Real-Time and Linux ............................ 31 3.6.1 Real-Time Kernel .......................... 31 3.6.2 Kernel Preemption ......................... 32 3.6.3 Resource Reservation ........................ 32 II Real-Time Operating Systems Analysis 33 4 Overview of Available Real-Time Operating Systems 35 4.1 Atomthreads ................................ 39 4.1.1 Structure of the Kernel ...................... 39 4.1.2 CPU Architecture Ports ...................... 41 4.2 eCos ..................................... 41 4.2.1 Design ................................ 42 4.3 VxWorks .................................. 42 4.3.1 Protection Domains Architecture ................. 43 4.4 μC/OS-III .................................. 43 4.4.1 Design ................................ 44 4.5 MontaVistaLinux .............................. 44 4.6 ThreadX ................................... 44 4.6.1 Pico Kernel ............................. 45 4.7 RTLinux ................................... 45 4.7.1 Structure of the Kernel ...................... 45 4.8 QNX, HLRT and RT-Preempt ....................... 46 5 Evaluating (POSIX) Real-Time Operating Systems 49 5.1 Unconsidered Aspects ........................... 49 5.2 Identify Technical Values .......................... 50 5.2.1 Benchmark Methodology ...................... 51 viii Contents 5.3 Case Scenarios for Real-Time Systems .................. 53 5.3.1 Development Board ........................ 54 5.4 Benchmark Test Framework ........................ 55 5.4.1 Measurement Details ........................ 55 5.4.2 Operating System Overhead .................... 57 5.4.3 Test Design ............................. 58 5.4.3.1 Task Period Accuracy .................. 59 5.4.3.2 Task Change Times ................... 60 5.4.3.3 Task Creation Time ................... 62 5.4.3.4 Interrupt Times ..................... 63 6 Case Study 1: RT-Preempt Patch 67 6.1 Background and Overview ......................... 67 6.2 Preemptable In-Kernel Locking Primitives ................ 68 6.2.1 Priority Inheritance for In-Kernel Locking Primitives ...... 70 6.3 Interrupt Handlers as Kernel Threads .................. 72 6.4 Real-Time Application Programming ................... 75 6.5 Benchmarking ................................ 75 6.5.1 Task Period Tests ......................... 76 6.5.2 Task Switch Tests ......................... 79 6.5.3 Task Creation Test ......................... 84 6.5.4 Interrupt Tests ........................... 86 6.6 Summary .................................. 89 7 Case Study 2: HaRTLinC 91 7.1 Background and Overview ......................... 91 7.2 CPU Reservation .............................. 92 7.2.1 The SCHED_HLRT Scheduling Policy ............... 92 7.2.2 Interrupt Routing ......................... 96 7.2.3 Necessary Adjustments ....................... 97 7.3 Time-Triggered Architecture ........................ 98 7.4 Real-Time Application Programming ................... 99 7.5 Benchmarking ................................100 7.5.1 Task Period Tests .........................100 7.5.2 Interrupt Tests ...........................103 7.6 Summary ..................................105 8 Case Study 3: QNX Neutrino 107 8.1 Background and Overview .........................107 8.2 Microkernel Architecture ..........................107 8.2.1 Process Management ........................108 8.2.2 Interrupt Handling .........................110 8.2.3 Message Passing ..........................110 8.3 Adaptive Partitioning Scheduler ......................111 ix Contents 8.4 Benchmarking ................................114 8.4.1 Task Period Tests .........................115 8.4.2 Task Switch Tests .........................117 8.4.3 Task Creation Test .........................121 8.4.4 Interrupt Tests ...........................121 8.5 Summary ..................................124 III A Hard Real-Time Linux Operating System 125 9 Requirements Discussion 127 9.1 Analysis ...................................130 9.2 Coverage ...................................131 10 Hard Real-Time Linux System Design 133