Real-Time Linux (RT-Preempt) + SCHED DEADLINE (Linux Kernel 3.14) Josef Widder BRDS 2015 I Real-time computing is equivalent to fast computing I Real-time programming is assembly coding I Real-time is meaningless if hardware will fail or software contains bugs I Real-time systems function in static environments Misconceptions about Real-Time Computing [Sta88] I Advances in supercomputer hardware will take care of real-time requiremens 2 I Real-time programming is assembly coding I Real-time is meaningless if hardware will fail or software contains bugs I Real-time systems function in static environments Misconceptions about Real-Time Computing [Sta88] I Advances in supercomputer hardware will take care of real-time requiremens I Real-time computing is equivalent to fast computing 2 I Real-time is meaningless if hardware will fail or software contains bugs I Real-time systems function in static environments Misconceptions about Real-Time Computing [Sta88] I Advances in supercomputer hardware will take care of real-time requiremens I Real-time computing is equivalent to fast computing I Real-time programming is assembly coding 2 I Real-time systems function in static environments Misconceptions about Real-Time Computing [Sta88] I Advances in supercomputer hardware will take care of real-time requiremens I Real-time computing is equivalent to fast computing I Real-time programming is assembly coding I Real-time is meaningless if hardware will fail or software contains bugs 2 Misconceptions about Real-Time Computing [Sta88] I Advances in supercomputer hardware will take care of real-time requiremens I Real-time computing is equivalent to fast computing I Real-time programming is assembly coding I Real-time is meaningless if hardware will fail or software contains bugs I Real-time systems function in static environments 2 general-purpose most of the time good behavior I high throughput I under normal (avg. case) behavior Real-Time vs. general-purpose OS real-time always predictable behavior I timing guarantees I under worst case behavior 3 Real-Time vs. general-purpose OS real-time always predictable behavior I timing guarantees I under worst case behavior general-purpose most of the time good behavior I high throughput I under normal (avg. case) behavior 3 (even with patch, convenient Linux features should not be touched) Why (Vanilla) Linux is not Real-time (bird's eye view) I general-purpose I high functionality I optimized for average case I latency my vary and may not be bounded 4 Why (Vanilla) Linux is not Real-time (bird's eye view) I general-purpose I high functionality I optimized for average case I latency my vary and may not be bounded (even with patch, convenient Linux features should not be touched) 4 Why Linux is not Real-time (a closer look) I Applications run in user space I Hardware interaction is in kernel I kernel not preempted by user threads ) I event triggers high-priority thread I thread cannot preempt kernel ) large latency possible > 100ms I \classic" time sharing scheduling 5 CONFIG PREEMPT RT patch to improve RT behavior, \reduce special status" of kernel I kernel code outside of I spinlocks protected areas I interrupt handlers may be preempted I WC latency drops to (single digit) ms Unlike semaphores, spinlocks may be used in code that cannot sleep, such as interrupt handlers. When properly used, spinlocks offer higher performance than semaphores in general. 6 I citical sections protected by spinlock t and rwlock t are now preemptible. rwlock t allows multiple readers in critical section I priority inheritance for in-kernel spinlocks and semaphores. I converting interrupt handlers into preemptible kernel threads I user space POSIX timers with high resolution. How does it work? [faq07] The patch converts Linux into a fully preemptible kernel: I in-kernel locking-primitives (using spinlocks) preemptible through reimplementation with rtmutexes. RT-mutexes extend the semantics of simple mutexes by the priority inheritance protocol 7 I converting interrupt handlers into preemptible kernel threads I user space POSIX timers with high resolution. How does it work? [faq07] The patch converts Linux into a fully preemptible kernel: I in-kernel locking-primitives (using spinlocks) preemptible through reimplementation with rtmutexes. RT-mutexes extend the semantics of simple mutexes by the priority inheritance protocol I citical sections protected by spinlock t and rwlock t are now preemptible. rwlock t allows multiple readers in critical section I priority inheritance for in-kernel spinlocks and semaphores. 7 I user space POSIX timers with high resolution. How does it work? [faq07] The patch converts Linux into a fully preemptible kernel: I in-kernel locking-primitives (using spinlocks) preemptible through reimplementation with rtmutexes. RT-mutexes extend the semantics of simple mutexes by the priority inheritance protocol I citical sections protected by spinlock t and rwlock t are now preemptible. rwlock t allows multiple readers in critical section I priority inheritance for in-kernel spinlocks and semaphores. I converting interrupt handlers into preemptible kernel threads 7 How does it work? [faq07] The patch converts Linux into a fully preemptible kernel: I in-kernel locking-primitives (using spinlocks) preemptible through reimplementation with rtmutexes. RT-mutexes extend the semantics of simple mutexes by the priority inheritance protocol I citical sections protected by spinlock t and rwlock t are now preemptible. rwlock t allows multiple readers in critical section I priority inheritance for in-kernel spinlocks and semaphores. I converting interrupt handlers into preemptible kernel threads I user space POSIX timers with high resolution. 7 . and it is still Linux ) certain features must not be used. What sort of real-time? [faq07] it depends. I speed of the CPU and the architecture I device drivers I hardware e.g., DMA activity can introduce significant latencies e.g., some firmwares can stop the system for housekeeping (SMI, Service Management Interrupts) SMIs can not be trapped by the OS 8 What sort of real-time? [faq07] it depends. I speed of the CPU and the architecture I device drivers I hardware e.g., DMA activity can introduce significant latencies e.g., some firmwares can stop the system for housekeeping (SMI, Service Management Interrupts) SMIs can not be trapped by the OS . and it is still Linux ) certain features must not be used. 8 I Create all threads at startup time of the application i.e., never start threads dynamically during mission time I Touch each page of the entire stack of each thread I Never use system calls that are known to generate page faults, such as fopen(). opening files does mmap() system call ) page fault Guidelines for writing RT applications [faq07] I Call mlockall() as soon as possible from main(). causes all of the pages mapped by the address space of a process to be memory resident i.e., preventing that memory is paged to the swap area 9 I Never use system calls that are known to generate page faults, such as fopen(). opening files does mmap() system call ) page fault Guidelines for writing RT applications [faq07] I Call mlockall() as soon as possible from main(). causes all of the pages mapped by the address space of a process to be memory resident i.e., preventing that memory is paged to the swap area I Create all threads at startup time of the application i.e., never start threads dynamically during mission time I Touch each page of the entire stack of each thread 9 Guidelines for writing RT applications [faq07] I Call mlockall() as soon as possible from main(). causes all of the pages mapped by the address space of a process to be memory resident i.e., preventing that memory is paged to the swap area I Create all threads at startup time of the application i.e., never start threads dynamically during mission time I Touch each page of the entire stack of each thread I Never use system calls that are known to generate page faults, such as fopen(). opening files does mmap() system call ) page fault 9 Power management Watts vs. latency Hyperthreading and out-of-order execution performance vs. latency How to not build RT applications [how07] Don't use fancy stuff! Direct memory access I latency in microseconds by I SATA/PATA/SCSI devices I network adapters I video cards I can sometimes be controlled by device drivers throughput vs. latency I this problem independent of OS 10 Hyperthreading and out-of-order execution performance vs. latency How to not build RT applications [how07] Don't use fancy stuff! Direct memory access I latency in microseconds by I SATA/PATA/SCSI devices I network adapters I video cards I can sometimes be controlled by device drivers throughput vs. latency I this problem independent of OS Power management Watts vs. latency 10 How to not build RT applications [how07] Don't use fancy stuff! Direct memory access I latency in microseconds by I SATA/PATA/SCSI devices I network adapters I video cards I can sometimes be controlled by device drivers throughput vs. latency I this problem independent of OS Power management Watts vs. latency Hyperthreading and out-of-order execution performance vs. latency 10 I take care of page faults (swapping) during start-up I Call directly from the main() entry the mlockall() call I Create all threads at startup time of the application. I Reserve a pool of memory to do new/delete or malloc/free I Never use system calls that are known to generate pagefaults. I do not configure application with priority 99 some watchdog threads need higher priority than appl. I use mmap to pass data around More Guidelines. [how07] I do not use VGA text console very large latency I use serial console, ssh, telnet 11 I do not configure application with priority 99 some watchdog threads need higher priority than appl.