LINUX AND REALTIME 1 PRESENTATION Pierre Morel - MNIS Paris and Toulouse - France [email protected] Linux Port on new architectures, Realtime and Virtualization OCERA european project on Realtime components in cooperation with UPVLC, Santa-Anna and Prague UWEP mexican/franco project for the development of an ARM Hypervisor in cooperation with Siteldi LHA IBM research project for application virtualization in Linux 2 LINUX AND REALTIME Linux is not a Realtime Operating System "...and will never be" Linus Torwald Possible solutions Modification of Linux core Interrupt virtualization and co-kernel Complete Virtualization and co-domain 3 WHY IS LINUX NOT REALTIME 4 RECALL: LINUX 2.6 / 3.X Interrupt latency Context switch latency Scheduling Memory handling Spinlocks and preemption 5 INTERRUPT LATENCY This is the time between receiving an interrupt and starting associated action In realtime system the action is done by a realtime thread not the driver Let see what is happening in Linux kernel 6 INTERRUPT LATENCY 7 INTERRUPT LATENCY Tasklets and soft-interrupts may differ thread without any determinism including kernel threads Network implementation is particularly bad for realtime Even with the NAPI, it makes heavy use of soft- interrupt and interrupt disabling 8 CONTEXT SWITCH LATENCY With Linux 2.6 we have better preemptions politics A good choice is essential to lower the latency and that is even for co-kernel adaptions as we will see later 9 NO PREEMPTION 10 VOLUNTARY PREEMPTION 11 PREEMPTIBLE KERNEL 12 SPINLOCKS Spinlocks are used to synchronize data access from concurrent processor in SMP/SMT. Due to kernel preemption the Linux implementation also disable preemption when a spinlock is locked This of course increase context switch latency The biggest trouble is caused by the Big Kernel Lock which only disappear after 2.6.38 13 LINUX SCHEDULER Linux scheduler is O1 This means equal time to choose a task whatever the load is Sadly, available scheduling policies are very primitive Rate Monotonic FIFO and RR (Round Robin) with 99 priority levels This is POSIX 1003.1b compliant 14 O1 SCHEDULER 15 PRIORITY INVERSION "Handling of priority inversion may be solved by a good application programming" Victor Yodaiken (RTLinux) There are also possibilities to handle this in the kernel itself or in the libraries. What is it exactly? What really happened on mars? http://en.wikipedia.org/wiki/Priority_inversion 16 PRIORITY INVERSION 17 PRIORITY INVERSION 18 PRIORITY INVERSION 19 PRIORITY INVERSION Two measures Priority ceiling priority inheritance In the Linux kernel, only rt-mutex(9) handle priority inheritance. They are only used in the i2c driver... In user space, GLIBC handle P.I. using priority ceiling: pthread_mutex_setprioceiling(3p) 20 MODIFYING LINUX 21 UNIVERSITY PROJECTS Linux-RT (Thomas Gleixner) Free community project is open and GPL ART-Linux Japanese project but is not open source 22 COMMERCIAL PROJECTS SYSLinux (Thomas Gleixner) is partially open source and working in the same direction as Linux-RT. Montavista Took also the same direction. 23 LINUX-RT Tasklet and interrupt routines are re-implemented as threads Spinlocks are modified to allow preemption A new EDF scheduler is actually developed 24 THREADED INTERRUPTS Interrupt routines are reimplemented as threads You may modify the priorities of the threads with tools like chrt(8) or sched_setscheduler(2) Some good examples for the optimization of audio threads: irqPriorities.html on linux-rt wiki 25 SPINLOCKS MODIFICATIONS Linux vanilla spinlocks disable preemption when holding the lock. This increase context switch latency from this duration Linux-RT replace this by having more dedicated spinlocks and disabling migration in for SMP BKL has been eliminated 26 EDF SCHEDULER The Earliest Deadline First scheduler makes RT thread programming easier It also allow a better CPU usage for periodic tasks The implementation is done by University Santa Anna - Pisa (ReTIS Lab) and the Evidence spin-off Not yet in the kernel because priority inversion is not handled (but encouraged by T. Gleixner) 27 ART-LINUX ART-Linux is used in japanese robotic projects Major change is done by changing standard interrupt handling implementation with interrupt poling. Priority inheritance on kernel mutexes has been implemented 28 MEMORY HANDLING Take care of COW and lazy allocations bypass the Linux On demand Paging by using mmap(2) and MAP_POPULATE flag pre-fault memory for stack and text code Lock memory so no swap occurs using lock(2) You may also use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free. 29 CO-KERNEL AND INTERRUPT VIRTUALIZATION 30 CO-KERNEL RT-Linux: Victor Yodaiken / Barbanov (FSMLabs) RTAI: Paolo Mantegazza (Aero-Polimiti Milano) Xenomai: Philippe Gerum 31 INTERRUPT VIRTUALIZATION 32 INTERRUPT VIRTUALIZATION 33 INTERRUPT VIRTUALIZATION 34 SECURITY AND MEMORY ACCESS In this implementation both co-kernel shared the same memory space Realtime applications also share the same memory space No protection against faults from Linux or from RTOS 35 ADEOS Adaptive Domain Environment for Operating Systems Karim Yagmour - Opersys who also done L.T.T. Virtualization of events - especially interruptions 36 ADEOS 37 ADEOS 38 ADEOS 39 ADEOS 40 ADEOS 41 XENOMAI/RTAI LXR 42 SECURITY AND MEMORY ACCESS At least in this implementation, realtime applications and standard Linux applications are protected against faults. But.. both kernel shared the same space 43 SCHEDULING By design, Xenomai scheduling must be full compatible with Linux scheduling Only SCHED_FIFO and SCHED_RR are supported 44 SCHEDULING LINUX 45 SCHEDULING XENOMAI 46 SCHEDULING LATENCY Scheduling latency is minimal with the usage of in kernel realtime tasks. For users space threads, the maximal scheduling latency is obtain when a Xenomai thread entered Linux scheduler until it reaches a preemption point. 47 MEMORY HANDLING In kernel memory allocation using standard Linux memory must on start. For example in _init() function of the RTLinux module You must use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free. TLSF is implemented by Universidad Politecnica de Valencia 48 MEMORY HANDLING In user space take care of COW and lazy allocations bypass the Linux On demand Paging by using mmap(2) and MAP_POPULATE flag pre-fault memory for stack and text code Lock memory so no swap occurs using lock(2) You may also use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free. 49 USEFUL FEATURES Emulation of different realtime operating systems VxWorks, PSOS+, VRTX, UITRON, POSIX, RTAI Comedi, a librairy of Analogic Digital converters Analogy, a library of sensors drivers RTNET, real time ethernet using TDMA 50 CONCLUSION Pro responsiveness, low latency Integration of RTNET RTOS emulations Contra: No good scheduler, only Linux schedulers 51 MEASURE AND TESTS 52 REAL-TIME HYPERVISORS 53 HYPERVISORS 54 HYPERVISORS Two types type 1: bare metal (xen, XtratuM, L4, ESX...) type 2: OS based (qemu, vmware workstation) Three families micro-kernels (L4) complete virtualization (xen) para-virtualization (xen, ESX, XtratuM) 55 HYPERVISORS Main features Spatial isolation, MMU virtualization CPU and Interrupt virtualization Peripherals virtualization Inter domain Communication Applications Security, cryptographic applications isolation System mobility, HAC, HPC 56 MMU 57 MMU VIRTUALIZATION 58 CPU VIRTUALIZATION end of 60th: IBM S360 first virtual machine Popeck and Goldberg define a virtualizable CPU CPU has two modes: normal and privileged "sensitive control instructions and sensitive behaving instruction must be privileged instructions" Privileged instructions in normal mode are trapped Most CPU respect this more or less but Intel CPU 2005: with VTX intel gains CPU virtualization 59 PRE VIRTUALIZATION 60 WHAT IF THE CPU CAN NOT BE VIRTUALIZED Para virtualization The source code is modified Pre virtualization The binary of the program is modified Binary rewriting The binary of the process is modified 61 PARA VIRTUALIZATION 62 PRE VIRTUALIZATION 63 GENERAL PURPOSE HYPERVISORS Qemu Qemu is also an emulator supporting ARM, MIPS, SH4, PowerPC... KVM, Virtualbox and Xen use Qemu Vmware, HyperV IBM, SUN and HP have their own solutions used for High Availabilities and High Performance Computing 64 SPATIAL ISOLATION 65 REAL-TIME HYPERVISORS Temporal isolation a domain can not delay the execution of another domain Hypervisor scheduler adds determinism, EDF, ARINC May or may not allow more than one RTOS 66 TEMPORAL ISOLATION 67 REAL-TIME HYPERVISORS VLX (Virtual Logix / Red Bend) L4 is a german project from Dresden and Karlsruhe OKL4 and Sysgo hypervisor are based on L4 XtratuM is a spanish project from Valencia Other projects: WindRiver, Freescale, Xen, Trangon/Vmware ... 68 XEN AS A REAL-TIME HYPERVISORS Corean Project - Chuck Yoo If all domains are scheduled by EDF schedulers and the Hypervisor uses EDF algorithm The tasks within each domain will be scheduled in deterministic manner. 69 DRIVER IMPLEMENTATION A domain has access to his own peripheral (VLX, XtratuM) A domain is dedicated as driver (L4, Xtratum) A domain is dedicated for all drivers (Xen) 70 DRIVER DOMAIN 71 COMPARING THE SOLUTIONS Next slides shows a comparison between the different solutions and where the different services are implemented: shared or private in user space, kernel space, domain or hypervisor 72 Linux Task Linux Task Linux syscall interface Scheduler IO Logic Tasklets IRQ routine Do_IRQ IRQ Vectors CPU Device PIC