LINUX AND REALTIME
1 PRESENTATION
Pierre Morel - MNIS Paris and Toulouse - France [email protected]
Linux Port on new architectures, Realtime and Virtualization
OCERA european project on Realtime components in cooperation with UPVLC, Santa-Anna and Prague UWEP mexican/franco project for the development of an ARM Hypervisor in cooperation with Siteldi LHA IBM research project for application virtualization in Linux
2 LINUX AND REALTIME
Linux is not a Realtime Operating System "...and will never be" Linus Torwald
Possible solutions Modification of Linux core Interrupt virtualization and co-kernel Complete Virtualization and co-domain
3 WHY IS LINUX NOT REALTIME
4 RECALL: LINUX 2.6 / 3.X
Interrupt latency
Context switch latency
Scheduling
Memory handling
Spinlocks and preemption
5 INTERRUPT LATENCY
This is the time between receiving an interrupt and starting associated action
In realtime system the action is done by a realtime thread not the driver
Let see what is happening in Linux kernel
6 INTERRUPT LATENCY
7 INTERRUPT LATENCY
Tasklets and soft-interrupts may differ thread without any determinism including kernel threads
Network implementation is particularly bad for realtime Even with the NAPI, it makes heavy use of soft- interrupt and interrupt disabling
8 CONTEXT SWITCH LATENCY
With Linux 2.6 we have better preemptions politics
A good choice is essential to lower the latency and that is even for co-kernel adaptions as we will see later
9 NO PREEMPTION
10 VOLUNTARY PREEMPTION
11 PREEMPTIBLE KERNEL
12 SPINLOCKS
Spinlocks are used to synchronize data access from concurrent processor in SMP/SMT.
Due to kernel preemption the Linux implementation also disable preemption when a spinlock is locked
This of course increase context switch latency
The biggest trouble is caused by the Big Kernel Lock which only disappear after 2.6.38
13 LINUX SCHEDULER
Linux scheduler is O1 This means equal time to choose a task whatever the load is
Sadly, available scheduling policies are very primitive Rate Monotonic FIFO and RR (Round Robin) with 99 priority levels This is POSIX 1003.1b compliant
14 O1 SCHEDULER
15 PRIORITY INVERSION
"Handling of priority inversion may be solved by a good application programming" Victor Yodaiken (RTLinux)
There are also possibilities to handle this in the kernel itself or in the libraries.
What is it exactly? What really happened on mars? http://en.wikipedia.org/wiki/Priority_inversion
16 PRIORITY INVERSION
17 PRIORITY INVERSION
18 PRIORITY INVERSION
19 PRIORITY INVERSION
Two measures Priority ceiling priority inheritance
In the Linux kernel, only rt-mutex(9) handle priority inheritance. They are only used in the i2c driver...
In user space, GLIBC handle P.I. using priority ceiling: pthread_mutex_setprioceiling(3p)
20 MODIFYING LINUX
21 UNIVERSITY PROJECTS
Linux-RT (Thomas Gleixner) Free community project is open and GPL
ART-Linux Japanese project but is not open source
22 COMMERCIAL PROJECTS
SYSLinux (Thomas Gleixner) is partially open source and working in the same direction as Linux-RT.
Montavista Took also the same direction.
23 LINUX-RT
Tasklet and interrupt routines are re-implemented as threads
Spinlocks are modified to allow preemption
A new EDF scheduler is actually developed
24 THREADED INTERRUPTS
Interrupt routines are reimplemented as threads
You may modify the priorities of the threads with tools like chrt(8) or sched_setscheduler(2)
Some good examples for the optimization of audio threads: irqPriorities.html on linux-rt wiki
25 SPINLOCKS MODIFICATIONS
Linux vanilla spinlocks disable preemption when holding the lock.
This increase context switch latency from this duration
Linux-RT replace this by having more dedicated spinlocks and disabling migration in for SMP
BKL has been eliminated
26 EDF SCHEDULER
The Earliest Deadline First scheduler makes RT thread programming easier
It also allow a better CPU usage for periodic tasks
The implementation is done by University Santa Anna - Pisa (ReTIS Lab) and the Evidence spin-off
Not yet in the kernel because priority inversion is not handled (but encouraged by T. Gleixner)
27 ART-LINUX
ART-Linux is used in japanese robotic projects
Major change is done by changing standard interrupt handling implementation with interrupt poling.
Priority inheritance on kernel mutexes has been implemented
28 MEMORY HANDLING
Take care of COW and lazy allocations bypass the Linux On demand Paging by using mmap(2) and MAP_POPULATE flag pre-fault memory for stack and text code Lock memory so no swap occurs using lock(2)
You may also use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free.
29 CO-KERNEL AND INTERRUPT VIRTUALIZATION
30 CO-KERNEL
RT-Linux: Victor Yodaiken / Barbanov (FSMLabs)
RTAI: Paolo Mantegazza (Aero-Polimiti Milano)
Xenomai: Philippe Gerum
31 INTERRUPT VIRTUALIZATION
32 INTERRUPT VIRTUALIZATION
33 INTERRUPT VIRTUALIZATION
34 SECURITY AND MEMORY ACCESS
In this implementation both co-kernel shared the same memory space
Realtime applications also share the same memory space
No protection against faults from Linux or from RTOS
35 ADEOS
Adaptive Domain Environment for Operating Systems Karim Yagmour - Opersys who also done L.T.T.
Virtualization of events - especially interruptions
36 ADEOS
37 ADEOS
38 ADEOS
39 ADEOS
40 ADEOS
41 XENOMAI/RTAI LXR
42 SECURITY AND MEMORY ACCESS
At least in this implementation, realtime applications and standard Linux applications are protected against faults.
But.. both kernel shared the same space
43 SCHEDULING
By design, Xenomai scheduling must be full compatible with Linux scheduling
Only SCHED_FIFO and SCHED_RR are supported
44 SCHEDULING LINUX
45 SCHEDULING XENOMAI
46 SCHEDULING LATENCY
Scheduling latency is minimal with the usage of in kernel realtime tasks.
For users space threads, the maximal scheduling latency is obtain when a Xenomai thread entered Linux scheduler until it reaches a preemption point.
47 MEMORY HANDLING
In kernel memory allocation using standard Linux memory must on start. For example in _init() function of the RTLinux module
You must use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free. TLSF is implemented by Universidad Politecnica de Valencia
48 MEMORY HANDLING
In user space take care of COW and lazy allocations bypass the Linux On demand Paging by using mmap(2) and MAP_POPULATE flag pre-fault memory for stack and text code Lock memory so no swap occurs using lock(2)
You may also use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free.
49 USEFUL FEATURES
Emulation of different realtime operating systems VxWorks, PSOS+, VRTX, UITRON, POSIX, RTAI
Comedi, a librairy of Analogic Digital converters
Analogy, a library of sensors drivers
RTNET, real time ethernet using TDMA
50 CONCLUSION
Pro responsiveness, low latency Integration of RTNET RTOS emulations
Contra: No good scheduler, only Linux schedulers
51 MEASURE AND TESTS
52 REAL-TIME HYPERVISORS
53 HYPERVISORS
54 HYPERVISORS
Two types type 1: bare metal (xen, XtratuM, L4, ESX...) type 2: OS based (qemu, vmware workstation)
Three families micro-kernels (L4) complete virtualization (xen) para-virtualization (xen, ESX, XtratuM)
55 HYPERVISORS
Main features Spatial isolation, MMU virtualization CPU and Interrupt virtualization Peripherals virtualization Inter domain Communication
Applications Security, cryptographic applications isolation System mobility, HAC, HPC
56 MMU
57 MMU VIRTUALIZATION
58 CPU VIRTUALIZATION end of 60th: IBM S360 first virtual machine
Popeck and Goldberg define a virtualizable CPU CPU has two modes: normal and privileged "sensitive control instructions and sensitive behaving instruction must be privileged instructions" Privileged instructions in normal mode are trapped Most CPU respect this more or less but Intel CPU 2005: with VTX intel gains CPU virtualization
59 PRE VIRTUALIZATION
60 WHAT IF THE CPU CAN NOT BE VIRTUALIZED
Para virtualization The source code is modified Pre virtualization The binary of the program is modified Binary rewriting The binary of the process is modified
61 PARA VIRTUALIZATION
62 PRE VIRTUALIZATION
63 GENERAL PURPOSE HYPERVISORS
Qemu Qemu is also an emulator supporting ARM, MIPS, SH4, PowerPC...
KVM, Virtualbox and Xen use Qemu
Vmware, HyperV
IBM, SUN and HP have their own solutions used for High Availabilities and High Performance Computing
64 SPATIAL ISOLATION
65 REAL-TIME HYPERVISORS
Temporal isolation a domain can not delay the execution of another domain
Hypervisor scheduler adds determinism, EDF, ARINC
May or may not allow more than one RTOS
66 TEMPORAL ISOLATION
67 REAL-TIME HYPERVISORS
VLX (Virtual Logix / Red Bend)
L4 is a german project from Dresden and Karlsruhe OKL4 and Sysgo hypervisor are based on L4
XtratuM is a spanish project from Valencia
Other projects: WindRiver, Freescale, Xen, Trangon/Vmware ...
68 XEN AS A REAL-TIME HYPERVISORS
Corean Project - Chuck Yoo If all domains are scheduled by EDF schedulers and the Hypervisor uses EDF algorithm
The tasks within each domain will be scheduled in deterministic manner.
69 DRIVER IMPLEMENTATION
A domain has access to his own peripheral (VLX, XtratuM)
A domain is dedicated as driver (L4, Xtratum)
A domain is dedicated for all drivers (Xen)
70 DRIVER DOMAIN
71 COMPARING THE SOLUTIONS
Next slides shows a comparison between the different solutions and where the different services are implemented: shared or private in user space, kernel space, domain or hypervisor
72 Linux Task Linux Task Linux syscall interface
Scheduler
IO Logic Tasklets IRQ routine
Do_IRQ
IRQ Vectors
CPU
Device PIC Device
73 Linux Task RT Task Linux-RT syscall interface
IRQ thread Tasklets thread
Scheduler IO Logic
IRQ Vectors
CPU
Device PIC Device
74 Linux Task RT-Linux syscall interface
Scheduler RT Task Tasklets IO Logic IRQ routine Scheduler
IRQ routine IO Logic
Do_IRQ IRQ logic
IRQ Vectors
CPU
Device PIC Device
75 RTAI / Linux Task syscall interface RT Task Xenomai Scheduler Mem Mgt Scheduler Tasklets RT Task
IO Logic IRQ routine IRQ routine Do_IRQ IRQ logic IO Logic IRQ Vectors
ADEOS - iPipe IRQ logic IRQ Vectors CPU Device PIC Device
76 Hypervisor
Linux Task RT Task syscall interface Driver Scheduler Scheduler Mem Mgt
Mem Mgt IRQ routine
Tasklets IO Logic IO Logic IO Logic IRQ logic IRQ logic IRQ routine
hypercall interface
Syscall Management Memory Management Scheduler IRQ logic
Device MMU CPU PIC
77 Micro-kernel Driver RT Task
Scheduler Scheduler Mem Mgt Mem Mgt IO Logic IRQ routine Tasklets Linux Task IRQ logic IO Logic IO Logic
IRQ logic IRQ routine
syscall interface hypercall interface
Syscall Management MemoryMem. Management Management Scheduler IRQ logic
Device MMU CPU PIC
78 THE END
Thank you ! Any questions ?
79