LINUX AND REALTIME

1 PRESENTATION

Pierre Morel - MNIS Paris and Toulouse - France [email protected]

Linux Port on new architectures, Realtime and

OCERA european project on Realtime components in cooperation with UPVLC, Santa-Anna and Prague UWEP mexican/franco project for the development of an ARM in cooperation with Siteldi LHA IBM research project for application virtualization in Linux

2 LINUX AND REALTIME

Linux is not a Realtime "...and will never be" Linus Torwald

Possible solutions Modification of Linux core Interrupt virtualization and co-kernel Complete Virtualization and co-domain

3 WHY IS LINUX NOT REALTIME

4 RECALL: LINUX 2.6 / 3.X

Interrupt latency

Context switch latency

Scheduling

Memory handling

Spinlocks and preemption

5 INTERRUPT LATENCY

This is the time between receiving an interrupt and starting associated action

In realtime system the action is done by a realtime thread not the driver

Let see what is happening in

6 INTERRUPT LATENCY

7 INTERRUPT LATENCY

Tasklets and soft-interrupts may differ thread without any determinism including kernel threads

Network implementation is particularly bad for realtime Even with the NAPI, it makes heavy use of soft- interrupt and interrupt disabling

8 CONTEXT SWITCH LATENCY

With Linux 2.6 we have better preemptions politics

A good choice is essential to lower the latency and that is even for co-kernel adaptions as we will see later

9 NO PREEMPTION

10 VOLUNTARY PREEMPTION

11 PREEMPTIBLE KERNEL

12 SPINLOCKS

Spinlocks are used to synchronize data access from concurrent processor in SMP/SMT.

Due to kernel preemption the Linux implementation also disable preemption when a spinlock is locked

This of course increase context switch latency

The biggest trouble is caused by the Big Kernel Lock which only disappear after 2.6.38

13 LINUX SCHEDULER

Linux scheduler is O1 This means equal time to choose a task whatever the load is

Sadly, available policies are very primitive Rate Monotonic FIFO and RR (Round Robin) with 99 priority levels This is POSIX 1003.1b compliant

14 O1 SCHEDULER

15 PRIORITY INVERSION

"Handling of priority inversion may be solved by a good application programming" Victor Yodaiken (RTLinux)

There are also possibilities to handle this in the kernel itself or in the libraries.

What is it exactly? What really happened on mars? http://en.wikipedia.org/wiki/Priority_inversion

16 PRIORITY INVERSION

17 PRIORITY INVERSION

18 PRIORITY INVERSION

19 PRIORITY INVERSION

Two measures Priority ceiling priority inheritance

In the Linux kernel, only rt-mutex(9) handle priority inheritance. They are only used in the i2c driver...

In , GLIBC handle P.I. using priority ceiling: pthread_mutex_setprioceiling(3p)

20 MODIFYING LINUX

21 UNIVERSITY PROJECTS

Linux-RT (Thomas Gleixner) Free community project is open and GPL

ART-Linux Japanese project but is not open source

22 COMMERCIAL PROJECTS

SYSLinux (Thomas Gleixner) is partially open source and working in the same direction as Linux-RT.

Montavista Took also the same direction.

23 LINUX-RT

Tasklet and interrupt routines are re-implemented as threads

Spinlocks are modified to allow preemption

A new EDF scheduler is actually developed

24 THREADED INTERRUPTS

Interrupt routines are reimplemented as threads

You may modify the priorities of the threads with tools like chrt(8) or sched_setscheduler(2)

Some good examples for the optimization of audio threads: irqPriorities.html on linux-rt wiki

25 SPINLOCKS MODIFICATIONS

Linux vanilla spinlocks disable preemption when holding the lock.

This increase context switch latency from this duration

Linux-RT replace this by having more dedicated spinlocks and disabling migration in for SMP

BKL has been eliminated

26 EDF SCHEDULER

The Earliest Deadline First scheduler makes RT thread programming easier

It also allow a better CPU usage for periodic tasks

The implementation is done by University Santa Anna - Pisa (ReTIS Lab) and the Evidence spin-off

Not yet in the kernel because priority inversion is not handled (but encouraged by T. Gleixner)

27 ART-LINUX

ART-Linux is used in japanese robotic projects

Major change is done by changing standard interrupt handling implementation with interrupt poling.

Priority inheritance on kernel mutexes has been implemented

28 MEMORY HANDLING

Take care of COW and lazy allocations bypass the Linux On demand Paging by using mmap(2) and MAP_POPULATE flag pre-fault memory for stack and text code Lock memory so no swap occurs using lock(2)

You may also use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free.

29 CO-KERNEL AND INTERRUPT VIRTUALIZATION

30 CO-KERNEL

RT-Linux: Victor Yodaiken / Barbanov (FSMLabs)

RTAI: Paolo Mantegazza (Aero-Polimiti Milano)

Xenomai: Philippe Gerum

31 INTERRUPT VIRTUALIZATION

32 INTERRUPT VIRTUALIZATION

33 INTERRUPT VIRTUALIZATION

34 SECURITY AND MEMORY ACCESS

In this implementation both co-kernel shared the same memory space

Realtime applications also share the same memory space

No protection against faults from Linux or from RTOS

35 ADEOS

Adaptive Domain Environment for Operating Systems Karim Yagmour - Opersys who also done L.T.T.

Virtualization of events - especially interruptions

36 ADEOS

37 ADEOS

38 ADEOS

39 ADEOS

40 ADEOS

41 /RTAI LXR

42 SECURITY AND MEMORY ACCESS

At least in this implementation, realtime applications and standard Linux applications are protected against faults.

But.. both kernel shared the same space

43 SCHEDULING

By design, Xenomai scheduling must be full compatible with Linux scheduling

Only SCHED_FIFO and SCHED_RR are supported

44 SCHEDULING LINUX

45 SCHEDULING XENOMAI

46 SCHEDULING LATENCY

Scheduling latency is minimal with the usage of in kernel realtime tasks.

For users space threads, the maximal scheduling latency is obtain when a Xenomai thread entered Linux scheduler until it reaches a preemption point.

47 MEMORY HANDLING

In kernel memory allocation using standard Linux memory must on start. For example in _init() function of the RTLinux module

You must use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free. TLSF is implemented by Universidad Politecnica de Valencia

48 MEMORY HANDLING

In user space take care of COW and lazy allocations bypass the Linux On demand Paging by using mmap(2) and MAP_POPULATE flag pre-fault memory for stack and text code Lock memory so no swap occurs using lock(2)

You may also use a dedicated memory management like (TLSF) if you want to use dynamic alloc/free.

49 USEFUL FEATURES

Emulation of different realtime operating systems VxWorks, PSOS+, VRTX, UITRON, POSIX, RTAI

Comedi, a librairy of Analogic Digital converters

Analogy, a library of sensors drivers

RTNET, real time ethernet using TDMA

50 CONCLUSION

Pro responsiveness, low latency Integration of RTNET RTOS emulations

Contra: No good scheduler, only Linux schedulers

51 MEASURE AND TESTS

52 REAL-TIME

53 HYPERVISORS

54 HYPERVISORS

Two types type 1: bare metal (, XtratuM, L4, ESX...) type 2: OS based (, )

Three families micro-kernels (L4) complete virtualization (xen) para-virtualization (xen, ESX, XtratuM)

55 HYPERVISORS

Main features Spatial isolation, MMU virtualization CPU and Interrupt virtualization Peripherals virtualization Inter domain Communication

Applications Security, cryptographic applications isolation System mobility, HAC, HPC

56 MMU

57 MMU VIRTUALIZATION

58 CPU VIRTUALIZATION end of 60th: IBM S360 first

Popeck and Goldberg define a virtualizable CPU CPU has two modes: normal and privileged "sensitive control instructions and sensitive behaving instruction must be privileged instructions" Privileged instructions in normal mode are trapped Most CPU respect this more or less but Intel CPU 2005: with VTX intel gains CPU virtualization

59 PRE VIRTUALIZATION

60 WHAT IF THE CPU CAN NOT BE VIRTUALIZED

Para virtualization The source code is modified Pre virtualization The binary of the program is modified Binary rewriting The binary of the process is modified

61 PARA VIRTUALIZATION

62 PRE VIRTUALIZATION

63 GENERAL PURPOSE HYPERVISORS

Qemu Qemu is also an emulator supporting ARM, MIPS, SH4, PowerPC...

KVM, Virtualbox and Xen use Qemu

Vmware, HyperV

IBM, SUN and HP have their own solutions used for High Availabilities and High Performance Computing

64 SPATIAL ISOLATION

65 REAL-TIME HYPERVISORS

Temporal isolation a domain can not delay the execution of another domain

Hypervisor scheduler adds determinism, EDF, ARINC

May or may not allow more than one RTOS

66 TEMPORAL ISOLATION

67 REAL-TIME HYPERVISORS

VLX (Virtual Logix / Red Bend)

L4 is a german project from Dresden and Karlsruhe OKL4 and Sysgo hypervisor are based on L4

XtratuM is a spanish project from Valencia

Other projects: WindRiver, Freescale, Xen, Trangon/Vmware ...

68 XEN AS A REAL-TIME HYPERVISORS

Corean Project - Chuck Yoo If all domains are scheduled by EDF schedulers and the Hypervisor uses EDF algorithm

The tasks within each domain will be scheduled in deterministic manner.

69 DRIVER IMPLEMENTATION

A domain has access to his own peripheral (VLX, XtratuM)

A domain is dedicated as driver (L4, Xtratum)

A domain is dedicated for all drivers (Xen)

70 DRIVER DOMAIN

71 COMPARING THE SOLUTIONS

Next slides shows a comparison between the different solutions and where the different services are implemented: shared or private in user space, kernel space, domain or hypervisor

72 Linux Task Linux Task Linux syscall interface

Scheduler

IO Logic Tasklets IRQ routine

Do_IRQ

IRQ Vectors

CPU

Device PIC Device

73 Linux Task RT Task Linux-RT syscall interface

IRQ thread Tasklets thread

Scheduler IO Logic

IRQ Vectors

CPU

Device PIC Device

74 Linux Task RT-Linux syscall interface

Scheduler RT Task Tasklets IO Logic IRQ routine Scheduler

IRQ routine IO Logic

Do_IRQ IRQ logic

IRQ Vectors

CPU

Device PIC Device

75 RTAI / Linux Task syscall interface RT Task Xenomai Scheduler Mem Mgt Scheduler Tasklets RT Task

IO Logic IRQ routine IRQ routine Do_IRQ IRQ logic IO Logic IRQ Vectors

ADEOS - iPipe IRQ logic IRQ Vectors CPU Device PIC Device

76 Hypervisor

Linux Task RT Task syscall interface Driver Scheduler Scheduler Mem Mgt

Mem Mgt IRQ routine

Tasklets IO Logic IO Logic IO Logic IRQ logic IRQ logic IRQ routine

hypercall interface

Syscall Management Memory Management Scheduler IRQ logic

Device MMU CPU PIC

77 Micro-kernel Driver RT Task

Scheduler Scheduler Mem Mgt Mem Mgt IO Logic IRQ routine Tasklets Linux Task IRQ logic IO Logic IO Logic

IRQ logic IRQ routine

syscall interface hypercall interface

Syscall Management MemoryMem. Management Management Scheduler IRQ logic

Device MMU CPU PIC

78 THE END

Thank you ! Any questions ?

79