Real Time in Embedded Linux Systems

Real Time in Embedded Linux Systems Michael Opdenacker Free Electrons http://freeelectrons.com

Created with OpenOffice.org 2.0

Thanks to Nicolas Rougier (Copyright 2003, http://webloria.loria.fr/~rougier/) for the Tux image Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 1 Rights to copy

Attribution – ShareAlike 2.0 © Copyright 20042005 You are free Michael Opdenacker to copy, distribute, display, and perform the work michael@freeelectrons.com to make derivative works to make commercial use of the work Document sources, updates and translations: Under the following conditions http://freeelectrons.com/articles/realtime Attribution. You must give the original author credit. Corrections, suggestions, contributions and Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license translations are welcome! identical to this one. For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above. License text: http://creativecommons.org/licenses/bysa/2.0/legalcode

This document is best viewed with a recent PDF reader or with OpenOffice.org itself! Take advantage of internal or external hyperlinks. So, don’t hesitate to click on them! Find pages quickly thanks to automatic search Use thumbnails to navigate in the document in a quick way If you’re reading a paper or HTML copy, you should get your copy in PDF or OpenOffice.org format on http://freeelectrons.com/articles/realtime!

Introduction Reducing latency in Linux Linux hard realtime extensions Commercial Linux realtime distributions Conclusion References

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 4 Real Time in Embedded Linux Systems

Introduction

A system is considered as a hard real time if it can answer to an internal or external stimulus within a given maximum amount of time. “Guaranteed worst case” Hard real time systems are used wherever failing to react in time can cause a system failure or damage, or put its users in danger. Typical examples Industrial process control Transportation Medicine (pacemakers, etc.)

A system is considered as soft real time if it is built to react to stimuli as quickly as it can. “Best effort” However, if the system loses events or fails to process them in time, there is no catastrophic consequence on its operation. There is just a degradation in quality. Typical examples Audio, video Airline reservation systems

Linux cannot be considered as a hard real time system There are long sections of code where all interrupts are masked Kernel code (system calls) not preemptible until they complete or decide to release the processor by calling the scheduler.

A Linux system can be considered as soft real time if it can react within a few hundreds of µs in most cases. A Linux system can be considered as hard real time if it can react within a few tens of µs in all cases.

Linux 2.6 improves this by offering a preemptible option (CONFIG_PREEMPT)

Kernel code can be interrupted at almost any time (except when spinlocks are held)

This reduces latency for high priority processes

Implications for driver writers

Anything can happen between 2 instructions

Uniprocessor machines behave like SMP ones. No issue for clean drivers written with SMP in mind.

Caution with perCPU variables

Much more efficient scheduler Can be built with no virtual memory support. Note: lots of RTOS systems don't use an MMU, even if one is available in the CPU. MMU management can add too much overhead. Improved POSIX threads (now available in mainstream) POSIX signals can't be lost Make it easier to set task priorities and schedule periodic tasks with precision (useful for polling resources)

Linux 2.4 approach: add more scheduler calls throughout the code. Addressed by the preemptible option in 2.6. Voluntary preemption patch (Ingo Molnar) http://people.redhat.com/mingo/voluntarypreempt/ Reduces latency by calling cond_reschedule() in the might_sleep() calls (used for debugging). However, Andrew Morton argued that it's redundant with the preemptible option. See http://lwn.net/Articles/92806/ for details. August 2004: Some audio users still complain that Linux 2.6 latency is still not sufficient, even with the low latency patches.

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 12 Real Time in Embedded Linux Systems

Linux hard realtime extensions

First realtime extension for Linux

However, the author filed a software patent covering the addition of realtime support to general operating systems as implemented in RTLinux.

“RTLinux Open Patent (!) License”: 2 conditions for royaltyfree use of this license

Either using only GNU GPL software. No room for proprietary software or even other free software (BSD, Apache...) Or using an unmodified version of RTLinux. See http://www.fsmlabs.com/openpatentlicense.html FSF: RTLinux and distributing RTLinux violates the GPL license

Community unfriendly. Example: http://www.rtlinux.org/ and http://www.realtimelinux.org/ now redirect to http://www.fsmlabs.com/

The patent issue and uncertainty drew many developers away and frightened users. Linux RTAI attracts much more attention, developers and users.

Offers 2 products

RTLinuxPro: much more advertised. In particular, includes major rewrites to the original common code base. RTLinuxFree: doesn't include the RTLinuxPro improvements... See http://www.fsmlabs.com/rtlinuxfree.html If you need commercial RT distributions, better use the other ones (Real Free Software with no patent burden).

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 15 The RTAI project http://www.rtai.org/ RealTime Application Interface for Linux Dipartimento di Ingegneria Aerospaziale Politecnico di Milano (DIAPM) Founded in 1999, by Prof. Paolo Montegazza (long time contributor to RTLinux). 1.x series for Linux 2.2 Supports Linux 2.6 since April 2004 (3.1 series)

Community project (mainly European contributors now). Large user base.

Excellent documentation: beginners guide, articles, conference pages, user and reference manuals...

Supported architectures (+ SMP support) x86, PowerPC, ARM (StrongARM; ARM7: clps711xfamily, Cirrus Logic EP7xxx, CS89712, PXA25x), MIPS, CRIS (ongoing)

License: LGPL (only requires RTAI source changes to be shared)

Modularity

Support for POSIX and other RTOS APIs (through Xenomai)

Hard realtime support in user space

User land Linux process Linux process (data acquisition) (GUI)

Realtime Kernel modules task 3 Realtime FIFO

Linux Kernel Realtime Realtime task 1 task 2 Linux interrupt handlers

Software interrupts Realtime kernel Hardware interrupts

Hardware. Interrupt Controller

RTAI adds a layer between the Linux kernel and the hardware The RTAI kernel manages realtime tasks according to their priorities. The Linux kernel is also a realtime task, with the lowest priority. All non realtime interrupts are handed out to the Linux kernel. Very good introduction: http://www.aero.polimi.it/~rtai/documentation/articles/guide.html

RTAI available as a kernel patch RTAI implemented through Linux kernel modules:

rtai: core module rtai_sched: realtime schedulers (3 available + other services) rtai_fifos: realtime fifos for communication between realtime and Linux user tasks. rtai_shm: sharing memory between realtime and Linux tasks lxrt: makes RTAI scheduler functions available to Linux rtai_pqueue, rtai_pthread, rtai_utils: POSIX RTAI modules Modular: you can only load the modules you need

Download RTAI Check the corresponding Linux version Download Linux Patch Linux with RTAI, compile it. Reboot Load the RTAI modules you need

/* rt_process.c */ int init_module(void) #include { #include RTIME tick_period; #include rt_set_periodic_mode(); #include rt_task_init(&rt_task, fun, 1, STACK_SIZE, TASK_PRIORITY, 1, 0); #include rtf_create(FIFO, 8000); #include tick_period = start_rt_timer(nano2count(TICK_PERIOD); #define TICK_PERIOD 1000000 rt_task_make_periodic(&rt_task, rt_get_time() + tick_period, tick_period; #define TASK_PRIORITY 1 return 0; #define STACK_SIZE 10000 } #define FIFO 0 void cleanup_module(void) { static RT_TASK rt_task; stop_rt_timer(); rtf_destroy(FIFO); static void fun(int t) rt_task_delete(&rt_task); { return; int counter = 0; } float sin_value;

while (1) { sin_value = sin(2*M_PI*1*rt_get_cpu_time_ns()/1E9); rtf_put(FIFO, &counter, sizeof(counter)); Realtime task rtf_put(FIFO, &sin_value, sizeof(sin_value)); counter++; Caution: written for RTAI 1.4. rt_task_wait_period(); } May not work with your version. }

/* scope.c */ #include /* Run */ #include #include sync #include insmod /home/rtai1.4/modules/rtai.o #include insmod /home/rtai1.4/modules/rtai_sched.o #include insmod /home/rtai1.4/modules/rtai_shm.o #include insmod /home/rtai1.4/modules/rtai_fifos.o static int end; insmod rt_process.o static void endme(int dummy) { end=1; } ./scope int main (void) rmmod rt_process { rmmod rtai_shm int fifo, counter; rmmod rtai_fifos float sin_value; rmmod rtai_sched if ((fifo = open("/dev/rtf0", O_RDONLY)) < 0) { rmmod rtai fprintf(stderr, "Error opening /dev/rtf0\n"); exit(1); } signal(SIGINT, endme); Execution script while (!end) { read(fifo, &counter, sizeof(counter)); read(fifo, &sin_value, sizeof(sin_value)); printf(" Counter : %d Seno : %f \n", counter, sin_value); } return 0; } Userland program

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 23 Adeos nanokernel http://home.gna.org/adeos/index.en.html Flexible environment to share hardware resources between several operating systems (or several instances of the same operating system). Used by RTAI as a replacement Hardware Abstraction Layer beneath the Linux kernel. Not impacted by the RTLinux patent! Also used for SMP clustering, patchless kernel debugging...

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 24 Xenomai project http://www.nongnu.org/xenomai/ Nanokernel mimicking APIs of traditional RTOS such as VxWorks, pSOS+, and VRTXsa. Can be plugged onto the Adeos layer, RTAI's RTHAL or the LinuxThreads POSIX interface. Goal: facilitate the porting of programs from traditional RTOS to RTAI on GNU / Linux. Now part of the RTAI project.

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 25 RealTime in Embedded Linux Systems

Commercial realtime distributions

Montavista Linux Professional Edition http://www.mvista.com/products/pro/realtime.html Employs some of the most active kernel hackers, in particular on the ARM platform All kernel development shared with the community kernel core and drivers (Linux 2.6 example: preemption option, many drivers...) O(1) fixed overhead / fixed priority scheduler for realtime running in front of the Linux scheduler. Handles realtime POSIX pthreads, and leaves the other to the standard scheduler.

Keeps the standard Linux API Emulates VxWorks and pSOS API Can be tried through a preview kit. Supports an impressive list of processors Tools to analyze performance and track down latency sources. Development tools seem to be proprietary

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 28 TimeSys http://timesys.com Exhaustive toolset as available from other vendors (need to check details on their site) Community friendly: share very interesting and generic technical whitepapers and articles. Free Software BSPs (Board Support Packages) available True soft to hard realtime OS product Development tools seem to be proprietary

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 29 LynuxWorks BlueCat http://www.lynuxworks.com/products/bluecat/

Complete Embedded Linux distribution and toolset Comprehensive processor support.

Based on Linux 2.6 with standard soft realtime improvements

Applications can run unmodified on LynuxWorks' LynxOS RTOS if hard real time is needed.

LynuxWorks: a lot of experience in traditional RTOS. Same development tools for BlueCat and LynxOS.

No visible contribution to the kernel. Looks like a traditional RTOS vendor trying to surf the Linux wave (could be wrong!)

Proprietary development tools

Metrowerks http://www.metrowerks.com/ Hardly mention Linux realtime support. More focus on their nonLinux RTOS offerings

LMBench: http://www.bitmover.com/lmbench/ Takes several latency measures on your system dbench: http://samba.org/ftp/tridge/dbench/README Produces filesystem load

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 32 RealTime in Embedded Linux Systems

Conclusion

System Latency Development complexity Simple Standard Linux No realtime: Standard Linux API 10100ms magnitude

Linux + patches Soft realtime: Applying patches BlueCat 100 µs magnitude Standard Linux API Montavista

Hard RT Linux Hard realtime: Applying patches RTAI ~10 µs magnitude Loading specific modules Montavista Implement drivers for POSIX or specific API Complex

Real Time in Embedded Linux Systems © Copyright 20042005, Michael Opdenacker Creative Commons AttributionShareAlike 2.0 license http://freeelectrons.com Nov 25, 2005 34 References http://www.realtimelinuxfoundation.org/ Community portal for realtime Linux

This document belongs to the more than 1000 page materials of an embedded GNU / Linux training from Free Electrons, available under a free documentation license. http://freeelectrons.com/training Linux on TI OMAP processors

Introduction to Unix and GNU/Linux Free Software development tools Embedded Linux kernel and driver development Introduction to uClinux Free Software tools for embedded Linux systems Realtime in embedded Linux systems Audio in embedded Linux systems What's new in Linux 2.6? Multimedia in embedded Linux systems Java in embedded Linux systems How to port Linux on a new PDA http://freeelectrons.com/articles

Embedded Linux optimizations Embedded Linux from Scratch... in 40 min!

If you support this work, you can help ... By sending corrections, suggestions, contributions and translations By asking your organization to order training sessions performed by the author of these documents (see http://freeelectrons.com/training) By speaking about it to your friends, colleagues and local Free Software community. By adding links to our online materials on your website, to increase their visibility in search engine results.

To the OpenOffice.org project, for their presentation and word processor tools which satisfied all my needs. To the Handhelds.org community, for giving me so much help and so many opportunities to help. To the members of the whole Free Software and Open Source community, for sharing the best of themselves: their work, their knowledge, their friendship. To people who sent corrections: Matti Aaltonen