EMBEDDED

Native mainline Linux: fit for embedded and real-time systems

By Carsten Emde, Automation Development Lab, and Thomas Gleixner, Linutronix

When Linux was born on August 1, 1991, nobody expected that 15 years later the scale of supported hardware would range from tiny micro - controllers up to large multi - processor mainframes, or that Linux would become a real-time .

Figure 1. Linux Fedora 7 system running a total of seven virtual systems simultaneously

I What is real-time and how is it measured? As start of the interrupt service routine is called tion by respondents planning to use Linux was generally agreed, it is not an average fast re - “interrupt latency”, and the interval between the the item “real-time limitations”. At this time, it sponse that makes a system real-time compliant input trigger and the execution of the user was a common understanding that a real-time but the reliability that a defined response in - space program is called “overall latency”, “pre- operating system must be conceived from the terval is not exceeded under whatever circum - emption latency”, or simply “latency” (figure 2). very beginning as such and that every single stances. A system, for example, that reliably Normally, a large number of single measure - component must be designed with all aspects of responds to external signals within 30 minutes ments is taken over several hours while gener - real-time execution in mind. Thus, retrofitting is a real-time system whereas another system, ating a realistic system load, and the longest real-time capabilities into the native mainline that relies on a timely response within 30 mi - measured overall latency is taken as the final re - Linux was deemed impossible. As a conse - croseconds but fails once in a year, is not. sult. This is then called “worst-case latency”. To quence, several Linux real-time projects were gain a better overview, the data points are best started that used the dual-kernel approach. This In order to determine the real-time capabilities displayed graphically in a latency histogram approach replaces the by a small of a system, an external signal is applied re - (figure 3). real-time kernel to which all interrupts are sent peatedly to a controller input such as a digital and which controls the entire system. One of its input line or a handshake line of a serial or par - Initially, the Linux operating system was adopt - tasks is a modified Linux kernel; this enables ex - allel communication controller. The analysis re - ed primarily as a server platform - probably due ecution of Linux programs and thus quires that a few lines of code are inserted into to its support of generally available and inex - provides binary Linux compatibility of the en - the beginning of the respective interrupt serv - pensive hardware in combination with its su - tire system (figure 4). ice routine and into the initial part of a user perior network capabilities and run-time space program waiting for this interrupt. These stability. At a later stage, support for architec - Examples of solutions using this approach are lines of codes are written in such a way that a tures other than x86 was added so the use of RTLinux, RTAI, and ; they have been pulse is generated at an output device. This Linux could have been extended immediately to used repeatedly and successfully in recent years pulse can then be measured along with the embedded systems. However, embedded sys - in a large number of industrial applications. input trigger using an oscilloscope. In addition, tems often require real-time capabilities which RTLinux, RTAI, and Xenomai offer, similarly to the respective interrupt line is also monitored. Linux as a server operating system was never in - the RT-Preempt patches, a POSIX compatible tended to provide. In fact, in a 2001 survey pub - API to access real-time related functionality. The interval between the input trigger and the lished in April 2002 as a Venture Development Xenomai, in addition, comes with a collection of corresponding level change of the interrupt line Corporation Whitepaper (http://linuxde - other APIs (called “skins”). They greatly facili - is called “gate latency” or “controller latency”. vices.com/articles/AT6328992055.html), the tate the migration from proprietary real-time The interval between the input trigger and the most important factor inhibiting Linux adop - operating systems. In the foreseeable future, it is

June 2007 2 EMBEDDED LINUX

The other features, such as PI mutexes, high-res - olution timers and interrupt threading, were only available when the kernel was patched with the “RT-Preempt” downloaded from Ingo Molnar's repository at RedHat http://people.red - hat.com/mingo/realtime-preempt/. The various elements of this patch described above were de - veloped over the last couple of years by Ingo Mol - nar and Thomas Gleixner with support from many kernel developers worldwide. There was al - ways hope that the RT-Preempt patch would be integrated into mainline Linux one day. But con - vincing Torvalds that making Linux a real- time operating system would do no harm, but greatly enhance its usability, was not an easy task. Figure 2. The various elements of the overall or pre-emption latency Fortunately, there were an increasing number of arguments to make Linux real-time compliant - well conceivable that Xenomai will adopt the low-priority process executing a kernel function not only for industrial automation projects but also RT-Preempt approach explained below and under locking conditions may block a high-pri - for standard server and desktop Linux systems: provide additional functionality on top of the ority process that is about to execute the same  Audio and video mixing and recording real-time capable mainline Linux kernel. In par - kernel function. This phenomenon is called pri -  Reliable timestamps in financial and allel, the mainline Linux evolved as well. A first ority inversion, and one of the solutions is to commercial transactions important step was to replace the O(n) sched - bequeath the priority level to the mutex and let  Voice over IP and streaming video services. uler by Ingo Molnar's O(1) scheduler. This new it execute under the same priority as the calling scheduler was already merged into the devel - process. This special mutex variant that is At the occasion of the Ottawa kernel summit in opment Linux 2.5 kernel tree in 2002 and be - called Priority Inheritance Mutex (abbreviated July 2006, finally accepted came the standard scheduler in Linux 2.6 PI Mutex) was implemented for use in the merging the first part of the RT-Preempt introduced in December 2003. It ensures that Linux kernel. patches and agreed to gradually enter the other the time needed to schedule is fixed and deter - components into mainline. As of kernel version ministic irrespective of the number of active In addition to kernel code, drivers and specif - 2.6.21, approximately 60% of the RT-Preempt tasks - an important prerequisite towards real- ically their interrupt service routine may also patches (complete kernel preemption, PI time performance. Another important step cause latencies. In consequence, to further re - mutexes and the high-resolution timers) are which was also added to the development ker - duce the overall latency, a mechanism must be already available in mainline Linux. It is ex - nel 2.5 and became readily available in 2.6 was found to enable during execution of pected that most of the rest will have followed to enable scheduling during execution of cer - the interrupt service routines. This may not be until the end of 2007. tain parts of the kernel. This feature is called meaningful during its initial part when hard - kernel pre-emption; it prevents a low-priority ware is accessed more frequently, e.g. to ac - process from blocking a high-priority process knowledge the interrupt and to read and write Linux goes Virtualisation while the former is executing a kernel call. Ini - the various data and status registers of a par - Virtualisation of an entire computer requires an tially, only small portions of the kernel code ticular controller. But it may be very effective additional level of privilege (the “” were made interruptible but they have been during the remaining stages of the interrupt privilege) to ensure that the virtual machine extended continuously. service routine where hardware accesses occur monitor can control both the system and user less frequently. This feature is called interrupt level of a virtual machine and also that the vir - Due to the very nature of the kernel tasks, there threading. tual machine cannot interfere with the activities will always be a certain part of the kernel code of the host system or another virtual machine. that must be executed exclusively, and thus re - Precise timing in real-time systems requires Up to about one year ago, this additional priv - quires locking against other processes. Such timers with a resolution in the range of mi - ilege level was realised mainly in software such mut ual ex clusion is done using so-named mu - croseconds. The Linux system clock, however, as products provided by VMware Inc. (Palo texes; but a general problem is, again, that a could only be set to an interval of 1, 4 or 10 mil - Alto, USA). However, most of the newer x86 liseconds corresponding to 1000, 250 and 100 processors from and AMD provide the re - Hz, respectively. Several coding attempts to in - quired hypervisor privilege level in hardware crease the system clock failed, since clock fre - which greatly simplifies the design of the virtual quencies above 1000 Hz did not scale. It was, machine monitor. The related CPU flag is therefore, decided, to separate the timers need - called vmx (Virtual Machine eXtension) in Intel ed for high-resolution timing purposes from and svm (Secure Virtual Machine) in AMD the system clock timers and to introduce a new processors, respectively. feature called high-resolution timers (“hrtimer” subsystem). This subsystem now also con - In October 2006, Avi Kivity posted a Linux char - tains the feature “dynamic tick”; it allows to acter that allows to manage a vir - completely disable the periodic tick when the tual system by means of I/O control commands. system is idle and is used to safe battery power In addition, he provided a modified version of Figure 3. Example of latency data presenta - in mobile systems. U ntil kernel release 2.6.18, the qemu emulator where the emulator was re - tion in a histogram. The vertical green bar only the O(1) scheduler and a certain level of ker - placed by the appropriate kvm commands marks the worst-case latency. nel pre-emption were part of the mainline kernel. (http://sourceforge.net/projects/kvm/). The orig -

3 June 2007 EMBEDDED LINUX

Figure 4. The “dual-kernel approach” to realize a real-time Linux system Figure 5. The “single-kernel approach” to realize a real-time Linux system inal qemu emulator was developed by Fabrice tem and then from the host system through the competitive companies to share their efforts Bellard (http://fabrice.bellard.free.fr/qemu/). physical PCI bus to the peripheral device. The and develop together base components of the Probably because of its elegant design, excellent hard disk transfer rate of the system shown in operating system. This common effort, how - code quality and minimum interference with the figure 1, for example, amounts to about 68 ever, could be more effective if an organisation kernel, Linus Torvalds decided only two months MByte/s in the physical Fedora system and to were available to synchronise these activities. after its initial submission to merge kvm into about 36 MByte/s in the virtual Fedora systems. The Open Source Automation Development mainline Linux, and it was first released in kernel Lab (OSADL) was founded for this purpose version 2.6.20 on February 4, 2007. In comparison to other virtualisation solutions, (http://www.osadl.org). OSADL member com - kvm has the advantage that it is part of main - panies are active in the field of industrial au - An example of a Linux system that uses the line Linux, and will therefore be continuously tomation such as machine, machine tool and kvm driver and the qemu emulator to simulta - developed and adapted to other kernel features. equipment manufacturers, computer hard - neously run a total of seven virtual systems is Of special interest is a real-time system running ware and software manufacturers and software given in Figure 1. None of the virtual systems a kvm virtual system with preserved real-time service providers. The membership fee is needed any modification prior to be installed capabilities of the host system. This would allow primarily used to delegate the development of under kvm. The performance of the virtual sys - to use a single computer hardware for machine machine and automation software compo - tems is not different from that of the host sys - control and graphical user interface - even if the nents requested by a majority of the OSADL tem as long as straight CPU code is executed, two functions require two different operating members. Current projects, among others, are: e.g. during execution of Dhrystone or Whet - systems. As a consequence, system reliability Linux RT-Preempt patches, industrial I/O stone benchmarks. I/O operations, however, would increase while costs would decrease. framework, migration tools, upstream sub - have a reduced performance, since the data mission of Linux kernel components, making must be transfered twice - first from the virtu - Th e advent of open source software makes it kvm real-time compliant, real-time Ethernet al system via the virtual PCI bus to the host sys - possible for the first time for different and even drivers for Linux. I

Open Source Automation Development Lab

OSADL eG Homagstr. 3-5 D-72296 Schopfloch Germany Phone: +4 9(7 443 )13- 3073 Fax.: +4 9(7 44313-8- 3073 E-Mail: [email protected]

June 2007 4