EMBEDDED LINUX Native mainline Linux: fit for embedded and real-time systems By Carsten Emde, Open Source Automation Development Lab, and Thomas Gleixner, Linutronix When Linux was born on August 1, 1991, nobody expected that 15 years later the scale of supported hardware would range from tiny micro - controllers up to large multi - processor mainframes, or that Linux would become a real-time operating system. Figure 1. Linux Fedora 7 system running a total of seven virtual systems simultaneously I What is real-time and how is it measured? As start of the interrupt service routine is called tion by respondents planning to use Linux was generally agreed, it is not an average fast re - “interrupt latency”, and the interval between the the item “real-time limitations”. At this time, it sponse that makes a system real-time compliant input trigger and the execution of the user was a common understanding that a real-time but the reliability that a defined response in - space program is called “overall latency”, “pre- operating system must be conceived from the terval is not exceeded under whatever circum - emption latency”, or simply “latency” (figure 2). very beginning as such and that every single stances. A system, for example, that reliably Normally, a large number of single measure - component must be designed with all aspects of responds to external signals within 30 minutes ments is taken over several hours while gener - real-time execution in mind. Thus, retrofitting is a real-time system whereas another system, ating a realistic system load, and the longest real-time capabilities into the native mainline that relies on a timely response within 30 mi - measured overall latency is taken as the final re - Linux was deemed impossible. As a conse - croseconds but fails once in a year, is not. sult. This is then called “worst-case latency”. To quence, several Linux real-time projects were gain a better overview, the data points are best started that used the dual-kernel approach. This In order to determine the real-time capabilities displayed graphically in a latency histogram approach replaces the Linux kernel by a small of a system, an external signal is applied re - (figure 3). real-time kernel to which all interrupts are sent peatedly to a controller input such as a digital and which controls the entire system. One of its input line or a handshake line of a serial or par - Initially, the Linux operating system was adopt - tasks is a modified Linux kernel; this enables ex - allel communication controller. The analysis re - ed primarily as a server platform - probably due ecution of Linux user space programs and thus quires that a few lines of code are inserted into to its support of generally available and inex - provides binary Linux compatibility of the en - the beginning of the respective interrupt serv - pensive hardware in combination with its su - tire system (figure 4). ice routine and into the initial part of a user perior network capabilities and run-time space program waiting for this interrupt. These stability. At a later stage, support for architec - Examples of solutions using this approach are lines of codes are written in such a way that a tures other than x86 was added so the use of RTLinux, RTAI, and Xenomai; they have been pulse is generated at an output device. This Linux could have been extended immediately to used repeatedly and successfully in recent years pulse can then be measured along with the embedded systems. However, embedded sys - in a large number of industrial applications. input trigger using an oscilloscope. In addition, tems often require real-time capabilities which RTLinux, RTAI, and Xenomai offer, similarly to the respective interrupt line is also monitored. Linux as a server operating system was never in - the RT-Preempt patches, a POSIX compatible tended to provide. In fact, in a 2001 survey pub - API to access real-time related functionality. The interval between the input trigger and the lished in April 2002 as a Venture Development Xenomai, in addition, comes with a collection of corresponding level change of the interrupt line Corporation Whitepaper (http://linuxde - other APIs (called “skins”). They greatly facili - is called “gate latency” or “controller latency”. vices.com/articles/AT6328992055.html), the tate the migration from proprietary real-time The interval between the input trigger and the most important factor inhibiting Linux adop - operating systems. In the foreseeable future, it is June 2007 2 EMBEDDED LINUX The other features, such as PI mutexes, high-res - olution timers and interrupt threading, were only available when the kernel was patched with the “RT-Preempt” patch downloaded from Ingo Molnar's repository at RedHat http://people.red - hat.com/mingo/realtime-preempt/. The various elements of this patch described above were de - veloped over the last couple of years by Ingo Mol - nar and Thomas Gleixner with support from many kernel developers worldwide. There was al - ways hope that the RT-Preempt patch would be integrated into mainline Linux one day. But con - vincing Linus Torvalds that making Linux a real- time operating system would do no harm, but greatly enhance its usability, was not an easy task. Figure 2. The various elements of the overall or pre-emption latency Fortunately, there were an increasing number of arguments to make Linux real-time compliant - well conceivable that Xenomai will adopt the low-priority process executing a kernel function not only for industrial automation projects but also RT-Preempt approach explained below and under locking conditions may block a high-pri - for standard server and desktop Linux systems: provide additional functionality on top of the ority process that is about to execute the same Audio and video mixing and recording real-time capable mainline Linux kernel. In par - kernel function. This phenomenon is called pri - Reliable timestamps in financial and allel, the mainline Linux evolved as well. A first ority inversion, and one of the solutions is to commercial transactions important step was to replace the O(n) sched - bequeath the priority level to the mutex and let Voice over IP and streaming video services. uler by Ingo Molnar's O(1) scheduler. This new it execute under the same priority as the calling scheduler was already merged into the devel - process. This special mutex variant that is At the occasion of the Ottawa kernel summit in opment Linux 2.5 kernel tree in 2002 and be - called Priority Inheritance Mutex (abbreviated July 2006, Linus Torvalds finally accepted came the standard scheduler in Linux 2.6 PI Mutex) was implemented for use in the merging the first part of the RT-Preempt introduced in December 2003. It ensures that Linux kernel. patches and agreed to gradually enter the other the time needed to schedule is fixed and deter - components into mainline. As of kernel version ministic irrespective of the number of active In addition to kernel code, drivers and specif - 2.6.21, approximately 60% of the RT-Preempt tasks - an important prerequisite towards real- ically their interrupt service routine may also patches (complete kernel preemption, PI time performance. Another important step cause latencies. In consequence, to further re - mutexes and the high-resolution timers) are which was also added to the development ker - duce the overall latency, a mechanism must be already available in mainline Linux. It is ex - nel 2.5 and became readily available in 2.6 was found to enable scheduling during execution of pected that most of the rest will have followed to enable scheduling during execution of cer - the interrupt service routines. This may not be until the end of 2007. tain parts of the kernel. This feature is called meaningful during its initial part when hard - kernel pre-emption; it prevents a low-priority ware is accessed more frequently, e.g. to ac - process from blocking a high-priority process knowledge the interrupt and to read and write Linux goes Virtualisation while the former is executing a kernel call. Ini - the various data and status registers of a par - Virtualisation of an entire computer requires an tially, only small portions of the kernel code ticular controller. But it may be very effective additional level of privilege (the “hypervisor” were made interruptible but they have been during the remaining stages of the interrupt privilege) to ensure that the virtual machine extended continuously. service routine where hardware accesses occur monitor can control both the system and user less frequently. This feature is called interrupt level of a virtual machine and also that the vir - Due to the very nature of the kernel tasks, there threading. tual machine cannot interfere with the activities will always be a certain part of the kernel code of the host system or another virtual machine. that must be executed exclusively, and thus re - Precise timing in real-time systems requires Up to about one year ago, this additional priv - quires locking against other processes. Such timers with a resolution in the range of mi - ilege level was realised mainly in software such mut ual ex clusion is done using so-named mu - croseconds. The Linux system clock, however, as products provided by VMware Inc. (Palo texes; but a general problem is, again, that a could only be set to an interval of 1, 4 or 10 mil - Alto, USA). However, most of the newer x86 liseconds corresponding to 1000, 250 and 100 processors from Intel and AMD provide the re - Hz, respectively. Several coding attempts to in - quired hypervisor privilege level in hardware crease the system clock failed, since clock fre - which greatly simplifies the design of the virtual quencies above 1000 Hz did not scale.