The RTLinux Manifesto Victor Yodaiken Department of Computer Science New Mexico Institute of Technology Socorro NM 87801 [email protected] http://www.rtlinux.org ABSTRACT moment an interrupt handler starts to execute is under 15 microseconds on RTLinux running on RTLinux is the hard realtime variant of Linux a generic x86. A RTLinux periodic task runs that makes it possible to control robots, data within 35 microseconds of its scheduled time on acquisition systems, manufacturing plants, and the same hardware. These times are hardware other time-sensitive instruments and machines. limited, and as hardware improves RTLinux will also improve. Standard Linux takes up to 600 microseconds to start a handler and can easily be more than 20 milliseconds (20,000 microsec- 1 Introduction onds) late for a periodic task2. As an unfair but fun comparison, an optimistic study of MS- Real-Time Linux (RTLinux) is a version of Windows/NT didn't even bother to try to mea- Linux that provides hard real time capability. A sure times under a millisecond and still found NASA computer running RTLinux ¯ew into the that NT numbers were essentially the same as eye of Hurricane Georges to collect data[14]; the the standard Linux numbers, while Windows/98 Jim Henson Creature Shop in Hollywood is de- was up to 140 milliseconds too late on a periodic veloping a RTLinux application to control ªan- task [7]. To be fair, there are now Window-NT imatronicº things used in movies; RTLinux has versions of the RTLinux method and these seem been used for video editors, PBXs, robot con- to get low level timings that are sometimes al- trollers, machine tools, and even to stop and start most as good and generally not more than two hearts in medical experiments1 times worse than RTLinux[6]. RTLinux provides the capability of running What makes RTLinux useful is that it extends special realtime tasks and interrupt handlers on the standard UNIX programming environment the same machine as standard Linux. These to realtime problems. RTLinux realtime inter- tasks and handlers execute when they need to rupt handlers and tasks can be connected to or- execute no matter what Linux is doing. The dinary Linux processes ± either via a device in- worst case time between the moment a hardware terface where Linux processes read/write data, interrupt is detected by the processor and the or via shared memory. A standard Linux pro- 1RTLinux is released, as is, with no warranty of any kind. Use at your own risk. 2A process using sched setsched cess, perhaps executing a shell script or a Perl Video displays should almost always keep program, can collect data from a realtime han- up with the frame rate. A half a sec- dler or task, process and log it and display the ond freeze will be noticed and a couple of results on X-Windows. Using Perl scripts to to freezes in a minute will make the system control a realtime device from an ordinary PC unpleasant. If the program starts dropping may seem ridiculous, but it works surprisingly frames, it can sometimes use an algorithm well. to hide the missing frames by interpolation. The rest of this paper is in ®ve parts. Sec- tion 2 is an introduction to realtime comput- The rocket shutdown sequence must meet ing. Section 3 is an explanation of why real- deadlines or the rocket may explode. A sin- time computing is so hard to integrate with non- gle freeze of a half a second at the wrong realtime computing and why obvious methods ± moment might have a spectacular and ter- like making the standard kernel directly support minal result. realtime ± are doomed to failure. Section 4 ex- So the text editor needs to be fast and respon- plains how RTLinux works and section 5 shows sive, the video display needs to usually meet how to write applications. The last section cov- timing deadlines and the rocket control needs ers future directions and has some random ac- to be able to guarantee response times. In the knowledgements. CS literature, the video display would be called a soft realtime system, and the rocket control would be called a hard realtime system [13, 12]. 2 An introduction to realtime Most hard realtime applications have failure programming. modes that are not as spectacular as those of liq- uid fuel rockets, but they need guaranteed timing ªRealtimeº is is an over-used term that can be nonetheless. If you are controlling a servo mo- used to mean ªright awayº or ªfastº as in ªre- tor through the parallel port of a cheap PC[8], altime stock quotesº. RTLinux is addressed at each timing jitter of 10 microseconds causes an hard realtime systems: those with timing dead- error of one degree. If you are collecting data lines that cannot be missed. The traditional from a scienti®c instrument or video frame grab- uses for these systems are to control or mon- ber, a missed deadline may result in missed data itor some physical system or device such as a or even a confused device. For high speed net- motor, an assembly line, a telescope, or an in- works, a delay of a couple of microseconds may strument. Telecommunications and networking drop a packet and cause a major performance devices often also need realtime control. Con- loss as the system times out and requests a re- sider the difference between the response time transmit. The distinctive property of hard real needed for text editor, a video display, and a time systems is this requirement for guaranteed program controlling the shutdown sequence of timing ± an average time response of 5 millisec- a liquid fuel rocket. onds on our rocket controller will not make up for a single worst case of 100 milliseconds. The text editor should respond quickly to user commands, but if it takes a half a sec- Hard realtime systems cannot use av- ond to update a display every now and then, erage case performance to compen- few users will notice. sate for worst case performance. Suppose that we have a board that samples was a much larger amount of memory available analog lines and produces an 8 bit result every Ð with close to zero cost. On the other hand, 100 microseconds. Most boards like this are de- a realtime task that misses its ®rst 10 pages will signed with hardware buffers right now so that take 5000 time units to do what could otherwise they can compensate for the non-realtime behav- be done in 10 time units and this worst case per- ior of Microsoft Windows and NT. The analog to formance is what matters. digital device generates samples and puts them Because of these problems, back in the far in the buffer. If the buffer is deep enough, say past3 realtime systems were hand made, simple with room for 512 samples, then we won't lose contraptions designed by tough, rugged, engi- any data if we read the buffer at least once ev- neering types who scorned compilers and vir- ery 50 milliseconds. There are three problems tual memory and thought that alphanumeric dis- with this scenario. The ®rst problem is that even plays were unnecessary luxuries. In these sys- though 50 milliseconds is a long time, it is easy tems, optimizing worst case performance was to miss the deadline in even a moderately loaded quite straightforward. In fact, many realtime system. Ordinary Linux can't keep up, even us- systems still are designed as loops that execute ing the so-called ªrealtimeº POSIX extensions. on a bare machine. The program loops through A second problem is that if the hardware has a list of simple tasks and the longest time before to cope with software timing uncertainties, the a task will run is the sum of the execution times hardware becomes more complex and costly. Fi- of all the tasks on the list. nally, if we need to control a device ± to react to counter=500; data ªin real timeº ± then 50 milliseconds may while(1){ be too long and the hardware buffers may cause if(data_on_sensor()){ instability as they introduce delays in the control read_sensor(); loop. compute_output(); RTLinux provides support for hard realtime counter--; programs and will, in the future, offer sup- } port for some kinds of quality of soft realtime. if(!counter){ RTLinux leaves the problem of ªfastº applica- output(); tions to standard Linux. Making non-realtime counter=500; applications run fast is incredibly complex. It } turns out that much of what you can do to } speed up the non-realtime performance of appli- The problem with this design is that is does cations introduces unpredictable delays that are not scale. As realtime applications get more unhealthy for realtime. The obvious example is complex, they need more complex support. If paging of virtual memory. If 99.99% of program you want to have a realtime control program for memory references are in the page cache, and if an aircraft engine diagnostic system that moni- a hit takes one time unit and a miss takes 500 tors hundreds of sensors and also needs to dis- time units, then over a suf®ciently large time pe- play graphical data, interact with a database, riod the average access time is time units ± connect to a network and even run a web inter- virtual memory is practically free. On average, face, then writing a control loop system is out of virtual memory is a major performance win, be- cause processes will be able to run as if there 3Well before Dave Miller was born.