Operating System Performance in Support of Real-time Middleware Douglas C. Schmidt, Mayur Deshpande, and Carlos O’Ryan Department of Electrical and Computer Engineering University of California, Irvine, 92612 g fschmidt,deshpanm,coryan @uci.edu th A subset of this paper appeared in the 7 IEEE Inter- such as Windows NT/2K. Since traditional real-time operating national Workshop on Object-oriented Real-time Dependable systems tend to be expensive and tedious to configure/program, Systems (WORDS’02), San Diego, California, January 7–9, the maturation of Linux/RT will be a welcome advance for 2002. DRE system developers. Keywords: Real-time CORBA Object Request Brokers, Real- Abstract time Operating System Middleware Support. Commercial-off-the-shelf (COTS) hardware and software is being evaluated and/or used in an increasing range of mission- 1 Introduction critical distributed real-time and embedded (DRE) systems. Due to substantial R&D investment over the past decade, Emerging trends. Two fundamental trends are having a pro- COTS middleware has recently matured to the point where found influence on the way in which new distributed real-time it is no longer the dominant factor in the overhead, non- and embedded (DRE) systems are being conceived and con- determinism, and priority inversion incurred by DRE systems. structed: As a result, the focus has shifted to the COTS operating sys- Commoditization of Information Technology (IT) —IT tems and networks, which are once again responsible for the of all forms is becoming highly commoditized i.e., majority of end-to-end latency and jitter. This paper compares and evaluates the suitability of popu- COTS hardware and software artifacts are getting faster, lar COTS operating systems for real-time COTS middleware, cheaper, and better at a relatively predictable rate. Dur- such as Real-time CORBA. We examine real-time operating ing the past decade, many application domains have ben- systems (VxWorks and QNX), general-purpose operating sys- efited from the commoditization of hardware, such as tems with real-time thread scheduling classes (Windows NT, CPUs and storage devices, and networking elements, Windows 2K, and Linux), and a hybrid real-time/general- such as IP routers. More recently, the maturation of pro- purpose operating system (Linux/RT). While holding the hard- gramming languages, such as Java and C++, operating ware and ORB constant, we vary these operating systems sys- environments, such as POSIX and Java Virtual Machines, tematically to measure platform-specific variations in context and enabling middleware, such as CORBA and Enter- switch overhead, throughput of the ORB in terms of two-way prise Java Beans, are helping to commoditize many soft- operations per-second and memory footprint of the ORB li- ware components and architectural layers, as well. braries. We also measure how the latency and jitter of high- Network-centric paradigm shift — There is a growing ac- priority operations are affected as the number of low-priority ceptance of network-centric systems, where applications operations increase. with a range of quality of service (QoS) needs are con- Our results indicate that real-time operating systems remain structed by integrating separate components connected the platforms of choice to deliver predictable, efficient, and by various forms of communication services. The nature scalable performance for DRE middleware and applications. of this interconnection can range from small and tightly However, the emergence of hybrid general-purpose/real-time coupled DRE systems, such as avionics mission comput- operating systems, such as Linux/RT, are a promising direc- ing, to the large and loosely coupled DRE systems, such tion for future DRE systems. Although Linux/RT is not yet as as global telecommunications. deterministic as traditional real-time operating systems, such The interplay of these two trends has led to new ar- as QNX and VxWorks, it does provide more predictable and chitectural concepts and services that are embodied in lay- scalable behavior compared to mainstream operating systems, ers of COTS middleware [1]. These middleware layers This work was supported in part by Siemens and SAIC. are interposed between applications and commonly available 1 COTS hardware and software infrastructure to make it fea- it unsuitable for applications with hard real-time deadlines. In sible, easier, and more cost effective to develop and evolve general, our results underscore the need for a measure-driven DRE systems. Middleware is the result of recognizing the methodology to pinpoint sources of overhead and priority in- need for more advanced and capable support–beyond simple version in ORB middleware and operating systems for DRE connectivity–needed to construct effective DRE systems. systems. Although the quality of COTS software has generally Paper organization. The remainder of this paper is orga- lagged behind hardware, recent improvements in frame- nized as follows: Section 2 describes the ORB/OS testbed works [2], patterns [3, 4], and development processes [5, 6] that we used to systematically benchmark the performance of have encapsulated the knowledge that enables COTS soft- TAO on popular real-time and general-purpose operating sys- ware to be developed, combined, and used in an increasing tems; Section 3 analyzes empirical results from benchmarks number of mission-critical DRE applications. Common ex- that measure the efficiency and predictability of TAO on the amples include e-commerce web sites, consumer electron- different OS platforms; and Section 4 presents concluding re- ics, avionics mission computing, hot rolling mills, backbone marks. routers, and high-speed network switches. Over the past sev- eral years, there has been substantial improvement in the QoS of COTS DRE middleware, based largely on the maturation 2 Overview of ORB/OS Testbed of implementations of industry standards, such as Real-time CORBA [7]. Evaluating the performance of complex DRE systems is hard The growing importance of operating system infrastruc- and becomes even harder if DRE system is built using a com- ture. First generation COTS middleware, such as initial im- bination of real-time middleware and operating systems [11]. plementations of CORBA, lacked appropriate optimizations This section describes the ORB/OS testbed that we used to and capabilities to support DRE systems with stringent QoS systematically benchmark the performance of TAO over dif- requirements [8]. Due to substantial R&D progress over the ferent operating systems, while keeping the hardware platform past decade [9, 10], COTS middleware, such as Real-time constant. This testbed helps to isolate the overhead due to the CORBA [7], has recently matured to the point where it is no OS. longer the dominant factor in the overhead, non-determinism, and priority inversion incurred by DRE systems [11]. As a re- 2.1 Hardware Overview sult, the focus has shifted to the COTS operating systems and networks, which are once again responsible for the majority of All of the tests in this section were run on a single-cpu Dell end-to-end latency and jitter. 866MHz Intel Pentium III system configured with 512 Mbytes This paper compares and evaluates the suitability of popu- of RAM and a 256Kb cache. We focus our experiments on a lar COTS operating systems for real-time COTS middleware, single CPU hardware configuration to: in particular Real-time CORBA. We examine three types of Factor out differences in network interface driver support operating systems: and Real-time operating systems, i.e., VxWorks and QNX Isolate the effects of OS design and implementation on ORB middleware and application performance. General-purpose operating systems with real-time thread scheduling classes, i.e., Windows NT, Windows 2K, and Linux, and 2.2 Operating Systems Overview A hybrid real-time/general-purpose operating system, i.e. Linux/RT. We ran the ORB/OS benchmarks described in this paper on Our findings extend earlier results in [12, 13] and illustrate that two well-established real-time operating systems (VxWorks general-purpose operating systems like Windows NT, Win- 5.4 and QNX-RTP 6.0), three general-purpose operating sys- dows 2K, and Linux are not yet suited to meet the demands tems with real-time scheduling classes (Windows NT 4.0 of applications with stringent QoS requirements. We also find Workstation with SP5, Windows 2K Professional, and Debian that real-time operating systems like QNX and VxWorks do GNU/Linux kernel version 2.2.14), and one hybrid general- enable predictable and efficient ORB performance, thereby purpose/real-time OS (Timesys Linux/RT version 2.2.14). On making them suitable as OS platforms for real-time CORBA all platforms, we used the TCP/IP stack supplied with the OS. applications. Each OS is described briefly below: This paper extends our previous research by including mea- VxWorks and QNX Both VxWorks and QNX are multi- surements for Linux/RT. We find that Linux/RT provides better threaded real-time OSes but QNX has a micro-kernel archi- control over QoS capabilities than Linux, but its jitter makes tecture that uses message-passing as a fundamental means for 2 inter-process communication (IPC). Both VxWorks and QNX DACE_LACKS_RTTI, check-new. The VxWorks gcc support preemptive priority-based first-in first-out (FIFO) compiler has bugs with native C++ exception handling and the scheduling of threads, in addition to semaphores that imple- 03 optimization level, so we could not use these options. ment a priority inheritance protocol [14]. QNX. We used gcc-2.95.2 for this platform. The Linux and Linux/RT Linux is a general-purpose, pre- options included: fno-exceptions, fcheck-new, emptive, multi-threaded implementation of SVR4 UNIX, BSD Wpointer-arith, O3, fno-implicit-templates, UNIX, and POSIX. It supports POSIX real-time process and DACE_NDEBUG, and D__ACE_INLINE__. thread scheduling. The Linux thread implementation inter- nally uses processes created by a variant of the fork() sys- Linux and Linux/RT We used gcc-2.95.2 tem function called clone().