Comparing the Real-time Performance of Windows NT to an NT Real-time Extension Kevin M. Obenland, Tiffany Frazier, Jin S. Kim, John Kowalik The MITRE Corporation, 1820 Dolley Madison Blvd. McLean, VA 22102 Contact: [email protected] Abstract Despite its shortcomings there are several efforts under- Because of the dominance of Microsoft® Windows® in the way to use NT as-is for real-time systems[6][7][8]. These efforts are motivated by the low cost of using NT based PC market there is a strong interest in using Windows NT® systems as well as the stability of the Windows Application as a platform for real-time process and control systems. Programming Interface (API)[9]. In this paper we quantify This type of solution is very cost effective because factors that determine under what conditions it makes applications and development tools are widely available. sense to use NT for real-time processing. However, Windows NT was designed as a general purpose Another alternative, that still allows the use of NT operating system and optimizes average not worst case COTS, is to add a real-time extension to NT[10]. This performance. In this paper we investigate two methods for extension is essentially a separate real-time operating sys- bring real-time process and control systems to NT based tem. In this Dual-OS architecture all real-time processing is platforms. We first evaluate NT as-is, using a series of real- performed under the real-time OS, and NT handles the non time benchmarks, and show that NT use in real-time time critical processing. In this system all NT COTS still systems is limited to soft real-time systems where there is run out-of-the-box. Therefore development time and cost low system load. The second approach for developing NT are still kept to a minimum. In the last several years a num- based real-time systems is to add a real-time extension to ber of vendors have produced products that follow this NT. We evaluate one such product, INtime® from RadiSys approach. These include the products: INtime® from Radi- and conclude that, even under a heavy system load, hard Sys[11], RTXTM from VenturCom[12][13], and Hyperker- real-time determinism is possible. nelTM from Imagination Systems[14]. In this paper we 1 Introduction evaluate one of these products, INtime, and show that it is significantly more deterministic than NT alone. The Windows suite of operating systems, Windows There have been several previous studies which evalu- 95®/98® and Windows NT®, dominate the personal com- ated the real-time performance of NT and/or NT extension puter operating system market[1]. Because of this large products[6][7][8][15][16][10][17]. However, in this paper market there is a vast amount of affordable Commercial off we provide a direct and unbiased comparison between the the Shelf (COTS) software products available for these two approaches. Many of the previous performance evalua- operating systems. The popularity of Windows also makes tions were performed by the manufacturers themselves. it desirable to use it in other types of systems like real-time Another key factor, that we examine in detail, is the issue process and control systems. Windows NT is the most of system load. We find that the determinism of NT is sig- robust of all the Windows operating systems and it is there- nificantly worse when considering a load. However, the fore the most likely candidate for use in real-time systems. determinism of processes running under the INtime real- Using Windows NT allows these systems to utilize low time extension are relatively unaffected by a system load. cost COTS software and increases the user friendliness of the system because they look and feel more like the per- 2 Evaluating real-time OS behavior sonal computer environment. Because of these market forces, the use of Windows has also been mandated by var- The correctness of a calculation performed by a real- ious agencies within the Department of Defense[2][3][4]. time system is determined by its ability to satisfy time con- Unfortunately NT was designed as a general purpose straints as well as its ability to operate error free[18]. In this operating system not a real-time one[5]. It was designed to paper we consider only real-time systems that are based on optimize average performance and not worst case perfor- COTS operating systems which means that it is impossible mance as is required of any real-time operating system. to analyze all possible paths in an application. The use of COTS also means that our analysis must be stochastic, amount of time and then calculates the actual sleep time based on data gathered empirically, rather than mathemati- using the Pentium performance counter. The jitter is the cal. We note, however, for complex hard real-time systems difference between the actual and desired sleep durations. (such as those deployed by the Department of Defense) The response benchmark tests the ability of a system to stochastic analysis is the technique generally used to verify perform a fixed amount of processing in a deterministic critical timing requirements. amount of time. The test repeats simple operations—such as floating point additions, block memory copies, or disk 2.1 Real-time operating systems write operations—for a base case duration of approxi- mately 10 milliseconds. The Bintime benchmark tests to In order to construct a real-time computer system the what degree the clock granularity is available to a real-time underlining operating system must abide by several application. It records the latency between successive calls requirements[5]. These requirements are outlined below: to a time of day clock function[19]. • Support for multiple preemptable threads - Individual To support deterministic response to external events an threads are typically used to process different hardware OS must process interrupts in a timely fashion. The ISR devices. It is important that the threads are preemptable latency benchmark measures the responsiveness of the OS so that a scheduler can be used to control the allocation to the assertion of the interrupt and the ISR release latency of system resources. benchmark measures the delay seen when transferring con- • Thread priority - A priority scheme is essential to trol from the ISR to the interrupt thread that completes the ensure that time critical processing takes precedence processing of the interrupt. The final type of benchmark over less critical processing. tests inter-thread/process communication by measuring the time it takes one thread to signal another. • Synchronization between threads - Threads that share data require a means of synchronization so that data is TABLE 1. Real-time benchmarks read and written in the proper order. • Deterministic timing of the operating system - This usu- Benchmark Description ally requires that the operating system be preemptable. Jitter Measures the difference between the A non-preemptable operating system could impose observed and desired delay times of vari- unbounded delays. ous OS timer functions • Support to prevent priority inversion - Priority inver- Response Measures the amount of time required to sion occurs when a high priority task is waiting on a perform a fixed amount of processing resource that is held by a lower priority task. The high priority task could be blocked indefinitely if a task with Bintime Tests the responsiveness of an operating intermediate priority preempts the low priority task. system by measuring the time between The typical solution for priority inversion is to increase calls to a time of day clock function the priority of the low priority task until it relinquishes ISR Latency Measures the amount of time from the the resource. assertion of an interrupt to the beginning of its ISR Using these five requirements one can evaluate the abil- ISR Release Measures the time it takes to switch from ity of an operating system to support real-time processing. Latency an ISR to an interrupt thread For example, NT provides multiple preemptable threads but allows only 32 levels of priority. NT also does not pro- Communi- Tests various inter-thread/process com- vide a mechanism to prevent priority inversion for the cation munication mechanisms highest priority threads. The design choices used in NT and how this relates to real-time systems are discussed in more 2.3 Hardware setup detail in Section 3. Table 2 describes the hardware and software configura- 2.2 Testing a real-time operating system tion of the prototype systems used in our evaluation stud- ies. Everything used is commercially available as standard We used the six different types of benchmarks shown in COTS. All timings in our studies were gathered by the Table 1 to evaluate the real-time capabilities of NT and the benchmarks themselves, we did not use any special pur- Dual-OS system. The Jitter benchmark measures the deter- pose hardware to stimulate the system or gather statistics. minism of timers of different duration and resolution. The We did not install INtime when evaluating the NT only sys- benchmark uses a sleep function to sleep for the specified tem because INtime modifies the Hardware Abstraction Layer (HAL) of NT. The HAL is the portion of NT that iso- Table 4 shows the different WinBench98 benchmarks lates the low level hardware details from the kernel and the used in our testing. The CPU test consists of two different device drivers. tests, one targeting the CPU itself and one targeting the FPU. Both the Disk and Graphics benchmarks give results TABLE 2. Prototype hardware and software for a set of Business applications and a set of High-end setup used in benchmark tests applications.