The following paper was originally published in the Proceedings of the USENIX Windows NT Workshop Seattle, Washington, August 1997 The RTX Real-Time Subsystem for Windows NT Bill Carpenter, Mark Roman, Nick Vasilatos and Myron Zimmerman VenturCom, Inc. Cambridge, MA For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: [email protected] 4. WWW URL: http://www.usenix.org The RTX Real-Time Subsystem for Windows NT Bill Carpenter, Mark Roman, Nick Vasilatos and Myron Zimmerman VenturCom, Inc. 215 First St. Cambridge, MA 02142 {carp, marcus, boxer, myron}@vci.com Abstract massive source code whacking – and the benefits of modularity vis a vis the separation of functionality in This paper describes a subsystem for the Windows NT systems engineering. 4.0 Operating System which implements a kernel-mode execution environment for Win32 compatible tasks and 2. Real-time Extensions threads that have hard real-time performance character- istics (deterministic interrupt response and dispatch la- Requirements for real-time extensions were developed tencies). This subsystem is a proper OS extension which in a process of consultation with company partners requires no modifications to the standard OS kernel and whose applications are concentrated in industrial auto- limited modifications to the NT Hardware Abstraction mation and telephony domains. These however span Layer (HAL). This gives the motivation for the ap- nearly the full range of functionality traditionally asso- proach, describes the design and evaluates the success ciated with hard real-time systems. Windows NT 4.0 of the implementation in the context of other strategies has significant real-time features but gating limitations for extending general purpose OS kernels. as well [Sommer 96, VENTU 96, Timmerman & Mon- fret 96, MICRO 95]. The problems most often cited are 1. Introduction priority inversion, the paucity of real-time thread priori- ties and unbounded system response times. Real-time features have been moving from special pur- pose operating systems into general purpose operating It was further clear from the consultation process that systems for at least 10 years. VenturCom has produced real-time extensions to the system should conform as a succession of real-time UNIX versions going back closely as possible to the standard interfaces of the host as far as 1984. Other implementations are described in OS. It was therefore stipulated that operations in com- [Furht et al. 91 and IEEE 93]. mon between real-time objects and normal objects should be accessible via a common interface. These were responses to the extremely reasonable de- sire to leverage the features of the underlying general This lead to the definition of a subset of the Win32 API purpose operating system in real-time applications de- which includes all basic execution control, memory velopment. This included both accessing the rich feature management, communications, synchronization, I/O and set of the general purpose OS and accessing available configuration operations to be supported by the added off the shelf applications, utilities and services in devel- real-time threads and to the definition of added inter- oping the particular real-time application at hand. faces for real-time operations (stack/heap pre-allocation and locking, interrupt and i/o port attachment, clocks This is even more so with Windows NT based applica- and timers) to be provided for both real-time and nor- tions. The system services are rich, diverse and wildly mal objects. The result is a unified – Real-time Appli- popular with the programming cognoscenti. The avail- cation Programming Interface (RTAPI) with which ap- able base of applications is vast and growing rapidly. plication elements intended for both the normal Win32 environment and the extension real-time environment This is about how real-time computing can be facilitated can be designed – and with which design and code can for a particular general purpose operating system – be shared. Windows NT 4.0 with the addition of a subsystem that supports the execution of very low latency real-time The implementation of these extensions spans two com- threads with Win32 compatible system services. This ponents. The first implements the real-time operation approach reflects lessons learned about the costs of extensions for normal Win32 objects. The second im- device for which an RTSS process registered an inter- plements the Real-Time Subsystem (RTSS). Both ex- rupt handler. ploit limited modifications developed for the NT HAL (which primarily provide extensions for clock and timer The net effect of these interrupts is that some number of device programming and interrupt management). RTSS threads will become ready to run. When all of the threads have finished their immediate work, the Figure 1 illustrates how the Real-Time Subsystem inter- thread manager becomes idle, the RTSS switches back faces to NT. The HAL and the RTSS driver provide the to the HAL stack and normal NT processing occurs. Real-Time Application Interface (RTAPI) to real-time processes, which are linked and loaded as NT drivers. 2.2 Interrupt Handling The difference between an NT driver and the real-time Interrupt objects are used to claim system interrupt re- process is that the real-time process uses RTAPI calls sources and handle interrupt events. The interrupt ob- instead of the NT device driver interface. ject is an abstract type which captures the resource us- age and the service routine associated with an interrupt. 2.1 Thread Scheduling The real-time thread manager offers 128 thread priori- All interrupt service routines in RTSS are realized as ties and controls priority inversion. The threads are real time threads. In designing the system in this way, a scheduled by priority and within a priority, in round small performance tradeoff was made. For a given in- robin order. There is no sharing of processors based on terrupt, the handler latency increases to include thread a fixed time slice. All RTSS threads must give up the switching time. This is a relatively small increase in processor by waiting, changing thread priority or other- latency, however. The benefit to this approach is that wise completing execution. All RTSS threads run be- all real time system activities are captured as real time fore any NT threads can run. threads, which provides a simple, coherent approach to setting priority across the real time subsystem, and al- lows for a more robust and verifiable implementation of the subsystem. Win32 Process using RTAPI Interrupt masking is achieved through lazy or optimistic User Mode interrupt masking. This technique manages the inter- Kernel Mode rupt level in software to reduce the overhead of hard- System Services ware programming. Actual hardware masking occurs only in the event of an actual hardware interrupt, at which point the mask is set to prevent further interrupts, Various NT Managers IO Manager and the hardware event is noted so that it can be dealt Real-Time Process strictly using RTAPI (Driver) with at the end of the critical code section. This tech- Windows NT Kernel nique is described in [Stodolsky et al. 93]. Real-Time Subsystem and RTAPI (Driver) 2.3 Communication with NT NT Executive In order for NT and the RTSS to communicate, two message queues are maintained by the RTSS. One Standard HAL Real-Time Extensions to HAL queue is for messages passing from NT to the RTSS and the other queue is for messages passing from the RTSS to NT. The queues are implemented as circular buffers of messages. This organization is necessary because at Figure 1 Real-Time Subsystem this lowest level, NT and the RTSS are not synchro- nized except for the ability to atomically write the index The thread manager gets control of the processor in of the last message written to the queue. response to interrupt processing by the HAL. Two of these interrupts, the clock and the RTSS software inter- RTSS processes request NT services through this chan- rupt are fixed interrupt sources. The clock interrupt nel. Although the NT device driver interface is in the handler handles timer expirations and the RTSS soft- same address space of the RTSS process, the process ware interrupt causes the RTSS to examine the queue of must not call these interfaces. This restriction is neces- messages from NT. The last type of interrupt is from a sary because the RTSS thread must wait at the message Remote access to an object depends upon which of the queue instead of being suspended by the NT executive. following categories an object belongs. 3. RTSS Objects 3.1. Subsystem Specific Objects Beside thread performance, the most important feature of the RTSS is the duplication of NT objects and the NT In terms of remote access, a subsystem specific object is object technology. The NT object technology is de- not remotely accessible by processes in the other sub- scribed in [Custer 1993]. Adhering to the NT object system. The prime example of this type of object is the technology gives Win32 fluent programmers immediate thread and objects derived from the subsystem threads, familiarity with the RTSS objects and interface. the RTSS timer and interrupt objects. This not a serious restriction since other process’ threads are not accessed For instance, the RTSS implements a real-time sema- except when a debugger controls a process. In the case phore. The RTSS semaphore calls are listed in the table of the two subsystems under consideration, debugging below with along with their Win32 equivalents. techniques are drastically different. Denying remote access to the subsystem specific objects Table 1 Semaphore Call Comparison does not limit the usefulness of interfaces in either envi- ronment. This is the case with the interrupt handler RTAPI Semaphore Win32 Semaphore object.