A Programming Model for Time-Synchronized Distributed Real-Time Systems Yang Zhao Jie Liu Edward A. Lee EECS Department Microsoft Research EECS Department UC Berkeley One Microsoft Way UC Berkeley Berkeley, CA 94720 USA Redmond, WA 98052, USA Berkeley, CA 94720 USA ellen [email protected] [email protected] [email protected] Abstract In addition to interacting over a communication network, the nodes in a distributed embedded system interact through Discrete-event (DE) models are formal system specifica- the physical world. Driving an actuator at one node, for ex- tions that have analyzable deterministic behaviors. Using a ample, may affect the data sensed at another node. More- global, consistent notion of time, DE components commu- over, actuation may need to be orchestrated across nodes. nicate via time-stamped events. DE models have primarily The required precision of that orchestration, of course, de- been used in performance modeling and simulation, where pends on the application. Robotic applications, e.g. in man- time stamps are a modeling property bearing no relation- ufacturing, may require precisions on the order of millisec- ship to real time during execution of the model. In this onds. Instrumentation, where stimuli are generated and re- paper, we extend DE models with the capability of relat- sponses are measured, may require precisions on the order ing certain events to physical time. We propose a program- of nanoseconds or even higher. The question we address in ming model, called PTIDES (Programming Temporally In- this paper is how to construct the distributed software for tegrated Distributed Embedded Systems), which has DE se- such systems. mantics, but with carefully chosen relations between model General-purpose distributed software is dominated by time and real time. Key to making this model effective is to distributed object-oriented programming [32] using frame- ensure that constraints that guarantee determinacy in the works such as CORBA, SOAP, DCOM, EJB, and XML semantics are preserved at runtime. To accomplish this, Web Services. Some extensions of these frameworks, such we give a distributed execution strategy that obeys DE se- as ACE/TAO [31], support real-time scheduling concepts, mantics without the penalty of totally ordered executions and have caught on in certain communities (such as avion- based on time stamps. Our technique relies on having a dis- ics) [30]. These technologies are viewed as being too tributed common notion of time, known to some precision. heavyweight for many embedded applications such as in- Based on causality analysis of DE models, we define rele- dustrial control, where software may be written in spe- vant dependency and relevant orders to enable out-of-order cial purpose languages (e.g. based on the International execution without compromising determinism and without Electrotechnical Commission’s IEC 61131) and executed requiring backtracking. on special purpose hardware called Programmable Logic Controllers (PLCs). Extensions of these techniques to dis- tributed control systems (e.g. IEC 61499), have not proved 1 Introduction satisfactory, because of the non-determinism of implemen- tation. That is, the same standard-compliant application Distributed embedded systems are computer-based sys- running in two different implementations of the runtime en- tems where multiple computers are connected on a network. vironment may result in different behaviors [7]. Typically, each computer is connected to sensors, actuators Our approach to the nondeterminism challenge in con- or human-computer interfaces. Applications include manu- structing distributed real-time system is to rely on network facturing, instrumentation, surveillance, multi-vehicle con- time synchronization [20], where the computing nodes on trol, avionics systems, automotive systems and scientific the network share a common notion of time to a known pre- experiments. Since each computer interacts with physical cision. This has the potential for being lightweight and de- processes, the passage of time becomes a central feature; it livering repeatable and predictable behaviors at a variety of is this key constraint that distinguishes these systems from timing precisions. distributed computing in general. Network time synchronization is available on a variety 1 of platforms, including standard computers on the Inter- immediately or it has to wait for other potentially proceed- net (e.g. NTP [29]), time-triggered buses such as TTA or ing events. Once it is sure that the current event can be pro- FlexRay [21], TCP/IP over Ethernet (e.g. IEEE 1588), and cessed according to DE semantics, it hands the event over to wireless networks (e.g. RBS [11]). Implementations of local resource scheduler, which may use existing real-time IEEE 1588 have demonstrated time synchronization as pre- scheduling algorithms, such as earliest deadline first (EDF) cise as tens of nanoseconds over networks that stretch over to prioritize the processing of all pending events. This pa- hundreds of meters, more than adequate for many manufac- per only focuses on the global coordination layer, which is turing and instrumentation systems. Such precise time syn- key to achieving DE semantics in distributed systems. We chronization enables coordinated actions over distances that leverage and improve on distributed DE techniques to relax are large enough that fundamental limits (the speed of light, constraints on execution. In particular, we define a partial for example) make it impossible to achieve the same coor- order called the relevant order that can be statically checked dination by conventional stimulus-response or client/server to enable the global coordination layer to release received mechanisms. events out of their time stamp order while preserving DE Our approach in this paper builds on discrete-event (DE) semantics and without requiring backtracking. This out-of- modeling techniques [6, 23, 34]. DE models are concurrent order execution also loosens some constraints for the local compositions of components that interact via events. An resource schedulers. event is a time-stamped value, where time is “logical time” This paper is organized as follows. Section 2 discusses or “modeling time” [22]. Correct execution of such models related work. Section 3 motivates our programming model requires only that the ordering of time stamps be respected. using a networked camera application. Section 4 devel- DE is usually a simulation technology (e.g. in hardware de- ops the relevant dependency concept using causality inter- scription languages such as Verilog and VHDL and network faces [25], and defines the relevant order on events based on modeling languages such as OPNET Modeler1 and Ns-22). relevant dependency to formally capture the ordering con- When DE models are executed on distributed platforms, the straints of temporally ordered events that have a dependency objective is usually to accelerate simulation, not to imple- relationship. A distributed execution strategy based on the ment distributed real-time systems [6, 12, 34]. relevant order of events is presented in section 5, and its We call our programming model PTIDES (pronounced implementaion is in section 6. Future work is discussed in “tides”), for Programming Temporally Integrated Dis- section 7. tributed Embedded Systems. In our approach, DE is not a simulation technology, but rather application specification 2 Related Work language, which serves as a semantic basis for obtaining de- terminism in distributed real-time systems. Applications are Synchronous languages [4] have been effectively applied given as distributed DE models, where for certain events, to safety-critical embedded systems [5]. These languages their modeling time is mapped to physical time. For exam- (which include Esterel, SCADE, Lustre, Signal, etc.) pro- ple, a programmer may specify that an actuator must pro- vide deterministic concurrent semantics, but at the expense duce a physical output at the time determined by the time of tight coordination that makes distributed implementation stamp of an event sent to the actuator. When these models difficult. are executed in a runtime environment that ensures DE se- Another important innovation is the development of mantics, we know that the applications will have determin- time-triggered languages and the concept of logical execu- istic behaviors regardless of the actual implementations. tion time [16]. One example is Simulink with Real-Time Preserving DE semantics at runtime can be challenging, Workshop (RTW), from The MathWorks, which is widely since the global, consistent notion of time may lead to a to- used for designing embedded control systems in applica- tal ordering of execution in a distributed system, an unnec- tions such as automotive electronics. Simulink with RTW essary waste of resources. PTIDES takes an event-driven leverages an underlying preemptive priority-driven multi- execution strategy. Unlike many hard real-time distributed tasking operating system to deliver real-time behavior based systems that depend on domain specific network architec- on rate-monotonic scheduling [26]. A related approach is tures, our only assumption of communication behavior is Giotto [16], which introduces additional latency but deliv- that it delivers packets reliably with a known bounded de- ers better schedulability analysis [28]. While Giotto and lay. We divide our execution strategies into two layers: Simulink/RTW are intended primarily for periodic real-time global