Business Computing THE PROCESS GROUP APPROACH TO RELIABLE DISTRIBUTED COMPUTING Kenneth P. Birman One might expect the reliability of a distributed system to correspond directly to the reliability of its constituents, but this is not always the case. The mechanisms used to structure a distributed system and to implement cooperation between components play a vital role in determining the reliability of the system. Many trading information. 1 It is not un- all system can be greatly enhanced. contemporary distributed operating common for brokers to coordinate Figure 1 illustrates a possible in- systems have placed emphasis on com- trading activities across multiple terface to a trading system. The dis- munication performance, overlooking markets. play is centered around the current the need for tools to integrate com- Trading strategies rely on accurate position of the account being traded, ponents into a reliable whole. The pricing and market-volatility data, showing purchases and sales as they communication primitives supported dynamically changing databases giv- occur. A broker typically authorizes give generally reliable behavior, but ing the firm's holdings in various purchases or sales of shares in a exhibit problematic semantics when equities, news and analysis data, and stock, specifying limits on the price transient failures or system config- elaborate financial and economic and the number of shares. These in- uration changes occur. The resulting models based on relationships be- structions are communicated to the building blocks are, therefore, unsuit- tween financial instruments. Any dis- trading floor, where agents of the able for facilitating the construction of tributed system in support of this brokerage or bank trade as many systems where reliability is important. application must serve multiple com- shares as possible, remaining within This article reviews 10 years of munities: the firm as a whole, where this authorized window. The display research on ISIS, a system that pro- reliability and security are key con- illustrates several points: vides tools to support the construc- siderations; the brokers, who depend tion of reliable distributed software. on speed and the ability to customize • Information backplane. The broker The thesis underlying ISIS is that the trading environment; and the would construct such a display by in- development of reliable distributed system administrators, who seek uni- terconnecting elementary widgets software can be simplified using pro- formity, ease of monitoring and con- (e.g., graphical windows, computa- cess groups and group programming trol. A theme of this article is that all tional widgets) so that the output of tools. This article describes the ap- of these issues revolve around the one becomes the input to another. proach taken, surveys the system, technology used to "glue the system Seen in the large, this implies the and discusses experiences with real together." By endowing the corre- ability to publish messages and sub- applications. sponding software layer with pre- scribe to messages sent from program It will be helpful to illustrate dictable, fault-tolerant behavior, the to program on topics that make up group programming and ISIS in a flexibility and reliability of the over- the "corporate information back- setting where the system has found plane" of the brokerage. Such a IAlthough this class of systems certainly de- backplane would support a naming rapid acceptance: brokerage and mands high performance, there are no real- trading systems. These systems inte- time deadlines or hard time constraints, such as structure, communication interfaces, grate large numbers of demanding in the FAA's Advanced Automation System access restrictions, and some sort of [14]. This issue is discussed further in the sec- applications and require timely reac- tion "ISIS and Other Distributed Computing selective history mechanism. For ex- tion to high volumes of pricing and Technologies." ample, when subscribing to a topic, ¢OMMUNICATIONSOWTHllACM December 1993/Vol.36, No.12 3~ ® Business Computing an application will often need key region, a doctor reviewing the status prevent users from realizing the po- messages posted to that topic in the of patients in a hospital from a work- tential of the distributed computing past. station at home, a design group col- infrastructure on which their appli- • Customization. The display suggests laborating to develop a new product, cations run. that the system must be easily cus- or application programs cooperating tomized. The information backplane in a factory-floor process control set- Process Groups must be organized in a systematic ting. The software of a modern tele- Two styles of process group usage way (so that the broker can easily communications switching product is are seen in most ISIS applications: track down the name of communica- faced with many of the same issues, tion streams of interest) and flexible as is software implementing a data- Anonymous groups: These arise (allowing the introduction of new base that will be used in a large dis- when an application publishes data communication streams while the tributed selting. To build applica- under some "topic," and other pro- system is active). tions for the networked envi- cesses subscribe to that topic. For an • Hierarchical structure. Although the ronments of the future, a technology application to operate automatically trader will treat the wide-area system is needed that will make it as easy to and reliably, anonymous groups in a seamless way, communication solve these types of problems as it is should provide certain properties: disruptions are far more common on to build graphical user interfaces wide-area links (say, from New York (GUIs) today. 1. It should be possible to send mes- to Tokyo or Zurich) than on local- A central premise of the 1SIS proj- sages to the group using a group ad- area links. This gives the system a ect, shared with several other efforts dress. The high-level programmer hierarchical structure composed of [2, 14, 19, 22, 25] is that support for should not be involved in expanding local-area systems which are closely programming with distributed groups the group address into a list of desti- coupled and rich in services, inter- of cooperating programs is the key to nations. connected by less reliable and solving problems such as the ones 2. If the sender and subscribers higher-latency wide-area communi- previously mentioned. For example, remain operational, messages should cation links. a fault-tolerant data analysis service be delivered exactly once. If the What about the reliability implica- can be implemented by a group of sender fails, a message should be de- tions of such an architecture? In Fig- programs that adapt transparently to livered to all or none of the subscrib- ure 1, the trader has graphed a com- failures and recoveries. The publica- ers. The application programmer puted index of technology stocks tion/subscription style of interaction should not need to worry about mes- against the price of IBM, and it is involves an anonymous use of pro- sage loss or duplication. easy to imagine that such customiza- cess groups: here, the group consists 3. Messages should be delivered to tion could include computations crit- of a set of publishers and subscribers subscribers in some sensible order. ical to the trading strategy of the that vary dramatically as brokers For example, one would expect mes- firm. In Figure 2, the analysis pro- change the instruments they trade. sages to be delivered in an order con- gram is '"shadowed" by additional Each interacts with the group sistent with causal dependencies: if a copies, to indicate that it has been through a group name (the topic), message m is published by a program made fault-tolerant (i.e., it would but the group membership is not that first received m] . mi, then m remain available even if the broker's tracked or used within the computa- might be dependent on these prior workstation failed). A broker is un- tion. Although the processes publish- messages. If some other subscriber likely to be a sophisticated program- ing or subscribing to a topic do not will receive m as well as one or more mer, so fault-tolerance such as this cooperate directly, when this struc- of these prior messages, one would would have to be introduced by the ture is employed, the reliability of the expect them to be delivered first. system--the trader's only action application will depend on the reli- Stronger ordering properties might being to request it, perhaps by speci- ability of group communication. It is also be desired, as discussed later. fying the degree of reliability needed easy to see how problems could arise 4. It should be possible for a sub- for this analytic program. This if, for example, two brokers monitor- scriber to obtain a history of the means the system must automatically ing the same stock see different pric- group--a log of key events and the replicate or checkpoint the computa- ing information. order in which they were received. 2 tion, placing the replicas on proces- Process groups of various kinds If n messages are posted and the first sors that fail independently from the arise naturally throughout a distrib- message seen by a new subscriber will broker's workstation, and activating a uted system. Yet, current distributed be message mi, one would expect backup if the primary fails. computing environments provide lit- messages ml • • • mi-i to be reflected The requirements of modern tle support for group communica- in the history, and messages m i . trading environments are not unique tion patterns and programming. m,, to all be delivered to the new pro- to the application. It is easy to re- These issues have been left to the cess. If some messages are missing phrase this example in terms of the application programmer, and appli- from the history, or included both in issues confronted by a team of seis- cation programmers have been mologists cooperating to interpret largely unable to respond to the chal- 2The application itself would distinguish mes- the results of a seismic survey under lenge.