The following paper was originally published in the Proceedings of the Third USENIX Conference on Object-Oriented Technologies and Systems Portland, Oregon, June 1997

Using the Strategy to Compose Reliable Distributed Protocols

Benoit Garbinato, Rachid Guerraoui Laboratoire de Systemes d'Exploitation Departement d'Informatique Ecole Polytechnique Federale de Lausanne, Suisse

For more information about USENIX Association contact: 1. Phone: 510 528-8649 2. FAX: 510 548-5738 3. Email: [email protected] 4. WWW URL: http://www.usenix.org Using the Strategy Design Pattern to

Compose Reliable Distributed Protocols

BenoÃõt Garbinato Rachid Guerraoui Laboratoire de SystemesÁ d'Exploitation Departement d'Informatique Ecole Polytechnique Fed erale de Lausanne, Suisse e-mail: [email protected]¯.ch

Abstract tems respectively. In [7], we also show how BAST allows distributed applications to be made fault- Reliable distributed systems involve many complex tolerant, by application programmers who are not protocols. In this context, protocol composition is necessarily skilled in reliability issues. a central concept, because it allows the reuse of robust protocol implementations. In this paper, we describe how the Strategy pattern has been recur- The BAST Framework sively used to support protocol composition in the Building reliable distributed systems is a chal- BAST framework. We also discuss design alterna- lenging task, as one has to deal with many com- tives that have been applied in other existing frame- plex issues, e.g., reliable communications, failure works. detections2, distributed consensus,replication man- agement, transactions management, etc. Each of these issues corresponds to some distributed proto- 1 Introduction col and there are many. In such a protocol ªjun- gleº, programmers have to choose the right proto- This paper presents how the Strategy pattern has col for the right need. Besides, when more than been used to build BAST1, an extensible object- one protocol is necessary, the problem of their in- oriented framework for programming reliable dis- teractions arises, which further complicates pro- tributed systems. Protocol composition plays a cen- grammers' task. The BAST framework aims at tral role in BAST and relies on the notion of protocol structuring reliable distributed systems by allow- class. In this paper, we focus on the recursive use of ing complex distributed protocols to be composed the Strategy pattern to overcome the limitations of in a ¯exible manner. For example, by adequately inheritance, when trying to ¯exibly compose proto- composing reliable multicast protocols with trans- cols. In a companion paper [6], we have presented actional protocols, BAST makes it possible to trans- how generic agreement protocol classes can be cus- parently support transactions on groups of repli- tomized to solve atomic commitment [10] and total cated objects. It relies heavily on the Strategy pat- order multicast [20], which are central problems tern, which is recursively used to get around the in transactional systems and to group-oriented sys- limitations of inheritance as far as protocol compo-

 Partially supported by OFES under contract num- sition goes. Our ®rst prototype is written in Small- ber 95.0830, as part of the ESPRIT BROADCAST-WG (num- talk [8] and is fully operational. It is currently being ber 22455) 1We named BAST after the cat-goddess of the Egyptian 2A failure detector is a high- abstraction that hides the mythology: cats are known to survive several ªcrashesº. timeouts commonly used in distributed systems [2]. Protobject abstract superclass unreliable message passing unreliable UMPObject message passing best-effort BMPObject message passing reliable reliable message passing RMPObject message passing fifo FMPObject message passing reliable RMCObject reliable multicast multicast failure detection FDTObject failure detection

uniform consensus CSSObject consensus

dynamic dynamic terminating multicast DTMObject terminating multicast (a) protocol dependencies (b) protocol class hierarchy

Figure 1: Protocols and Protocol Classes in BAST used for teaching reliable distributed systems and marizes the contribution of this paper, as well as for prototyping new fault-tolerant distributed pro- some future developments in the BAST framework. tocols. Adding more and more protocol classes will help us to further test our approach. BAST has also been recently ported to the Java [9] programming 2 Protocol Objects environment. Performance is not yet good enough for practical application development, but we are The BAST framework was designed to help pro- currently working on performance evaluations and grammers in building reliable distributed systems, code optimization [7]. and is based on protocols as basic structuring com- ponents. With BAST, a distributed system is com- Overview of the Paper posed of protocol objects that have the ability to remotely designate each other and to participate in

Section 2 introduces the concept of protocol object various protocols. A distributed protocol  is a set

as de®ned in BAST, and how it helps to structure dis- of interactions between protocol objects that aim at  tributed systems and to deal with failures. Section 3 solvingdistributedproblem  .Weusea Object

discusses why inheritance alone is limited in sup- to name a protocol object capable of participating  porting ¯exible protocol composition and presents in protocol  , and we say that Object is its

how we applied the Strategy pattern to break these protocol class. Each  Object provides a set of limitations. We also show how the Strategy pattern operations that implement interface protocol  , i.e., is transparently used in a recursive manner, and these operations act as entry points to the protocol. we present what steps have to be performed in or- Abstract class Protobject is the root of the pro- der to extend BAST through protocol composition. tocol class hierarchy. Section 4 discusses various design alternatives, and With such broad de®nitions, any interaction be- compares our approach with other research works tween objects located on distinct network nodes is described in the literature. Finally, Section 5 sum- a distributed protocol, even a mere point-to-point communication. For example, class RMPObject 3 Strategy Pattern in BAST implements a reliable point-to-point communica- tion protocol and provides operations rSend() Composing Protocols and rDeliver() that enable to reliable sending With protocol objects, managing protocol depen- and receiving, respectively, of any object3; callback dencies is not only possible during the design and operation rDeliver() is rede®nable and is said implementation phases (between protocol classes), to be triggered by the protocol. Note that such a ho- but also at runtime (between protocol objects). This mogeneous view of what distributed protocols are is partly due to the fact that protocol objects can ex- does not contradict the fact that some protocols are ecute more than one protocol at a time. In this more basic than others. Communication protocols, context, trying to compose protocols comes down for example, are fundamental to almost any other to answering the question ªHow are protocol layers distributed protocol. assembled and how do they cooperate?º. Figure 2 (a) presents a runtime snapshot of aCSSObject, some protocol object of class CSSObject that implements an algorithm for solving the distributed consensus problem. The

Dealing with Failures. Because failures are part consensus problem is de®ned on some set  of dis- of the real world, there is the need for reliable dis- tributed objects as follows: all correct objects in tributed protocols, e.g., consensus, atomic commit-  propose an initial value and must reach agree- ment, total order multicast. Reliable distributed ment on one of the proposed values (the deci- protocols are challenging to implement because sion) [3]. Class CSSObject de®nes operations they imply complex relationships with other un- propose() and decide(), which mark the be- derlying protocols. For example, both the atomic ginning and the termination of the protocol re- commitment and the total order multicast rely on spectively [2]. Besides consensus, protocol ob- consensus, while the latter is itself based on failure ject aCSSObject is also capable of executing detections, on reliable point-to-point communica- any protocol inherited by its class, e.g., reliable tions, and on reliable multicasts. In turn, reliable point-to-point communications and reliable multi- multicasts can be built on top of reliable point-to- casts, as well as failure detections. In Figure 2 (a), point communications. Figure 1 (a) presents an aCSSObject is concurrently managing ®ve dif- overview of some distributed protocol dependen- ferent protocol stacks for the application layer, and cies. issuing low-level calls to the transport layer. Focus- ing on the consensus stack, protocol composition In BAST, protocol classes are organized into a means here to assemble various layers, each being single inheritance hierarchy which follows protocol necessary to execute the consensus protocol, into dependencies, as pictured in Figure 1 (b). Each the protocol stack pictured in Figure 2 (b). The protocol class implements only one protocol, but assembling occurs at runtime and creates a new

instances of some  Object class can execute any stack each time the application invokes operation

protocol inherited from  Object's superclasses. propose(). Protocol objects are able to run several executions of identical and/or distinct protocols concurrently. Inadequacy of Inheritance Alone. With BAST, distributed applications are structured according to their needs in protocols: they are made of protocol

3 objects, which act as distributed entities capable of We mean here any object that is not a protocol object. executing various protocols. With this approach, it Allowing the sending of protocol objects across the network implies the solving of the distributed object migration problem. all comes down to choosing the right class for the We did not address this issue in our framework yet. right problem. We believe that inheritance is an application layer rSend() rmcast() uDeliver() decide() propose() doSuspect() propose() decide()

CONSENSUS ALGORITHM

FDT Algo

RMC Algo

aCSSObject RMP Algo send() recv() send() recv()

transport layer

(a) a protocol object in action (b) interacting protocol layers

Figure 2: Protocol Layers and Protocol Objects

appropriate tool to achieve this: by passing appro- each  Object protocol class independent of the

priate arguments to protocol operations and by im- algorithm supporting protocol  is precisely what plementing callback operations, programmers have we need to be able to compose reliable distributed the ability to tailor generic protocol classes to their protocols in a ¯exible manner. needs. However, we claim that inheritance alone is In the BAST framework, strategy objects repre- not suf®cient as far as protocol composition goes, sent protocol algorithms and they are instances of because it does not offer enough ¯exibility. For subclasses of class ProtoAlgo.AProtoAlgo

example, inheritance does not allow for the easy subclass that implements an algorithm for solving  implementation of a new algorithm for some ex- problem  is referred to as class Algo.Inthe isting protocol, and then to use it in various proto- Strategy pattern terminology, a protocol algorithm,

col classes that are scattered in the class hierarchy. instance of some  Algo class, is a strategy,anda

Furthermore, inheritance is not appropriate when protocol object, instance of some  Object class, it comes to choosing among several protocol algo- is a context. A strategy and its context are strongly rithms at runtime. These limitations lead us to seek coupled and the application layer only deals with

an alternative solution for ¯exible protocol compo- instances of  Object classes, i.e., it knows noth- sition. ing about strategies.

Protocol Algorithms as Strategies Strategy/Context Interactions. Figure 3 (a) sketches the way protocol objects and algorithm According to Gamma et al., the intent of the Strat- objects interact. On the left side, protocol object

egy pattern is to ªde®ne a family of algorithms, a Object offers the services it inherits from its encapsulate each one, and make them interchange- superclasses, as well as the new services that are

ableº [5, page 315]. This is usually achieved by speci®c to protocol  . The actual algorithm im-  objectifying the algorithm [4], i.e., by encapsulat- plementing protocol  is not part of a Object's ing it into a so-called strategy object; the latter is code; instead, the latter uses services provided by

then used by a so-called context object. Making strategy a Algo (on the right side of Figure 3 (a)). specific services (π) based on provided services (π) aπObject aπAlgo based on inherited services required services propose() decide() propose() decide()

CONSENSUS ALGORITHM

(a) Context and strategy (objects) aCSSAlgo FDT Algo startMonitor() rmcast() stopMonitor() rmDeliver() doSuspect() RMC Algo doNotSuspect() rSend() π isSuspecting() rDeliver() Object [ Context ] RMP Algo inherited services [ ContextInterface() ]

πAlgo [ Strategy ] (c) Strategy as protocol layer provided services [ AlgorithmInterface() ]

(b) Context and strategy (classes)

Figure 3: Strategy Pattern in BAST

Whenever an operation related to protocol  is in- Consequences. The context/strategy separation

voked on a Object, the execution of the pro- enables the limitations of inheritance to be over-

tocol is delegated to strategy a Algo. In turn, come, as far as protocol composition goes. One the services required by the strategy to run proto- could for example optimize the reliable multicast

col  are based on the inherited services of context algorithm and use it in some protocol classes, while

a Object. Such required services merely iden- leaving it unchanged in others. Protocol algorithms tify entry point operations to underlying protocols could even be dynamically edited and/or chosen,

needed to solve problem  . according to criteria computed at runtime; this fea-

Each instance of class  Algo represents one ture is analogous to the dynamic interpositioning

execution of protocol  implemented by that class, of objects. There is a minor compatibility con- and holds a reference to the context object for straint among different protocol algorithms in order

which it is running; any call to the services re- to make them interchangeable: new algorithm class

 

quired by the strategy will be issued to its context Algon can replace default Algo in protocol

 

object. There might be more than one instance of class Object if and only if Algon requires a

the same ProtoAlgo's subclass used simultane- subset of the services featured by  Object.

ously by a Object. At runtime, the latter main- tains a table of all strategies that are currently in This approach also helps protocol programmers

execution for it. Each message is tagged to enable to clearly specify, for each protocol  , its depen-

a Object to identify in which execution of what dencies with other protocols. One drawback of the protocol that message is involved, and to dispatch it Strategy pattern is the overhead due to local interac-

to the right strategy. Figure 3 (b) presents the rela- tions between strategies and contexts. In distributed  tionship between classes  Object and Algo, systems however, this overhead is small compared using a class diagram based on the Object Model- to communication delays, especially when fail- ing Technique notation [19]. The correspondence ures and/or complex protocols are involved. More

between  Algo strategy objects and layered pro- speci®cally, the time for a local Smalltalk invo-

tocol stacks is pictured in Figure 3 (c): at runtime, cation is normally under 100 s, whereas a reli- each strategy object represents a layer in one of the able multicast communication usually takes more

protocol stacks currently in execution. than 100 ms when three or more protocol objects application layer

rSendrDeliver rmcastrmDeliver startMonitorstopMonitordoSuspectdoNotSuspectproposedecide

CSSAlgo

FDTAlgo

RMPAlgo RMCAlgo CSSObject

send recv

transport layer

Figure 4: Recursive Use of the Strategy Pattern

4

are involved (without even considering failures). operation, context aRMCObjecti ®rst creates a

The gain in ¯exibility clearly overtakes the local strategy aRMCAlgoi , and then invokes operation overhead caused by the use of the Strategy pattern. rmcast() on it, with the arguments it just re-

ceived. Strategy aRMCAlgoi builds message Äm, Reliable Multicast: an Example containing both m and destSet. It then issues a reliable point-to-point communication with each We now present how we implemented reliable mul- protocol object referenced in destSet;inorder ticast communications using the Strategy pattern.

to do this, strategy aRMCAlgoi relies on inher- In BAST,classRMCObject provides primitives

ited service rSend() of context aRMCObject i . rmcast() and rmDeliver() that enable the

When message Äm reaches aRMCObjectt , one of sending and receiving, respectively, of a message m the target objects, operation rDeliver() is trig- to a set of protocol objects referenced in destSet, gered by the protocol. Operation rDeliver() in a way that enforces reliable multicast properties. detects that Äm is a multicast message and forwards The current implementation of class RMCObject

it to aRMCAlgot , the strategy in charge of that relies on strategy class RMCAlgo. particular execution of the reliable multicast pro-

tocol. When aRMCAlgot receives Äm for the ®rst Overview of the Protocol. The protocol starts time, it re-issues a reliable point-to-point commu- when operation rmcast() is invoked on some nication with each protocol object referenced in destSet (extracted from Äm), and then invokes

initiator object aRMCObject i , passing it a mes-

sage m and a destination set destSet.Inthis rmDeliver() on its context aRMCObjectt , passing it message m (also extracted from Äm). This 4On a 10 Mbits Ethernet connecting Sun SPARCstations 20. retransmission scheme is necessary because of the Overview of the Protocol. The protocol starts agreement property of the reliable multicast prim- by the invocation of operation dtmcast() on itive, which requires that either all correct objects an initiator object, passing it a message m and in destSet or none receive message m [2]. a set of protocol object references destSet. This invocation results in a reliable multicast of m to the set of participants objects. When mes- Recursive Use of the Strategy Pattern sage m reaches some participant, the protocol triggers operation dtmReceive(), passing m

When solving distributed problem  , one can strictly focus on the interaction between class as argument. The participant object then com-

putes a reply and returns it. Eventually, opera-   Object and class Algo, while forgetting dtmInterpret() about how other protocols are implemented. In tion is triggered by the pro- particular, all protocols needed to support proto- tocol on each non-faulty participant object, taking replySet, a subset of the participants' replies, col  are transparently used through inherited ser- as argument. The protocol insures that all correct vices of class  Object. Those services might also be implemented applying the Strategy pattern, participant objects get the same subset of replies, but this is transparently managed by inherited oper- i.e., a consensus has been reached on that set.

ations of  Object.Inthatsense,BAST uses the Strategy pattern in a powerful recursive manner. Methodology for Extending BAST. A®vesteps The recursive use of the Strategy pattern is il- methodology guides programmers in extending the lustrated in Figure 4. The latter schematically BAST framework using the Strategy pattern. We presents a possible implementation of protocol illustrate each of these steps below, by presenting class CSSObject presented in Section 3, which how the methodology was applied to the design enables the solution of the distributed consensus of class DTMObject. Figure 5 summarizes the problem by providing operations propose() and methodology. decide(). In Figure 4, the gray oval is context class CSSObject, while inner white circles are

1. Establish what services the new protocol  various  Algo strategy classes ( being different class DTMObject provides, i.e., what op- protocols). Arrows show the connections between erations are given to programmers want- provided services (top) and required services (bot- ingtouseDTMObject; those opera- tom) of each strategy class. Operations provided tions are dtmcast(), dtmReceive() and by class CSSObject are grouped on the applica- dtmInterpret(). tion layer side (top). Each strategy class pictured in Figure 4 is managed by the corresponding context 2. Choose an algorithm implementing DTM and class in the protocol class hierarchy presented in determine what services it requires, by decom- Figure 1 (b). posing it in a way that allows to reuse as many existing protocols as possible; those services Extending the BAST Framework are: consensus, failure detections, as well as reliable point-to-point and reliable multicast Basing the BAST framework on the Strategy pattern communications (see [11] for algorithmic de- has the advantage of making it easily extensible. tails). To illustrate this, we now present how we built DTMObject, a protocol class supporting the Dy- 3. Implement the chosen algorithm in some namic Terminating Multicast (DTM) protocol [11] DTMAlgo class; all calls to the above required from existing contexts and strategies. The DTM services are issued to an instance variable rep- protocol can be understood as a common denomi- resenting the context object, i.e., an instance nator of many reliable distributed algorithms [12]. of class DTMObject. dtmcast() 1. Establish the DTM specification of dtmReceive() s dtmInterpret() 1. dtmcast() protocol DTM. (i.e., provided services). dtmReceive() startMonitor() rmcast() 3. dtmInterpret() stopMonitor() rmDeliver() 2. Determine the services required by doSuspect() DTMAlgo doNotSuspect() rSend() 2. startMonitor() rmcast() the chosen algorithm for DTM . isSuspecting() rDeliver() stopMonitor() rmDeliver() s doSuspect() propose() doNotSuspect() rSend() 3. Implement that algorithm in a decide() isSuspecting() rDeliver() propose() subclass of class ProtoAlgo. decide() 4. (Let’s name it DTMAlgo). CSSObject rmcast() startMonitor() rmDeliver() 4. Choose a subclass of class stopMonitor() rSend() 5. dtmcast() doSuspect() rDeliver() dtmReceive() Protobject offering at least doNotSuspect() dtmInterpret() all services defined by step 2. isSuspecting() propose() dtmcast() decide() dtmReceive() (Let’s take class CSSObject). dtmInterpret() DTMAlgo 5. Subclass CSSObject into class startMonitor() startMonitor() rmcast() rmcast() stopMonitor() stopMonitor() rmDeliver() rmDeliver() DTMObject. The implementation doSuspect() doSuspect() doNotSuspect() doNotSuspect() rSend() rSend() merely “connects” provided and required isSuspecting() isSuspecting() rDeliver() rDeliver() propose() services of DTMAlgo, to specific decide() and inherited services of DTMObject propose() DTMObject respectively. decide()

Figure 5: Extending BAST with Protocol Class DTMObject

4. Choose the protocol class that will be derived Because protocol objects are the basic address- to obtain new class DTMObject; the choice able distributed entities in our approach, it is not of class CSSObject is directly inferred from possible to guarantee that there will never be more step 2, since the chosen superclass has to pro- than one protocol execution involving each proto- vide at least all the services required by pro- col object at a given time. For example, we cannot tocol DTM. make sure that there will not be two concurrent mul- ticast communications and/or transactions involv- 5. Implement class DTMObject by connecting ing the same protocol objects. Allowing concur- services provided by class DTMAlgo to new rency at this level is an essential feature. Moreover, DTM-speci®c services of class DTMObject, as far as protocol composition is concerned, single and by connecting services required by class inheritance is inadequate for offering a satisfactory DTMAlgo to corresponding inherited services degree of ¯exibility. of class DTMObject. For all these reasons, we made BAST evolve through a second implementation of which the main goal was to overcome the limitations mentioned 4 Design Alternatives above. We now discuss some design alternatives that were considered in the process of implement- Our ®rst implementation of BAST was not based ing this second prototype of BAST, together with on the Strategy pattern, i.e., distributed algorithms design issues that we studied from other existing were not objects, and protocol objects were not frameworks described in the literature. capable of participating in more than one proto- col execution concurrently. Furthermore, proto- col composition was only possible through single Multiple Inheritance and 5 inheritance . Although our prototyping language does not offer 5Remember that we used Smalltalk as implementation lan- multiple inheritance, assembling the various pro- guage for prototyping. tocol layers through this code reuse mechanism is

very appealing6. The idea is to make each pro- mation chunks, which can be assembled in order to  tocol class  Object implement only protocol , create protocol layers and protocol stacks. Various while accessing all required underlying protocols patterns are also provided to help programmers in through unimplemented operations; each protocol building protocols. class is then an abstract class and we usually say However, there is no such thing as protocol ob- it is a class or simply a mixin. Before being ject in either of the above frameworks. Since our able to actually instantiate a protocol object, one main intent is to provide programmers with a pow- ®rst has to build a new class deriving from all the erful unifying concept, the protocol object,wedid necessary mixins. not choose a toolbox approach for BAST.Further- There are three major drawbacks with this ap- more, ASX does not promote protocol composition, proach. First, protocol classes are not more ready- whereas CONDUITS+ does it in a slightly different to-use components: a fairly complex multiple sub- waythanBAST, as we discuss below. classing phase is now required. As consequence, programmers have to deal with protocol relation- Black-box Framework ships ªmanuallyº. Second, protocol layers can only be assembled through subclassing, and it is thus CONDUITS+ offers basic elements that helps pro- dif®cult if not impossible to compose protocol at grammers build protocol layers. The use of design runtime: in several programming languages, e.g., patterns is motivated by the fact that traditional lay- C++, classes are only compile-time entities. Third, ered do not allow code reuse across we still have to manage concurrent protocol exe- layers, which is precisely what CONDUITS+ aims at. cutions within the same protocol object, while this Protocols can then be composed with CONDUIT+, problem is handled nicely as soon as algorithms are at lower-level than BAST, through the assembling manipulated as objects. of conduits and information chunks, which are el- ementary blocks used to build protocol layers. In Toolbox Approach other words, the CONDUIT+ framework does not allow the manipulation of protocol layers as ob- Another possible approach to the reuse of protocol jects, but only the manipulation of pieces of proto- implementations is to provide programmers with a col layers. Compared to BAST, protocol algorithms toolbox containing reusable components and asso- are further decomposed in CONDUIT+: conduits ciate them with . Both ASX [21] and and information chunks are ®ner grain objects than CONDUITS+ [13] frameworks can be seen as such BAST's strategies. Indeed, strategies represent pro- toolboxes. The ASX framework provides collabo- tocol layers, while conduits and information chunks rating C++ components, also known as wrappers, are internal components of protocol layers. CON- that help in producing reusable communication DUIT+ goes one step further in the process of ob- infrastructures. These components are designed jectifying protocol algorithms. to perform common communication-related tasks, This approach makes it easy for CONDUIT+to e.g., event demultiplexing, event handler dispatch- be a pure black-box framework, while BAST com- ing, connection establishment, routing, etc. Several bines features of both black-box and white-box design patterns, such as the and frameworks7. With BAST, we are considering com- the Acceptor pattern, act as architectural blueprints pletely getting rid of inheritance but this issue has that guide programmers in producing reusable and to be carefully studied, because it would have im- portable code. In CONDUITS+ [13], two kinds of portant consequences on the way BAST can be used objects are basically offered: conduits and infor- by application programmers, i.e., those who have

6Ingalls and Borning have shown how re¯ective facilities of 7In a black-box framework, reusability is mainly achieved Smalltalk can be applied to extend the language with multiple by assembling instances, whereas in a white-box framework, inheritance [14], so we could have used that technique if we it is mainly achieved through inheritance. A black-box frame- really wanted to. work is easier to use, but harder to design. very limited skills in fault-tolerant distributed algo- goals of BAST [7]. Moreover, their approach does rithms. not rely on design patterns. Similarlyto CONDUIT+, they go one step further in their decomposition of Modeling Communications protocol algorithms, by de®ning the notion of vir- tual protocols. The latter ªare not truly protocols in Several systems model communications but do not the traditional senseº [16, page 131] : virtual pro- really address reliability issues, e.g., STREAMS [18] tocols are actually used to remove IF-statements and the x-Kernel [17]. AVOCA [24] de®nes the no- and to place them in the microprotocol graph in- tion of protocol objects, but not in the sense that stead. All those differences can be best understood BAST does; furthermore, it mainly applies to high- by looking at the background domains of the BAST performance communication subsystems. Other library and the x-Kernel respectively. The latter systems offer reliable distributed communications, aims at helping system programmers to customize either based on groups as elemental addressing fa- any communication protocol usually found in mod- cilities, e.g., CONSUL [15], ISIS [1] and HORUS [23], ern operating systems, while the former aims at or based on transactions, e.g., ARJUNA [22]. providing ready-to-use protocol classes, in order to help any programmer to build fault-tolerant ap- Microprotocols and the x-Kernel plications, and at allowing skilled programmers to build news fault-tolerant protocols easily. The work done by O'Malley and Peterson [16] is the closest to BAST that we could ®nd. They extended the x-Kernel with the notion of micro- protocol graph, and they described a methodology for organizing network software into a complex graph, where each microprotocol encapsulates a Composing Protocol Stacks in HORUS single function. In contrast, conventional ISO and TCP/IP protocol stacks have much simpler proto- As far as protocol composition is concerned, the col graphs, with each layer encapsulating several HORUS system enables the building of protocol related protocol functions. They argue that such a stacks from existing layers only in a strictly ver- ®ne-grain decomposition allows for better tailoring tical manner. Furthermore, it is based on groups of communication protocols to application needs; as fundamental addressing and communication fa- our conclusion concurs with theirs perfectly on that cility, and provides no framework and/or pattern point. In their paper, O'Malley and Peterson mainly for building new protocols layers. HORUS merely apply their approach to RPC communications (with provides a ®nite set of ready-to-use protocol lay- only one very short discussion of what they call a ers, which can only be composed around the group fault-tolerant multicast). Compared to BAST,their membership protocol. approach is very close to what we have done and is based on the same basic assumption: compos- With BAST, we have tried to model any kind ing (micro-)protocols is essential when it comes to of interaction between distributed objects, not only customizing complex distributed applications (and group communications. This is essential in order to fault-tolerance implies such complexity). In their deal with failures in an extensible way, because re-

terminology, what we call problem  is referred to liable protocols tend to be much more complex than

as metaprotocol  . normal communications. By making protocol ob- There are also some important differences, how- jects BAST's basic distributed entities, we can build ever. They do not provide ready-to-use protocol both the group model and the transaction model [6]. classes to application programmers who are not Furthermore, the Strategy pattern provides a pow- skilled at understanding and/or building complex erful scheme for creating new protocols through protocol graphs, whereas this is one of the main composition. 5 Concluding Remarks cal report, Department of Computer Science, Yale University, June 1983. In this paper, we presented how protocol objects [4] E. Gamma, R. Helm, R. Johnson, and J. Vlis- can help in building reliable distributed systems. sides. Design patterns: Abstraction and reuse We focused on how the Strategy pattern allows the of object-oriented design. In European Confer- limitations of inheritance to be overcome, when ence on Object-Oriented Programming Proceed- trying to compose protocols. As far as we know, ings (ECOOP'93), volume 707 of Lecture Notes in BAST is the only environment to provide both a Computer Science. Springer-Verlag, July 1993. set of ready-to-use protocol objects for building [5] E. Gamma, R. Helm, R. Johnson, and J. Vlissides. fault-tolerant distributed applications, and a com- Design Patterns, Elements of Reusable Object- plete framework based on design patterns, for com- Oriented Software. Addison-Wesley, 1995. posing new protocols from existing ones. We see it [6] B. Garbinato, P. Felber, and R. Guerraoui. as our contribution to the design of well-structured Protocol classes for designing reliable dis- reliable distributed systems. tributed environments. In European Confer- Our current prototype of BAST is fully opera- ence on Object-Oriented Programming Proceed- tional and is available for Smalltalk and Java. At the ings (ECOOP'96), volume 1098 of Lecture Notes moment, inheritance is still partly involved when in Computer Science, Linz (Autriche), July 1996. composing distributed protocols; although a minor Springer Verlag. drawback, this does not make protocol composi- [7] B. Garbinato and R. Guerraoui. Flexible protocol tion as ¯exible as one might expect. This is due to composition in BAST. Technical report, Operat- the fact that programmers have to know something ing System Laboratory (Computer Science Depart- about the implementation of the protocol classes ment) of the Swiss Federal Institute of Technology, they reuse, namely their inheritance relationships. March 1997. This is not surprising, since inheritance is known [8] A.J. Goldberg and A.D. Robson. SMALLTALK-80: to violate encapsulation and to hinder modularity. The Language and its Implementation. Addison Future work will consist of trying to decide if get- Wesley, 1983. ting rid of inheritance, at least as far as protocol [9] J. Gosling and H. McGilton. The Java language composition goes, is a good way to achieve even environment: A white paper. Technical report, more ¯exibility. We are also extending BAST with Sun Microsystems, Inc., October 1995. new protocol classes, supporting frequently used protocols in reliable distributed systems, and op- [10] R. Guerraoui. Revisiting the relationship between timizing existing protocol classes to improve per- non-blocking atomic commitment and consensus. In J.-M. HÂelary and M. Raynal, editors, Distributed formance. Further information about BAST can be Algorithms - 9th International Workshop on Dis- found at http://lsewww.ep¯.ch/bast; our public-free tributed Algorithms (WDAG'95), volume 972 of implementation is also available there. Lecture Notes in Computer Science, pages 87±100. Springer Verlag, September 1995. References [11] R. Guerraoui and A. Schiper. Transaction model vs virtual synchrony model: Bridging the gap. In [1] K. Birman and R. Van Renesse. Reliable Dis- Theory and Practice in Distributed Systems,vol- tributed Computing with the Isis Toolkit. IEEE ume 938 of Lecture Notes in Computer Science, Computer Society Press, 1993. pages 121±132. Springer Verlag, 1995.

[2] T.D. Chandra and S. Toueg. Unreliable failure [12] R. Guerraoui and A. Schiper. Consensus service: detectors for reliable distributed systems. Journal A modular approach for building agreement proto- of the ACM, 34(1):225±267, March 1996. cols in distributed systems. In Proceedings of the 26th Annual International Symposium on Fault- [3] M. J. Fischer. The consensus problem in unreli- Tolerant Computing (FTCS-26), pages 168±177. able distributed systems (a brief survey). Techni- IEEE Computer Society Press, June 1996. [13] H. HÈuni, R. Johnson, and R. Engel. A framework for network protocol software. In Object-Oriented Programming Systems, Languages and Appli- cations Conference Proceedings (OOPSLA'95). ACM Press, 1995. Special Issue of Sigplan No- tices. [14] D.H.H. Ingalls and A.H. Borning. Multiple inher- itance in smalltalk-80. In Proceedings of the Na- tional Conference on Arti®cial Intelligence, pages 234±237. AAAI, 1982. [15] S. Mishra, L. Peterson, and R. Schlichting. Experi- ence with modularity in Consul. Software Practice and Experience, 23(10):1053±1075,October 1993. [16] S. W. O'Malley and L. L. Peterson. A Dynamic Network . ACM Transactions on Com- puter Systems, 10(2):110±143, May 1992.

[17] L. Peterson, N. Hutchinson, S. O'Malley, and M. Abott. Rpc in the x Kernel: Evaluating new design techniques. In Proceedings of the ACM Symposium on Operating Systems Princi- ples (SOSP'89), pages 91±101, November 1989. [18] D. Ritchie. A stream input-output system. Bell Laboratories Technical Journal,63(8):1897±1910, 1984. [19] J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorenson. Object-Oriented Modeling and Design. Prentice-Hall, 1991. [20] A. Schiper and R. Guerraoui. Faul-tolerant total order ªmulticastº with an unreliable failure detec- tor. Technical report, Operating System Labora- tory (Computer Science Department) of the Swiss Federal Institute of Technology, November 1995. [21] D.C. Schmidt. ASX: an object-oriented frame- work for developing distributed applications. In

Proceedings of the 6 th USENIX C++ Technical Conference. USENIX Association, April 1994. [22] S.K. Shrivastava, G.N. Dixon, and G.D. Parring- ton. An overview of the Arjuna distributed pro- gramming system. IEEE Software, 1991. [23] R. van Renesse, K. Birman, R. Friedman, M. Hay- den, and D. Karr. A framework for protocol com- position in Horus. In Proceedings of the ACM Symposium on Principles of Distributed Comput- ing (PODC'95), August 1995. [24] M. Zitterbart, B. Stiller, and A. Tantawy. A Model for High-Performance Communication Subsys- tems. IEEE Journal on Selected Areas in Com- munication, 11(4):507±519, May 1993.