Group Orientation: a Paradigm for Mo dern Distributed Systems

Paulo Verssimo Lus Ro drigues Werner Vogels

[email protected] [email protected] [email protected]

Tech. Univ. of Lisb oa Tech. Univ. of Lisb oa



IST - INESC IST - INESC INESC

Abstract of consistency, user-friendliness, p erformance and de-

p endability.

When lo oking at the several classes of distributed

Increasing use of distributed systems, with the cor-

activities that may take place in a distributed system,

responding decentralisation, stimulates the need for

the group concept app ears intuitively: when a group

structuring activities around groups of participants,

of participants co op erate in an activity (e.g. manage-

for reasons of consistency, user-friend liness, perfor-

ment of a fragmented database, distributed do cument

mance and dependability. Although there is a signi -

pro cessing or distributed pro cess control), comp ete in

cant number of group communication protocolsinthe

an activity (e.g. to share a given resource), or execute

literature, they arepenetrating too slow ly in operating

a replicated activity for p erformance or fault-tolerance

systems technology. Two important reasons are: the

reasons (e.g. replicated database server, replicated ac-

literal interpretation general ly made of the end-to-end

tuator).

argument, and the lack of a layer mapping end-user

needs (management of replication, competition, coop-

Group-orientation, as a general rule, do es not imply

eration and group membership) into what is general ly

group-oriented programming for the end user. That

providedbythecommunication layer: agreement and

is, group to ols can help the programmer build appli-

order properties.

cations where groups are visible , b ecause they are part

of the problem sp eci cation. However, they mayal-

The paper discusses both problems, proposing ways

so b e useful to implement system supp ort function-

for structuring systems and de ning building blocks

s used transparently by the programmer, that is, in

for group-oriented activity, using concepts like objec-

an invisible way. In fact, the utility of groups p er-

tgroups. It suggests that the group concept should

vades all layers of a distributed architecture, from

pervade the whole architecture, from network multi-

multicasting network infrastructures, through group

casting, to group communications and management.

communication and memb ership, to the group activi-

Emerging technology wil l help materialise these con-

ty management services. They may b e at the core of

cepts.

ecient implementations at several levels of abstrac-

tion: op erating system (e.g. microkernel p ort group

management), internetworking (e.g. routing and state

Intro duction

dissemination and management of groups of router-

s), distribution supp ort (e.g. replication management,

The increasing use of distributed systems is in part

distributed lo cks), applications (e.g. co op erativeedi-

due to the requirements of inherently decentralised

tors, teleconferencing, manufacturing cells). In some

activities, such as computer supp orted collab orative

applications, the end user will not see groups at al-

working, or distributed computer control. It b ecomes

l, reasoning in terms of ob ject invo cations, RPCs,

then natural for a numberofsuch computations to b e

and so forth. However, underlying that users's pro-

structured around groups of participants, for reasons

gramming environment, groups may b e ful lling their

r^ole in transparently supp orting a distributed shared



Instituto de Engenharia de Sistemas e Computadores,R.

o

memory mechanism, a replicated ob ject mapp er, a dis-

Alves Redol, 9 - 6 - 1000 Lisb oa - Portugal, Tel.+351-1-

3100281. This work has b een supp orted in part by the CEC,

tributed lo ck manager, etc.

through Esprit Pro jects BROADCAST and DINAS, and JNIC-

Regardless of the current prep onderance of other

T, through Programs Ci^encia and STRIDE. This pap er is an ex-

paradigms, other than group-orientation, for program-

tended and revised version of a pap er that app eared in Pro cs of

the ACM SIGOPS 1992 Workshop, MontSaint Michel-France. ming distributed applications, a few observations are

The combined notion of object and group may provide in order:

designers and programmers with a suitable framework

 Real-time { or responsive { systems deal with

to achieve those ob jectives. Practically all works cited

the environment, thus handling of events is in-

ab ovehave addressed a part of the problem, and a few

evitable. In event-driven systems, events \travel"

of them have tried to systematise solutions.

inside the (distributed) system in the form of mes-

In this pap er, we discuss how to structure groups

sages, or message-interrupts, b efore b eing trans-

in distributed systems, from networking to application

formed. There is a numb er of applications b eing

supp ort. Westartby dissecting two main arguments

b etter addressed in the domain of events [40];

against using groups: the end-to-end argumentand

group visibility.We continue bypresenting an overall

 in highly concurrent and/or interactive system-

picture of the necessary building blo cks of a group-

s (e.g. collab orative group work or distributed

oriented system, then detailing eachoftheblocks. Fi-

computer control) message-passing paradigms are

nally,we do a discussion ab out programming over a

necessary;

group-oriented system, either with group visibilityor

 remote pro cedure call, b eing blo cking, unilater-

not.

al and asymmetric (client-to-server), has some

shortcomings, more evident in the kind of systems

just mentioned; it should b e complemented with

The arguments against groups

paradigms supp orting multilateral, non-blo cking

1

and p eer-to-p eer interactions .

An existing reluctance to having services with high

These observations lead us to the conclusion that a

quality (e.g. reliable multicast) provided o -the-shelf

distributed computing platform can only b ene t from

in a system, has b een related to the literal interpreta-

the co existence of remote-op eration based paradigms,

tion often made of the end-to-end argument [35]. In

such as RPCs, with message-passing, di usion-based

fact, the argument is against providing more func-

ones, such as group proto cols. This, without denying

tionality than needed at a given layer of a system,

the validity and usefulness of either one.

b ecause this go es against optimising eciency,risks

The requirements of highly distributed activities

intro ducing redundant proto col actions like error re-

are not adequately satis ed by the basic interfaces

covery steps, and ultimately, go es against go o d sense.

traditionally supplied as\distribution supp ort", such

Nevertheless, if a class of applications requires a cer-

as \so ckets" or \streams": these are semantically to o

tain functionality (or quality of service), however com-

p o or, and most of them are not multi-participant.

plex it may b e, the lower layer or more generally the

Building blo cks for group activityhave b een studied

operating system support, should provide it. It frees

in the past in pioneering pro jects suchastheV-kernel

the user from programming it, and will probably have

[11], ISIS at Cornell [5], the Circus pro ject [12], and

b een optimised and widely tested when supplied with

also [1,19]. They are currently the sub ject of great in-

a system.

terest, illustrated by pro jects as the PSYNC/x-Kernel

In fact, a numb er of classes of distributed activities

workatUniv. of Arizona [29], the workonobjec-

can b e de ned whose requirements are solved bya

t groups by ANSA [3], the work of Molina [27], the

set of distributed algorithms. In that case, supplying

IBM ight control AAS [17], the Europ ean DELTA-4

a suite of the corresp onding proto cols is in essence a

pro ject [30], the work ab out groups in Arjuna [23], the

correct interpretation of the end-to-end argument, if

currentwork on Isis/Horus [31].

done in a way such that the users not requiring them

In order for applications with high levels of concur-

are not p enalised by their existence. This reasoning

rency to b e correctly designed, have acceptable p erfor-

applies to all levels of the architecture, from network

mance, and remain op erational for long enough, what-

hardware through communications to computing.

ever distribution supp ort environment to b e conceived

One such \semantically-loaded" class of proto col-

must combine: encapsulation, mo dularity and diversi-

s is that of group communication. Though there is

ty; fault tolerance and timeliness; distributed algorith-

a signi cant b o dy of research in this area, they have

mics; supp ort for actions and data (events and state).

had a slow p enetration in op erating system technolo-

1

Engineering-wise, some of these problems have of course

gy.We are convinced that there is a ma jor reason for

b een solved long ago one way or the other (replicated RPC,

this fact, b eyond the end-to-end argument one: vis-

"asynchronous" RPC or RSR, calling pro cess fork, etc.). How-

ibility of the group communication protocols is often

ever, they should b e prop erly addressed at the mo del level, if

awkward for the applications programmer. It is nec- p ossible. ing and co op eration. User User User

Applic. Applic. Applic.

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Structuring Group Supp ort in Dis-

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Replication AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

tributed Systems

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Competition Cooperation AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Replication AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Membership AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

The necessary building blo cks for group-oriented

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Competition Cooperation AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

system structuring and programming are represent-

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Membership AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Replication A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

ed in gure 1. The notion of group p ervades all layers

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

of a distributed architecture, from multicasting com-

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Competition Cooperation A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

unication infrastructures, group communication and

Membership m

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A

time services, to the group activity management ser-

PARTICIPANT

vices just mentioned.

oriented modules

Let us call participant to the end-user of the system

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

SITE AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

supp ort. On the other hand, let us call site to the ma-

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

oriented modules AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

chine as seen from the p ersp ective of the network: no

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA Communication AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

matter how many participants it hosts, messages di-

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Time sGM AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

rected to them are funneled through this single attach-

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Low-level Protocols AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

ment, the machine's network adapter. In consequence,

Network Infrastructure

given that group to ols are fundamentally proto cols, it

is useful to structure them dep ending on among what

they run: inter-participant proto cols run among pro-

cesses,inter-site proto cols run among processors (even

Figure 1: Group Supp ort Building Blo cks

if on b ehalf of pro cesses) [38]. Figure 1 makes explicit

which are the site- and the participant-level mo dules.

essary for the services mentioned ab ove to b e system-

Network Infrastructure

atised and mapp ed into the programmer's universe.

A convincing suggestion is hiding them under an R-

Anumb er of distributed proto cols in the re-

PC environment. This solves only part of the prob-

cent years have b een designed to be network-

lem, namely when one wishes to replicate, maintain-

indep endent [21,5,27]. However, in trying to b e generic

ing a client-server relationship. It do es not contribute

and scalable, they do not takeadvantage of the exis-

greatly when that relation do es not apply or is contra

tence and the emergence of network technologies such

natura {wehave cited some examples in the intro duc-

as LANs and MANs. In consequence, while these will

tion. For these cases, paradigms which adjust to the

p erform well either in lo cal or wide areas, synchronous

event-driven and p eer-to-p eer nature of autonomous

or asynchronous environments, etc., users will b e less

pro cesses apply. They should however b e encapsulat-

and less prepared to pay the cost of technology inde-

ed in a wayastoavoid the inconvenients of directly

p endence when technology is there. Most lo cal enter-

programming with message-passing.

prise, institution or factory settings run over LANs.

We b elieve what is missing in a group-oriented sys- The near future will see organisation-wide distribut-

tem structure, is what might b e called in group jar- ed systems over metrop olitan-area networks. These

gon, a group activity management layer. In essence, are reliable b oth in terms of error rate and availabil-

a building blo ckthatcontrols how groups of partici- ity, display from high to very high sp eed and band-

pants interact, in a number of well-de ned classes of width, have broadcasting/multicasting facilities, and

distributed activities. This layer should address prob- some are capable of real-time op eration and have rea-

lems such as: concurrency control, mutual exclusion, sonably lowdelivery delays. More remotely, meshes of

distributed parallel pro cessing, replicated pro cessing, ATM switches will provide very high interconnectivity

and sp eed down to the workstation, in a fairly large replicated data, etc. Though there are a large num-

geographical scop e. b er of sp eci c solutions for these problems published,

We suggest that proto cols b e prepared to takethe some of them group-based, a systematics for these ser-

b est advantage of the technology they haveavailable. vices is not clear yet. Werisksaying that it should

In the mo del we follow, there are a few key rules: ab- consist of a combination of proto cols for controlling

stract from particular networks but recognise a few basic group activities: memb ership, replication, shar-

rules, without worrying ab out how they are secured. network classes (ex. LAN, INTERNET, MAN, ATM,

The semantics of group communications services can etc.); de ne a set of prop erties for each class; admit

be characterised by combinations of agreement, order di erent abstract networks at di erentlayers of the

2

and synchronism prop erties. These elementary prop- infrastructure ; run parts of the proto col at di erent

erties havebeencharacterised in the literature in the abstract network layers, by delegation [?].

recentyears. Surveys can b e found in [8,30,28]. For the sake of example, a reliable multicast proto-

col running over a large-scale network like the Internet,

For example, the strongest form of agreementis

would b egin its execution by seeing an INTERNET

unanimity, where any message delivered to a recipi-

class network with its weak prop erties, but might del-

ent, is delivered to all correct recipients. Unanimity

egate part of its execution on proto cols running lo cal

may b e unnecessary in some situations. For instance,

to the destination LANs, seeing a LAN-typ e abstract

queries to a replica group need only reach one of repli-

network, and obviously taking advantage of that.

cas, or a quorum of them, it do es not matter exactly

We nd this is a satisfactory compromise for the

which. Relaxed forms of agreement apply then, like

long-lasting contradiction b etween the low-level ap-

ensuring delivery to a numb er of recipientsN(N=0

proach (not-scalable) and the high-level one (not ef-

is the well-known datagram semantics).

cient). Wemaynow present a set of requirements of

In a distributed system, participants must p erceive

the network infrastructure in order to eciently sup-

the order in which actions and events take place. The

p ort groups. In view of the ab ove, some of these fea-

cause-e ect relation is the natural ordering of events

tures may b e provided by only a part of the infrastruc-

in a system. It is called a causal order. This order may

ture, but still play their r^ole in group communications

b e relaxed in some cases, for example to a FIFO ( rst-

eciency:

in- rst-out) order, if senders are not causally related.

For example, when requests from di erent clients to a

 logical group addressing for recipientnumber and

server are commutative. On the other hand, if a par-

lo cation transparency;

ticipant is actively replicated, messages to the replicas

should b e sent in the same order. This is called a total

 hardware multicast and selective addressing for

order.

group and sub-group addressing;

Synchronism of a group proto col can b e measured

 address resolution: handling a large number of

by its steadiness, the greatest di erence b etween de-

addresses eciently;

livery times observed at one site, and its tightness,the

greatest di erence b etween delivery times observed in

 topology should ease multip oint communication;

one execution. According to this, there is a sp ec-

trum from tightly-synchronous [14], through lo osely-

 reliability and availability b ecome more imp ortan-

synchronous [37], to asynchronous proto cols [29,5], de-

t: there are more players involved than in tradi-

p ending on whether those di erences are large or smal-

tional p oint-to-p oint.

l, compared to the execution time, or even not b ound-

ed at all. From the degree of synchronism dep end not

Group Communication

only real-time but also ordering capabilities [40].

In conclusion, a group communication subsystem

The mo dules in the communication supp ort sys-

should:

tem are concerned with site-level proto cols. It may

sometimes b e relevant to include a sub-layer of low-

level protocols which the group communication proto-

 b e prepared to accommo date grades of agreement,

cols may rely on. This o ccurs for example as a set of

order and synchronism;

\sup er-services" of the network infrastructure which

one do es not wish to include in the latter. As an exam-

 supply a range of services each formed byacom-

ple, one mayhave a basic service of ecient large-scale

bination of some of the prop erties ab ove;

multi-destination dissemination, with varying reliabil-

ity.

 select those services to ful ll user needs, i.e. mak-

Group communications services rely on the low-

ing a good use of the end-to-end argument.

level network services. They ensure that a group

of participants exchange messages following a set of

2

Rationale for this exercise can b e found in [21,5,30,

There is an analogy with the ISO layering here, though

26]. using di erent criteria.

from which replies are exp ected). Site memb ership

The problem of memb ership is the problem of

Time and Timing

knowing who or what b elongs to a system or group,

or is present in an activity.Atthispoint, it is im-

The imp ortance of time in distributed systems has

p ortanttodraw a distinction not taken into account

b een largely underestimated. Real-time systems re-

by most group-oriented systems weknow of: the dif-

quire the ability to control duration of activities, re-

ference b etween the site and participantmemb ership

sp onse time, etc. A global reliable time-base accessible

problems.

by all no des is thus mandatory for distributed real-

In a distributed application, there are several par-

time systems. However, since systems are b ecoming

ticipants, sometimes more than one at each site. Hav-

more interactive(multimedia, CSCW, etc.), real-time

ing rules to assess, and sometimes control, their ar-

tends to b e a necessary attribute of systems in general.

rival or departure, is most of the times useful if not

On the other hand, a numb er of distributed algo-

mandatory [32]. This kind of activityis participan-

rithms are based on the existence of a global time no-

t membership management. However, for eachsetof

tion. Even non-real-time or soft real-time applications

participants engaged in a distributed activity, there is

can b ene t from these algorithms and thus we claim

another imp ortant set: the set of sites hosting those

that a reliable, precise and accurate time-base is a very

participants, smaller than or equal to the participant

useful building blo ckinany distributed system. With

set. The management of this set has di erent require-

regard to the way \time" is shipp ed in most existing

ments from the management of participants, so call it

distributed op erating systems, this involves more at-

site memb ership management [34].

tention to:

We b elieve it is of extreme imp ortance to recog-

 precise clo ck synchronisation algorithms, i.e.

nise these memb ership levels, so to sp eak. Firstly,it

maintaining clo cks together;

may simplify the construction of proto cols and of the

participant group management proto cols. As an ex-

 fault-tolerance, i.e. maintaining the time correct,

ample, consider a site with 500 groups. When it fails,

reliable and available;

a system without 2-level memb ership will trigger 500

3

executions of the group memb ership proto col in par-

 external synchronisation ,i.e. accuracy vis-a-vis

allel. In a system with 2-level memb ership, the site

absolute time references (TAI, GMT, UTC).

memb ership proto col may handle the site failure and

intelligently propagate that information to the partic-

Group Activity Management

ipant memb ership proto cols.

Additionally, it separates pro cessor and communi-

Whilst group communication is concerned with al-

cation (site) failure detection from pro cess (partici-

lowing participants of a group to exchange messages

pant) failure detection. While improving the accura-

and establish rules for that exchange, group activity

cy and fairness of such detection, this separation has

management is concerned with de ning and control-

an imp ortant consequence in asynchronous systems.

ling the group ob jectives.

For these systems there is a well-known result stating

Distributed activities can b e reduced to combina-

that it is imp ossible to guarantee consensus [18], exact-

tions of three fundamental op erations, sharing, repli-

4

ly due to the imp ossibility of telling a slow participant

cation, and co op eration . So in the making of a dis-

from a failed one. Theoretically,by separating site and

tributed application we can conceptually visualise pro-

participant failure detection, one narrows the domain

to cols falling in one or more of these activity classes,

under the reach of the FLP result. Site failure de-

complemented with memb ership management:

tection remains unreliable, whereas participant failure

 sharing is the activity concerned with a number

detection, p erformed lo cally, can b e made reliable.

of participants commonly accessing an entityor

Incidentally, in the eld of global-clo ck-less,

resource;

acknowledgement-based proto cols [9,4,29,37], the site

and participant memb ership functions have often b een

3

GPS systems are decreasing impressively in price. The time

aggregated. This separation makes it clear that the re-

is coming when it will b e feasible to have at least one GPS recep-

tor p er system, injecting highly accurate absolute time whichis

quirements of these proto cols, to do correct inter-site

then distributed via internal clo cksynchronisation algorithms.

communication, p ertain to site memb ership manage-

4

With slightly di erent designations, the three-fold charac-

ment (e.g. if the proto col uses acknowledges, it is nec-

terisation of distributed activities was established by LeLan-

n [22]. essary to have a coherent view of the group of sites

 layer compaction, layer transparency or even  replication is the activity where a numberofpar-

layer-lessness; ticipants execute replicas of the same action, or

hold replicas of the same data;

 interface recursivity(up-anddown-calls);

 cooperation is the activity where a numberofpar-

 encapsulation using \large" ob jects;

ticipants execute fragments of an action, or hold

fragments of a data entity;

 op erating system supp ort providing threads, ef-

cient IPC, timer and bu er management, fast

An example of co op eration management is a proto-

user-network information path, user-de nable

col to control a task to b e p erformed in parallel bya

scheduling, easy emb edding of external proto col-

5

group of pro cessors, or a proto col to control the simul-

s .

taneous editing of a do cumentinCSCW[2]. An exam-

ple of replication management are proto cols to control

Two alternative mo dels for op erating system sup-

a set of replicated pro cesses, in order that they p er-

p ort are: (a) top-level opaque interface (e.g. Unix)

form fault-tolerant computations, ensuring whatever

provided by a monolithic op erating system; (b) layer-

actions needed, likevoting, collating, etc. [30].

transparentinterface (one of top interfaces p ossibly U-

Participant memb ership (we discussed site mem-

nix), provided byseveral layers and mo dules on top of

b ership in a previous section) has b een addressed in

and alongside a micro-kernel, allowing visibilityofthe

di erent contexts [15,32,25,36]. In the measure that

inner-most layers from user space, and the addition of

distributed applications are decentralised, dynamic,

extra functionality like the group supp ort proto cols.

unreliable, recon gurable, and sometimes large-scale,

The second is clearly the preferred environment for a

memb ership assumes an essential r^ole in distributed

group-oriented architecture. It is based on the micro-

systems. Memb ership proto cols know who is in and

kernel approach, receiving large supp ort lately.

who is out, and control joins and leaves according to

In this pap er we go as far as suggesting, for the main

prede ned rules. For example, detecting failure and

building blo cks (group management, group communi-

re-establish the level of replication of a group, ensur-

cations + time, network infrastructure):

ing that the necessary \skills" for a co op erative activ-

 they should haveawell-de ned encapsulation and

ity are present in a group, preventing partitioned data

interface;

divergence by allowing progress only in the ma jority

partition.

 inside these blo cks a closer interaction b etween

Take the example of a fragmented and replicated

proto cols may b e desirable for p erformance rea-

database: (i) there should b e memb ership manage-

sons;

ment to control whether all the necessary mo dules

are present | the parts of each replica, the necessary

 they should b e visible to each other and to anyus-

numb er of replicas, the ma jority criterion; (ii) repli-

er program at anylevel, up to the top-level users.

cation management will assist in maintaining consis-

As an example, certain clo ck synchronisation pro-

tency of the replica set (e.g. active or passive repli-

to cols rely on clo ck-less reliable group communication.

cation); (iii) since the database is fragmented among

On the other hand, certain reliable group communica-

several sites, management of co op eration b etween the

tion proto cols (such as the
-typ e proto cols [14]) are

servers controlling the parts of each replica is neces-

clo ck-driven: they rely on the existence of the global

sary; (iv) sharing of access to the database is handled

clo ck provided by the time service and makeheavy

by concurrency control, whichmay b e implemented

use of it. The group communications and time service

in a decentralised way via atomic or causal multicast

should thus share context. The time service should b e

proto cols.

implemented at low level, as close to the network as

p ossible. This reduces the errors in clo cksynchroni-

System Architecture Issues

sation and in time-stamping.

Another example concerns the group managemen-

Lo oking at gure 1, one maywonder howdothose

tlayer. Group memb ership, replication, co op eration

blo cks map into a real architecture. There is no gener-

and sharing are broad designations which illustrate the

al solution, but it is intuitive that such an organisation

basic activities concerned with supp orting any avour

is highly related with some versatility in programming

5

over and within the op erating system. In consequence,

Network requirements have b een discussed in a previous

section. it might b e useful to see attributes such as:

of distributed computation, in combination with, and

User User User making use of, group communication. However, prac-

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

User AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

tical algorithms and proto cols will often combine pro-

AAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Gi AAAA

AAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

User AAAA

AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Gi AAAA

cessing steps having to do with more than one of these

AAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Gi AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

actions. In the example already given, of the control-

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

Gi AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

site oriented AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

site oriented AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

ling proto cols of a fragmented and replicated database,

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

protocols AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

protocols AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

the user (the application builder) will probably see a

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

artitioning/Replication To olBox", where the

\Data P Gi

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

elementary memb ership, co op eration and replication

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA A AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA

AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA AAAA User AAAA

Gi User

t proto cols will make use of each other managemen User

Gi

to p erform the desired functionality. In consequence,

(a) (b)

they should share context.

As a matter of fact, nothing prevents group man-

Figure 2: (a) group visibility; (b) group invisibility

agement and group communications from b eing in the

same pro cess. This may ultimately improve ecien-

cy of their mutual interface, but they should remain

orative (group) work (CSCW); distributed (computer)

as separate entities. Programming these facilities as

control systems (DCS).

kernel extensions is largely advisable.

Rules and means for collab orative group working

are a necessary requirement of practically any collec-

tive activity, more so when assisted by the eciency

Group-Oriented Programming

and cost-e ectiveness of computer supp ort. CSCW

has b een a discipline of growing interest, in the mea-

We will now discuss how groups can assist program-

sure where widespread, ever-increasing use of commu-

ming, using an ob ject-oriented framework for refer-

nications and distributed systems make p ossible its

ence. As noted in the b eginning, groups may b e p er-

application in a large numb er of activities, b oth in

ceived in two fundamental ways by users. One of them

lo cal and geographically broad areas.

is as supp ort for genuinely group-oriented activities,

Applications for collab orativework intro duce par-

that is, ones where groups app ear in the problem sp ec-

ticular requirements on the underlying system. A ma-

i cation, such as in collab orative applications, where

jor requirement for the progress of CSCW is the ex-

many often it is required that the participants have

istence of fundamental supp orting mechanisms, like

col laboration awareness. In that sense, the way to set-

replicated data typ es, distributed lo cks with varying

up the mo del in gure 1, is as depicted in gure 2a:

semantics, group negotiation metho ds, whichmust b e

user ob jects dialog b etween themselves through a pub-

merged b oth with user interface concepts likemulti-

lic ob ject interface whichismuch of a \group agent"

ple views and dialogue encapsulation, and with the

or \group manager". The user ob jects merely con-

adequate architectural supp ort (e.g. reliable and real-

tain their sp eci c functionality, and they use the group

time group communication). Group-orientation is ex-

manager to handle all the asp ects related with their

p ected to provide a unifying approach to the task of

work as a group. Distribution is, so to sp eak, direc-

constructing co op erative applications in a distributed

t[7,41]. The other interaction style concerns problems

environment [13].

where groups, not b eing part of the sp eci cation, may

Distributed computer control systems (DCS) are a

b e part of the solution. That is, they may materialise

very challenging and fast evolving eld. The target

the supp ort for co ordinating the sharing of an ob jec-

systems encountered in the pro cess control area are

t, managing a replicated or fragmented ob ject, and

an ideal eld to explore the notions of direct distribu-

so forth. For clients of such a supp ort environment,

tion, concurrency and groups. However, p erformance

groups are indeed transparent[33,24,23]. This is de-

requirements and the imp ortance attached to these

picted in gure 2b. Note that ob jects in this mo del

problems, mostly money-critical, if not life-critical, de-

have their own metho ds, plus the metho ds to enforce

ter the fast intro duction of new concepts.

the group management p olicies.

Distribution in computerised control has so far b een

almost exclusively limited to networking facilities, to It is p erhaps relevant to sp ecify a little b etter what

down-load and up-load information (e.g. shop- o or we mean bygenuinely group-oriented programming.

data or CNC control programs), or to replace p oint- One may nd examples of its relevance in twovery

to-p oint cabling (e.g. centralised p olling eld-buses). diverse application elds: computer supp orted collab-

supp ort of alternative paradigms, such shared mem- Decentralised approaches where some no de autonomy

ory or transactions. Still, this do es not invalidate the is conferred are normally sp ecialised application-level

concept, and in our opinion the solution may lie in solutions.

taking problem-oriented approaches that provetogive

For an evolution to take place in DCSs, suchnew

correct and satisfactory solutions to those alternative

generation distributed computer control systems must

paradigms.

b e able to provide assurances ab out: correctness;

dep endability and real-time b ehaviour; p erformance;

testability. Recentarchitectural work (MARS [20],

AAS [16], DELTA4-XPA[41]) has shown some path-

References

s to the combination of distribution, fault-tolerance

[1] Mustaque Ahamad and Arthur J. Bernstein. Multicast

and real-time, in what one could call responsive sys-

communication in Unix 4.2bsd. In Proceedings of the The

tems. The notion of groups is of paramount imp or-

5th Intern. Confer. on Distributed Comp. Systems,Den-

tance in these settings, given the natural division of

ver, USA, May 1985. IEEE.

these (sometimes large) systems in small replicated

[2] P.Antunes, N. Guimar~aes, and R. Nunes. Extending the

or co op erative mo dules, with highly concurrentoper-

User Interface to the Multiuser Environment. In Proceed-

ation. Groups obviate the need to reason in terms

ings of the European Conf. on CSCW, CSCW Developers

Workshop, Amsterdam, Septemb er 1991.

of the global system when making and proving asser-

[3] Architecture Pro jects Management, Ltd, Cambridge, UK.

tions ab out correctness, dep endability and timeliness.

The ANSA Reference Manual, release 1.1 edition, July

Group to ols assist the programmer to reason ab out

1989.

correctness of the interactions among group memb ers.

[4] K. Birman and T. Joseph. Reliable Communication in the

Presence of Failures. ACM, Transactions on Computer

Systems, 5(1), February 1987.

Conclusions

[5] Kenneth P. Birman. The pro cess group approach to reli-

able distributed computing. Technical rep ort, Cornell U-

niversity, Ithaca, USA, July 1991.

The concepts advanced in this pap er need not

[6] Kenneth P. Birman. A Resp onse to Cheriton and Skeen's

b e taken as whole, or in rupture with traditional

Criticism of Causal and Totally Ordered Communication.

paradigms. Firstly, they are valid at di erent levels

Technical rep ort, Cornell University, Octob er 1993.

of abstraction: the user depicted in gure 1 may ei-

[7] K.P. Birman and T.A. Joseph. Exploiting replication in

ther b e a programmer or an op erating system nucle-

distributed systems. In Sap e Mullender, editor, Distributed

Systems, pages 319{366. ACM Press Frontier Series, 1989.

us. Secondly, there is a migration path which start-

s in the lower layers of the system. Alotmaybe

[8] K.P. Birman and T.A. Joseph. Reliable broadcast proto-

cols. In Sap e Mullender, editor, DistributedSystems.ACM

done in improving the way distributed systems work

Press Frontier Series, 1989.

at low level, while preserving the traditional shared-

[9] J. Chang and N. Maxemchuck. Reliable broadcast pro-

memory and/or RPC-based mo dels to the applications

to cols. ACM, Transactions on Computer Systems, 2(3),

programmer. Micro-kernels can takeadvantage of the

August 1984.

group networking supp ort mechanisms mentioned, or

[10] D. Cheriton and D. Skeen. Understanding the Limita-

of low-level site memb ership management (e.g. for

tions of Causally and Totally Ordered Communication.In

managing p ort sets). Low-level group communica-

Proceeding s of the 14th Symposium on Operating Systems

Principles, Asheville, NC, USA, Decemb er 1993.

tion proto cols may assist in implementing distribut-

[11] D. Cheriton and W. Zwaenep o el. Distributed pro cess

ed shared-memory approximations, or in p erforming

groups in the V-kernel. ACM Tran. on Computer Systems,

consistent table up dates.

3(2), May1985.

Though the r^ole of low-level group mechanisms

[12] Eric C. Co op er. Replicated distributed programs. In Pro-

to supp ort high-level group-oriented programming is

ceedings of the 10th ACM Symposium on Operating Sys-

probably not controversial, we argue that a system

tems Principles, Berkeley, California 94720, USA, Novem-

structured in this waymay b e as e ective in supp ort-

b er 1985. ACM.

ing group-oriented programs as in supp orting tradi-

[13] Francois Cosquer and Paulo Verssimo. Groupware platfor-

m de nition (in preparation).Technical Rep ort RT-63/93,

tional ones.

INESC, September 1993.

Still, a lot has to b e done in what regards implemen-

[14] F. Cristian, Aghili. H., R. Strong, and D. Dolev. Atom-

tation of these concepts. A recent controversy [10,6]

ic Broadcast: From simple message di usion to Byzantine

shows that there are no general solutions. Implemen-

Agreement. In Digest of Papers, The 15th Internation-

tations conceived to supp ort group-oriented program-

al Symposium on Fault-Tolerant Computing, Ann Arb or-

USA, June 1985. IEEE. ming may present shortcomings when evaluated as

[30] D. Powell, editor. Delta-4 - A Generic Architecturefor [15] Flaviu Cristian. Agreeing on who is presentandwhoisab-

Dependable Distributed Computing. ESPRIT ResearchRe- sent in a synchronous distributed system. In Digest of Pa-

p orts. Springer Verlag, Novemb er 1991. pers, The 18th International Symposium on Fault-Tolerant

Computing,Tokyo - Japan, June 1988. IEEE.

[31] R. van Renesse, Ken Birman, Rob ert Co op er, Bradford

Glade, and Patrick Stephenson. Reliable multicast b etween

[16] Flaviu Cristian. Synchronous atomic broadcast for redun-

microkernels. In Proceedings of the USENIX Workshop

dant broadcast channels. Technical rep ort, IBM Almaden

on Micro-Kernels and Other Architectures, pages 269{283,

Research Center, San Jose,California, USA, 1990.

Seattle, Washington, April 1992.

[17] Flaviu Cristian, Rob ert D. Dancey, and Jon Dehn. Fault-

[32] Aleta M. Ricciardi and Kenneth P. Birman. Using pro cess

tolerance in the Advanced Automation System. In Digest

groups to implement failure detection in asynchronous en-

of Papers, The 20th International Symposium on Fault-

vironemnts. Technical Rep ort TR 91-1188, Cornell Uni-

Tolerant Computing,Newcastle-UK, June 1990. IEEE.

versity, Department of Computer Science, 1991.

[18] M. J. Fischer, N. A. Lynch,andM.S.Paterson. Im-

[33] L. Ro drigues and P.Verssimo. Replicated ob ject man-

p ossibility of distributed consensus with one faultypro-

agement using group technology.InProceedings of the

cess. Journal of the Association for Computing Machin-

4th Workshop on FutureTrends of Distributed Computing

ery, 32(2):374{382, April 1985.

Systems, pages 54{61, Lisb oa, Portugal, Septemb er 1993.

[19] Larry Hughes. A multicast interface for Unix 4.3. Software

Also as INESC AR/28-93.

Practice and Experience, 18(1):15{27, January 1988.

[34] L. Ro drigues, P.Verssimo, and J. Ru no. A low-level pro-

[20] Hermann Kop etz, Andreas Damm, Christian Koza, Marco

cessor group memb ership proto col for LANs. In Proceed-

Mulazzani, Wolfgang Schwabl, Christoph Senft, and Ralph

ings of the 13th International Conference on Distributed

Zainlinger. Distributed Fault-Tolerant Real-Time System-

Computing Systems, pages 541{550, Pittsburgh, Pennsyl-

s: The Mars Approach. IEEE Micro, pages 25{41, Febru-

vania, USA, May 1993. Also as INESC AR/30-93.

ary 1989.

[35] J. Saltzer, D. Reed, and D. Clark. End-to-end arguments in

[21] R. Ladin, B. Liskov, and L. Shrira. Lazy replication: Ex-

system design. ACM Transactions on Computer Systems,

ploiting the semantics of distributed services. In Proceed-

2(4), November 1984.

ings of the Workshop on the Management of Replicated

[36] Andre Schip er and Aleta Ricciardi. Virtually-synchronous

Data, pages 31{34, Houston - USA, Novemb er 1990. IEEE.

communication based on a weak failure susp ector. In

[22] G. Le Lann. An analysis of di erent approaches to dis-

Digest of Papers, The 23th International Symposium

tributed computing. In Proceedings of the 1st International

on Fault-Tolerant Computing, pages 534{543, Toulouse,

Conference on Distributed Computing Systems, Alabama-

France, June 1993. IEEE.

USA, 1979. IEEE.

[37] P.Verssimo and Jose A. Marques. Reliable broadcast for

[23] M.C. Little, D.L. McCue, and S.K. Shrivastava. Main-

fault-tolerance on lo cal computer networks. In Proceedings

taining information ab out p ersistent replicated ob jects in

of the Ninth Symposium on Reliable Distributed Systems,

a distributed system. In Proceedings of the 13th Inter-

Huntsville, Alabama-USA, Octob er 1990. IEEE. Also as

national Conference on Distributed Computing Systems,

INESC AR/24-90.

pages 491{498, Pittsburg, Pennsylvania, USA, May 1992.

[38] P.Verssimo and W. Vogels. The changing face of technolo-

IEEE.

gy in distributed systems. In Proceedings of the 4th Work-

[24] Mesaac Makpangou, Yvon Gourhant, Jean-Pierre Narzul,

shop on FutureTrends of Distributed Computing Systems,

and Marc Shapiro. Fragmented ob jects for distributed

pages 119{127, Lisb oa, Portugal, Septemb er 1993. also as

abstractions. In Advances in Distributed Systems.IEEE,

INESC AR/15-94.

1992.

[39] Paulo Verssimo. Real-time Data ManagementwithClock-

[25] Shivakant Mishra, Larry L. Peterson, and Richard D.

less Reliable Broadcast Proto cols. In Proceedings of the

Schlichting. A memb ership proto col based on partial or-

Workshop on the Management of Replicated Data,Hous-

der. In Proceedings of the 2nd Intl. Working Conf. on De-

ton, Texas-USA, Novemb er 1990. IEEE. also as INESC

pendable Computing for Critical Applications, pages 1{18,

AR/25-90.

Tucson, AZ 85721, USA, February 1991. IFIP WG10.4.

[40] Paulo Verssimo. Real-time Communication. In S.J. Mul-

[26] Shrivakanth Mishra and Richard D. Schlichting. Ab-

lender, editor, Distributed Systems, 2nd Edition,ACM-

stractions for constructing dep endable distributed system-

Press, pages 447{490. Addison-Wesley, 1993.

s. Technical Rep ort TR 92-19, The University of Arizona,

[41] Paulo Verssimo, P.Barrett,P. Bond, A. Hilb orne, L. Ro-

Departement of Computer Science, Tucson, Arizona, USA,

drigues, and D. Seaton. The Extra Performance Archi-

1992.

tecture (XPA). In D. Powell, editor, Delta-4 - A Generic

[27] H. Garcia Molina and Annemarie Spauster. Message or-

Architecture for Dependable Distributed Computing, ES-

dering in a multicast environment.InProceedings of the

PRIT Research Rep orts, pages 267{294. Springer Verlag,

9th Internacional Conference on Distributed Computing

Novemb er 1991.

Systems, pages 354{361. IEEE, June 1989.

[42] Werner Vogels and Paulo Verssimo. Supp orting pro cess

[28] S.J. Mullender, editor. Distributed Systems, 2nd Edition.

groups in internetworks with lightweight reliable multicast

ACM-Press. Addison-Wesley, 1993. proto cols. In Proceedings of the ERCIM Workshop on Dis-

tributedSystems, Lisb oa, Portugal, Novemb er 1991. Also

[29] Larry L. Peterson, Nick C. Buchholdz, and Richard D.

AR/51-91.

Schlichting. Preserving and Using Context Informationin

Interpro cess Communication. ACM Transactions on Com-

puter Systems, 7(3), August 1989.