Quick viewing(Text Mode)

1 Introduction

1 Introduction

Abstraction and Mo dularity Mechanisms

for Concurrent

Gul Agha Svend Frlund Wo oYoung Kim

Ra jendra Panwar Anna Patterson and Daniel Sturman

Department of Science

W Springeld Avenue

University of Illinois at UrbanaChampaign

Urbana IL USA

Email aghafrolundwooyoungrajuannapsturmancsuiucedu

Intro duction

The transition from sequential to parallel and has b een widely ac

cepted as a ma jor paradigm shift o ccurring in It has b een observed that

the sequential programming mo del is inadequate for parallel and distributed computing

At a conceptual level one can compare the fundamental intuition b etween sequential and

as follows

The sequential programming mo del essentially denes a as a sequence of

instructions which manipulate a global store The standard abstraction mechanism for

sequential programs is pro cedures Pro cedures glue a sequence of instructions they

allow genericity by parameterization and reuse by substitution While this provides a

go o building blo ck for much of sequential programming it is an unsatisfactory mo del

for parallel and distributed computing b ecause it do es not provide a communication

mo del and it is not a meaningful abstraction of co ordination b etween concurrent

comp onents

In a paral lel computation some actions overlap in time by implication these events

must b distributed in space Concurrency refers to the potential ly parallel execution

of programs In a concurrent computation the execution of some parts of a program

may b e sequential or it may b e parallel Since concurrent programs sp ecify a par

tial order of actions it provides us with the exibility to interleave the execution of

commands in a program or to run them in parallel Therefore some of the details of

the order of execution are left unsp ecied We can instead concentrate on conceptual

issues without necessarily b eing concerned with the particular order of execution that

may b e the result of the quirks of a given system

Part of the complexity of reasoning ab out concurrent programs results from the fact

that partial orders allow considerable indeterminacy in execution In other words there

are many p otential execution paths Furthermore concurrent programs are complicated by

the fact that there are a numb er of dierent kinds of design concerns such as lo cality and

synchronization that are transparent in sequential execution environments To simplify the

construction of concurrent systems concurrent abstractions must supp ort a separation of

design concerns by providing modularity

The complexity of concurrent systems requires new abstraction metho ds to b e develop ed

There are four imp ortant requirements for concurrency abstractions First the abstraction

must allow sp ecication of the complex organizational and co ordination structures that are

common in concurrent computing Second they must provide genericity and reuse of the

co ordination patterns much as pro cedures do for sequential programming Third concur

rency abstractions must simplify the task of programming by separating design concerns

And nally the abstractions must allow ecient execution on concurrent architectures

This pap er describ es a numb er of radical concepts that supp ort

abstraction and provide mo dularity in concurrent systems Sp ecically the constructs we

prop ose allow abstract and mo dular sp ecication of co ordination patterns temp oral order

ing resource management and dep endability proto cols In particular sp ecications using

these constructs are generic and reusable The next four sections develop our metho dology

and apply it to a numb er of problems as follows

Actors we describ e the Actor mo del of concurrent computation The Actor mo del pro

vides the basic building blo cks for concurrent programming which may b e used to

build a wide variety of computational structures

Communication abstractions three communication abstractions are discussed These

are callreturn communication patterndirected and constraints on

reception To provide a concrete representation we show how callreturn communi

cation is transformed to primitive actor messagepassing

Ob ject oriented design we discuss the use of classes inheritance and incremental mo d

ication of co de

Mo dular decomp osition we describ e a set of abstractions and discuss how they may b e

used to factor out multiactor co ordination patterns resource management strategies

and proto cols for dep endability

Actors

The universe we live in is inherently parallel and distributed This suggests that the natural

language constructs we use to describ e the world may also b e useful for mo deling compu

tational systems It can b e reasonably asserted that the most imp ortant concept we use to

mo del the world is categorizing it in terms of ob jects In fact the rst elements of natural

language children learn are names of ob jects

Computational ob jects encapsulate a state and an exp ected b ehavior Furthermore

ob jects provide an interface dened in terms of the names of pro cedures that are visible

These pro cedures called methods manipulate the lo cal state of the ob ject when invoked

In particular this implies that representations which supp ort the same functionality may

b e interchanged transparently This is an imp ortant engineering advantage which

has proved its utility in sequential ob jectbased programming

Traditional ob jectoriented programming is limited by a mind set which views program

ming as a sequence of actions In particular this mo de confounds the natural autonomy

and concurrency of ob jects sequential ob jectoriented languages allow only one ob ject to

b e active at a time An ob jects b ehavior is viewed as a sequence of actions and this se

quence is blo cked by invoking a metho d in another ob ject This is a rather contrived view

it is more natural to view ob jects as virtual computational agents which may compute

concurrently

The Actor mo del unies ob jects and concurrency The mo dels building blo cks can b e

describ ed and justied in fairly intuitive terms Actors are autonomous and concurrently

executing ob jects which execute asynchronously ie at their own rate Actors may send

each other messages Since actors are conceptually distributed in space communication

b etween them is asynchronous Asynchronous communication preserves the available p o

tential for parallel activity an actor sending a message asynchronously need not blo ck until

the recipient is ready to receive or pro cess a message If a mo del requires a sender to

blo ck it reduces the concurrency which may b e available

In resp onse to receiving a message an actor may take the following sorts of actions

send asynchronously send a message to a sp ecied actor

create create an actor with the sp ecied b ehavior

b ecome sp ecify a new b ehavior lo cal state to b e used by the actor to resp ond to the

next message it pro cesses

The message send primitive is the asynchronous analog of pro cedure invo cation It

is the basic communication primitive causing a message to b e put in an actors mailb ox

mail queue To send a message the identity mail address of the target of a communi

cation needs to b e sp ecied Finally note that although the arrival order of messages is

nondeterministic every message sent to an actor is guaranteed to b e eventually delivered

The become primitive gives actors a historysensitive b ehavior necessary for shared mu

table data ob jects This is in contrast to a purely mo del The

create primitive is to concurrent programming what pro cedure abstraction is to sequen

tial programming Newly created actors are autonomous and have a unique mail address

Furthermore create dynamically extends computational space it thus subsumes the func

tionality of new in Pascal or malloc in C Actor primitives form a simple but p owerful set

up on which to build a wide range of higherlevel abstractions and concurrent programming

paradigms

Communication Abstractions

Although p ointtop oint asynchronous message sending is the most ecient form of com

munication in a distributed system concurrent languages must provide a numb er of com

munication abstractions to simplify the task of programming Programmers using parallel

or distributed computing need to understand the advantages and limitations of dierent

communication abstractions We describ e three basic communication abstractions namely

callreturn communication patterndirected communication and constrained reception

CallReturn Communication

In callreturn communication an ob ject invokes a numb er of other ob jects and waits for

them to return a value b efore continuing execution A standard mechanism for callreturn

communication in concurrent programming is remote procedure cal l a pro cedure calls an

other pro cedure at a remote no de and waits for the result The result is returned to the

p oint where the call is made RPC extends the sequential pro cedure call mo del where

pro cedure calls follow a stack discipline which can b e eciently implemented on sequential

pro cessors In case of highlevel actor languages concurrent RPCstyle calls allow a simple

expression of functional parallelism In actor languages whether two actors are on the same

no de or on dierent no des is transparent to the application co de

Blo cking a sender in a callreturn communication is generally not desirable if the actor

invoked is on a dierent no de available concurrency may b e unnecessarily lost If the sender

holds the pro cessor while busy waiting for results pro cessor time is wasted Otherwise

extra context switching is needed to change the executing actor from the sender to another

actor

Whenever feasible we allow the calling actor to continue computation as so on as it has

asynchronously sent a request In order to supp ort ease of programming without incurring

an unnecessary p erformance p enalty we transform a program containing a callreturn com

munication to a semantically equivalent one containing asynchronous message sends only

The transformations used preserve the maximal concurrency in a program Optimizing this

form of communication by using a concurrent analog of continuation passing style program

transformation avoids incurring unnecessary costs see inset

INSET Program Transformations for CallReturn Communication

We use one of two transformations on callreturn communication First if the

resp onse of a sending actor to the next message is not dep endent on the results from

a callreturn communication the program is transformed by changing the calls to asyn

chronous sends and creating a join continuation actor The join continuation actor

p erforms a part of computation of the original sender actor that is dep endent on the re

sults Consider the following expression

send B v Crequest Drequest

send represents an asynchronous send When executed actor C and D receive messages

request and request resp ectively Then actor B is sent a message with results from

actor C and D along with v Figure and pictorially represent the execution of the

program b efore and after the transformation resp ectively

request2 D reply2 rpc style send AB asynchronous send C request1 reply

reply1

Figure Before the transformation creating join continuation

asynchronous send request1 D reply1

A JC B

C

request2 reply2

Figure After the transformation creating join continuation

Second if the resp onse of an actor to the next message is partly determined by the

results of the calls to other actors we separate out the continuation as a metho d within

the original actor Note that no purp ose would b e served by creating a join continuation

actor the original sender cannot pro cess other messages until a result is received The

continuation metho d is triggered by the results of the remote actor invo cations In order to

guarantee consistency b etween state changes the transformation creates a synchronization

constraint see next section for the continuation metho d This new constraint prevents

other messages from b eing pro cessed until the continuation metho d has b een invoked

Patterndirected Communication

An advantage of p ointtop oint asynchronous communication mechanism is that it allows

lo cality to b e directly expressed and optimized However in some cases it is sucient to

communicate with an arbitrary memb er of a group If the recipient must name all p otential

receivers the b o okkeeping involved can b e cumb ersome Furthermore a level of abstraction

is lost The use of patterndirected communication allows an abstract sp ecication of a

group of p otential recipients Thus the actual recipients may b e transparently changed

none of clients needs to know the exact identities of p otential receivers or to p oll them to

determine if they satisfy some pattern

In the ActorSpace mo del a communication mo del based on destination patterns is

dened An actorSpace is a computationally passive container of actors which acts

as a context for matching patterns Note that actorSpaces may overlap in particular

an actorSpace may b e wholly contained in another Patterns are matched against listed

attributes of actors and actorSpaces that are visible in the actorSpace Both visibility and

attributes are dynamic Messages may b e sent to one or all memb ers of a group dened

by a pattern An actor may send a message to a single arbitrary memb er of a group or

broadcast it to the entire group In particular broadcasting can b e used to disseminate

common proto cols to an entire group

ActorSpace provides a useful mo del for many distributed applications For example if

an actorSpace of servers is dened none of the clients need to know the exact identities

of the p otential servers or explicitly p oll them to determine if particular ones are suitable

This provides an abstraction that allows replication of services for example to enhance

reliability or increase p erformance

Linda denes a communication abstraction similar to that of actorSpace however the

semantics of Linda unlike ActorSpace require explicit read op erations by recipients This

results in at least two signicant dierences First race conditions may o ccur as a result

of concurrent access by dierent pro cesses to a common space Second communication

cannot b e made secure against arbitrary readers for example there is no way for a sender

to sp ecify that a pro cess with certain attributes may not consume a message called a tuple

in Linda By contrast in ActorSpace the attributes of a messages p otential recipient are

determined by the sender

Synchronization Constraints

In sequential programs there is a single of control and the programmer must ex

plicitly x a calling sequence for all ob jects essentially by calling them one at a time and

passing the control to them Generally a stack discipline is used and control returns to the

calling ob ject once the called ob ject has nished executing In fact this calling discipline

corresp onds quite p o orly to the nature of many In concurrent systems more

distributed forms of synchronization supp orting partial orders are needed

Consider pro ducer actors and consumer actors which communicate through a buer

actor Since actors are autonomous and asynchronous they do not know if a particular

message they send may b e meaningfully pro cessed in the current state of a receiving actor

Therefore it is p ossible for a pro ducer actor to send a put request to a full buer or for a

consumer actor to send a get request to an empty buer Figure Thus it is necessary to

sp ecify when dierent computational ob jects may b e invoked Rather than reject messages

which may not b e pro cessed in the current state we defer the request BUFFER ACTOR

PUT GET

PRODUCER ACTOR CONSUMER ACTOR

Figure Pro ducers and Consumers The pro ducer actor dep osits a data item into the

buer by invoking put metho d dened in the buer actor The consumer actor retrieves

a data item from the buer by invoking get metho d of the buer actor The pro ducer

actor may not invoke put metho d on a full buer And the consumer actor may not

invoke get metho d on a empty buer

Our view is that constraints which limit invo cations of a concurrent ob ject ie its

synchronization constraints should b e a part of the actors interface A programmer may

asso ciate such constraints with each metho d The separation of synchronization constraints

frees the programmer from explicitly sp ecifying co de to manage messages received by an

actor which it is not in a state to pro cess A message not satisfying the constraint is buered

until such time when the actors state satises the constraint Essentially synchronization

constraints provide an abstract representation of the reactive b ehavior of an actor Using

synchronization constraints it is p ossible to reason ab out the eect of comp osing actors

Actor programs unlike iteration in sequential programming languages do not use global

lo ops for sequencing actions the usual semantics of such lo ops implies that an iteration

should b e completed b efore the next one may b e initiated Instead actors supp ort message

driven programming Synchronization constraints may b e used to local ly ensure consistent

sequencing of iterations By not creating unnecessary data dep endencies message driven

programming maintains the maximal concurrency available in an see inset

INSET Overlapping Communication and Computation

Actor programming naturally leads to ecient parallel execution in a numb er cases We

illustrate this by an iterative matrix algorithm for the Cholesky Decomp osition CD of a

dense matrix Sp ecically the matrix is represented by a collection of actors which

execute dierent iterations in resp onse to messages they receive Since the arrival order

of messages is indeterminate lo cal synchronization constraints are used to ensure that the

iterations are executed in the correct order

Table compares the results of an implementation of CD algorithm which pip elines

the execution of iterations with an implementation which completes the execution of one

iteration b efore starting the next Note that the pip elining naturally follows from the fact

that control is distributed b etween actors

Matrix

Nodes Seq Pipe Seq Pipe Seq Pipe Seq Pipe

Table Results from an implementation of the CD algorithm on an Intel iPSC The

columns Seq represent the implementation which completes the execution of one iteration

b efore starting the execution of the next iteration The columns Pipe show the values

obtained by pip elining the execution of iterations The times shown are in seconds

Ob jectOriented Design

A natural progression from naming individual ob jects is to name categories or classes of

ob jects Ob jectoriented design provides a hierarchical framework which naturally mo dels

the world using classication Such classication allows us to build categories which provide

shared attributes and functionality By providing a to ol for parsimony of representation

classication simplies our ability to mo del the world

In more concrete terms a class may b e thought of as a category of computational ob jects

which can b e sp ecialized A canonical example of a class is a vehicle which has certain

attributes such as position velocity occupants weight etc and allows certain

metho d invo cations to change some of its attributes A memb er of this class is dened by

sp ecifying its initial attributes state Some of these attributes may never change if the

class do es not contain op erations to change them

A sub class may further sp ecialize the op erations of a sup erclass For example a car

may have more sp ecic attributes and functions as well as those of a vehicle By allowing

a car to inherit co de from a vehicle ob jectoriented languages supp ort incremental

renement and co de reuse Dep ending on the ob ject oriented language a metho d may b e

extended or redened in a sub class

Actors like any ob jects may b e organized into classes and include notions of inheritance

Note that b ecause we sp ecify synchronization constraints in a mo dular fashion they may

b e inherited and incrementally mo died Sp ecically a synchronization constraint may b e

further strengthened weakened or overwritten in a sub class indep endently of whether the

constrained metho ds are changed

Alternately actors may use forms of inheritance only to supp ort metho d co de reuse

Sp ecically delegation is a variant of inheritance which allows the co de of a prototyp e

ob ject to b e reused For example it may not b e meaningful to think of a stack as a kind

of an array but a stack may b e dened by using the representation and op erations of an

array A stack may delegate invo cations of its metho ds to an array

A go o d survey of research in ob jectoriented programming can b e found in A

description of concurrent ob jectoriented programming can b e found in

INSET Incremental Mo dications of Synchronization Constraints

Consider the example of pro ducers and consumers which communicate through a buer

see Figure The buer defers requests from consumers if it is empty The co de for such

a buer may b e sp ecied as follows

class Buffer

var first count

restrict get with count

init

count

end

method putx

method get

end

Now supp ose we want to create a buer which takes in requests as long as the cumulative

size of the p ending requests is smaller than its buering capacity If the buer represents

a fast cache accessed by a numb er of printers it could b e dened as an instance of the

sizedBuffer class which extends the buer class denitions as follows

class sizedBuffer inherits Buffer

var usedCapacity totalCapacity

restrict putx with xsize usedCapacity totalCapacity

init bufferCapacity

usedCapacity

totalCapacity bufferCapacity

end

method putx

superputx

usedCapacity usedCapacity xsize

end

method get

end

Separating Design Concerns

We describ e three mechanisms to develop mo dular and reusable comp onents for concurrent

systems These mechanisms allow

The use of abstractions to sp ecify multiactor co ordination patterns The co ordination

patterns include atomicity and temp oral ordering

Separation of functionality and resource management strategies For example p oli

cies for actor placement may b e sp ecied in terms of an actor group abstraction

indep endent of the representation and invo cations of a particular group satisfying the

abstraction

The ability to develop generic application indep endent co de for proto cols which in

crease dep endability In particular the architecture we prop ose allows the dynamic

installation or removal of proto cols to change the faulttolerance and security charac

teristics of a running system

Synchronizers

Synchronization constraints provide mo dular expression of constraints which need to b e

satised by a single actor b efore it may pro cess a communication it has received Although

synchronization constraints are often promoted as a way to describ e co ordination of con

current ob jects eg they are unsatisfactory when describing multiactor co ordination

synchronization constraints dep end only on the lo cal state of a single actor

In distributed computing a group of ob ject invo cations often must satisfy certain tem

p oral ordering or atomicity constraints Conventional programming languages do not allow

multiob ject constraints to b e sp ecied in a mo dular and reusable manner This creates a

numb er of problems Considerable programming eort is required to express multiob ject

constraints in terms of low level message passing Moreover expressing these constraints

by explicit message passing hard wires b oth the constraints and their implementation

into the application software Thus the same abstract multiob ject constraints must b e re

programmed for use with dierent ob jects Finally the implementation of the multiob ject

constraints may not b e transparently changed These diculties suggest that new highlevel

co ordination language constructs are needed to simplify the task of programming

We have made some progress in this area Sp ecically we have develop ed high level lan

guage constructs which allow multiactor constraints to b e directly expressed We dene

two typ es of multiactor constraints temp oral orderings on and atomicity of invo cations of

shared distributed actors Multiactor constraints are describ ed in terms of conditions that

must b e satised for a group of metho d invo cations to b e accepted if the conditions are

not met the invo cations are delayed Thus multiactor constraints co ordinate concurrent

ob jects by restricting their activation As the following examples suggest a large class of

co ordination schemes can b e eciently expressed using invo cation constraints

Consider a group of co op erating resource administrators who must share a limited

resource The administrators must therefore adhere to a collective p olicy limiting

the total numb er of resources allo cated at a given p oint in time Enforcement of

a collective p olicy can b e expressed as a multiactor constraint on invo cations that

request resources an allo cation request can only b e serviced if there are resources

available

A group of dining philosophers is organized so that each philosopher shares her two

chopsticks with two others The numb er of philosophers is equal to the numb er of

chopsticks and a philosopher needs the two chopsticks next to her in order to eat

Deadlo cks may b e avoided by enforcing a multiactor constraint that requires atomic

invo cation of the pick metho d in two chopstick actors by a single philosopher

We build on the observation that multiactor constraints can b e sp ecied indep endent

of the representation of the actors b eing co ordinated Sp ecifying multiactor constraints in

terms of the interfaces enables b etter description reasoning and mo dication of multiactor

constraints Sp ecically utilizing only knowledge ab out interfaces to describ e multiactor

constraints allows co de for co ordination to b e separated from that for the actors function

ality This separation enables system design with a larger p otential for reuse Actors may

b e reused indep endent of how they are co ordinated conversely multiactor co ordination

patterns may b e reused on dierent groups of actors In particular it is p ossible to abstract

over co ordination patterns and factor out generic co ordination structures

Key:

Constrained , object group

Object

, Atomicity constraints A Disabled pattern

, ,, Enabled patterns

, , ,, Messages

B C

D

E

Figure Overlapping actor groups dened by two synchronizers Messages sent to actors

in the two groups are placed outside the synchronizer b oundaries In this gure pattern

matching is depicted as matching shap es b etween patterns and messages Synchronizers

may disable certain patterns disabled patterns are black and enabled patterns are white

The messages at B are disabled whereas the message at C is enabled The E message is

unconstrained Patterns may b e group ed together into atomicity constraints that ensure

that multiple invo cations are scheduled as an atomic action depicted by b oxes around

the groups The messages at A satisfy the atomicity constraint whereas the message at D

is blo cked waiting for another message

We describ e multiactor constraints using synchronizers Conceptually a synchronizer

is a sp ecial kind of actor that observes and limits the invo cations accepted by a group

of actors The functionality of synchronizers is illustrated in Figure Op erationally

synchronizers are implemented using primitive actor communication The advantage of

synchronizers is that the involved messagepassing is transparent to the programmer who

sp ecies multiactor constraints in a highlevel and abstract way The implementation

of synchronizers may either involve direct communication b etween the constrained actors

indirect communication with a central co ordinator or a hybrid Thus by using a high

level sp ecication of multiactor constraints we provide the exibility to map the same

multiactor constraint to dierent implementations

A synchronizer can b e dened and instantiated by a client actor when accessing shared

servers Thus clients can use constraints to enforce customized access schemes Alternately

a synchronizer can b e p ermanently asso ciated with a group of servers when the servers are

rst put into op eration In this case the constraints can express the default interdep endence

b etween servers

Two other approaches to constraints are develop ed in the systems Kaleidoscop e and

Rapide Constraints in Kaleidoscop e capture relations b etween instance variables of

multiple ob jects Thus Kaleidoscop e constraints are formulated in terms of the repre

sentation of the constrained entities rather than their abstract interfaces The Rapide

prototyping system develop ed by Luckham et al involves patternbased triggering of

concurrent ob jects Thus in Rapide it is not p ossible to express constraints on the invo

cations accepted by the involved actors

INSET Co op erating Resource Administrators

Consider two co op erating resource administrators sp o olers which manage a common

printer p o ol Supp ose the p o ol has n printers When an administrator receives a print

request it p erforms some b o okkeeping computations and then sends the request through a

common bus so that one of the free printers can grab the request and start printing The use

of two sp o olers allows greater concurrency and increases availability We use a co ordination

constraint to ensure that requests are not relayed when there are no free printers The

constraint also ensures that the two sp o olers co op erate to maintain the correct count of

available printers

Maintenance of common information ab out the numb er of free printers can b e describ ed

external to the sp o olers as a synchronizer Figure contains a synchronizer which enforces

the global allo cation p olicy The names spooler and spooler are references to the two

constrained sp o olers The synchronizer prevents the pro cessing of a request when there is

no free printer in the p o ol

Synchronizers are general to ols for describing interdep endence b etween servers p erform

ing a service Using synchronizers interdep endence is expressed indep endent of the repre

sentation of the servers The resulting mo dularity makes it p ossible to mo dify the co ordi

nation scheme without changing the servers and vice versa In particular it is p ossible to

AllocationPolicyspoolerspoolernumPrinters

f numUsed

numUsed numPrinters disables spoolerprint and spoolerprint

spoolerprint or spoolerprint up dates incrementnumUsed

spoolerdone or spoolerdone up dates decrementnumUsed

g

Figure A synchronizer which co ordinates two print sp o olers A synchronizer has an

encapsulated state that is up dated through an up dates op erator The state of the ab ove

synchronizer is held by the variable numUsed The disables op erator delays invo cation of

the constrained actors Delays of invo cations are expressed as conditions over the state of

the synchronizer

dynamically add new printers to the printer p o ol or add new administrators to the system

without changing co des for already existing printers or administrators new synchronizers

may simply b e instantiated

Mo dular Sp ecication of Resource Management Policies

Expressing a in terms of primitive actors provides a logical sp ecication

of the algorithm Such a sp ecication may b e called an ideal algorithm The time taken

by the ideal algorithm in the presence of unb ounded resources and zero communication

cost is determined by the sequential depth of the longest path in the partial order dened

by the actor computation However neither of these assumptions is realistic

In particular communication costs for an algorithm are a function of the latency and

bandwidth of an architecture Latency is the time taken to send a message from one no de

to another and bandwidth is the rate at which information may b e transmitted b etween

two halves of an architecture For example if a problem such as sorting requires half the

data on a distributed computer to b e moved the p erformance of an algorithm solving the

problem will b e b ound by the bandwidth In any physically realizable architecture the

23

bandwidth may grow by at most P where P is the numb er of pro cessors This follows

from the fact that space is three dimensional therefore a given technology yields a constant

23

bandwidth p er unit area and a bisecting plane may grow by at most P There is a

similar theoretical b ound on IO In case of sorting this means that the sp eed up is b ound

p

23

by P in general and P on a two dimensional network

Since the p erformance of an algorithm is dep endent on how many messages have to

b e sent and to which no des the eciency of execution dep ends in part on the placement

and of ob jects In general the problem of nding an optimal placement p olicy

is intractable However for a given algorithm a user may b e able to determine the most

ecient placement p olicy

Sp ecications of resource management p olicies such as placement intro duce a new

layer of complexity to programming concurrent architectures In particular the same ideal

algorithm executed on the same architecture may yield a dierent eciency dep ending on

the resource management p olicy used Sp ecically the eciency obtained may dep end on

a numb er of factors such as

the problem or input size

characteristics of the concurrent computer including its latency bandwidth size and

pro cessor sp eeds

the placement p olicy used to map ob jects to physical resources

scheduling strategies used to manage the concurrency

Current programming metho ds for concurrent intermix sp ecication of re

source management p olicies with the co de sp ecifying the ideal algorithm The resulting

conation of design goals complicates the co de and reduces its reusability We prop ose to

separate the sp ecication of an ideal algorithm from the strategies used to map it to a con

current architecture Sp ecically we describ e a mapping p olicy in terms of ActorSpace Type

AST An AST may b e thought of as a group of actors together with b oth the abstract

op erations and the concurrent access constraints on them

For example consider an n n array The array can b e mapp ed on a twodimensional

mesh of p p pro cessors in a numb er of ways including

th th

placement policy the i j element is assigned to the i div k j div k pro

cessor where k np

th th

Shue placement policy the i j element is assigned to the i mo d p j mo d p

pro cessor

Although the abstract op erations and concurrent access constraints of an array are the

same dierent ideal may b e executed more eciently using dierent mapping

p olicies Linear equation solution techniques such as Gaussian Elimination or Cholesky

Decomp osition are generally ecient when the matrix is mapp ed using a shue placement

p olicy On the other hand algorithms for lowlevel image pro cessing applications domain

decomp osition techniques for solving Partial Dierential Equations p erform eciently when

their matrix representation is mapp ed using the blo ck placement p olicy

Note that the corresp ondence b etween an algorithm and the optimal placement of its

AST is not one to one Dierent placement p olicies may b e more ecient for the same ideal

algorithm on dierent architectures and sometimes for the same algorithm and architec

ture but a dierent input size Organizing a computation in terms of its ASTs provides

mo dularity and promotes reuse The task of programming can b e simplied by comp osing

and reusing mo dules from a rep ository of placement p olicies for a given AST P P A1

P' P''

P' Placement Repositories A1 Linking

A1

A2 A3 P

A3

Ideal Algorithms Architecture Specific Efficient Executable

Code

Figure Combining ideal algorithm sp ecication with a placement p olicy to obtain archi

tecture sp ecic ecient executable co de

INSET Reuse of Resource Management Strategies

Although we are developing b etter language supp ort for resource management functions

in terms of ASTs the functionality we will provide may in some cases b e mimicked using

current to ols For example consider a dense matrix representation A simple set of functions

may b e used to sp ecify a placement of matrix elements which may b e simply used to execute

for example a Gaussian elimination co de or a matrix multiplication co de on a distributed

memory machine Changing the denitions of these functions changes the placement p olicy

and can aect the p erformance drastically

The following Clike co de sp ecies the blo ck placement p olicy N is the numb er of rows

and P is the numb er of pro cessors

row mappingglobal row num f returnglobal row num N g

indexglobal row num f returnglobal row num N g local

global indexlocal row num f returnlocal row num me N g

The ab ove functions may b e easily redened to implement a dierent placement p olicy

such as the shue placement p olicy without changing the co de for an ideal algorithm using

the mapping p olicy

mappingglobal row num f returnglobal row num P g row

indexglobal row num f returnglobal row num P g local

global indexlocal row num f returnlocal row num P me g

Customizing Dep endability

Currently a proto col for dep endability must either b e built into the system architecture or

b e reimplemented for each application Moreover development of dep endable software is

exp ensive the increased complexity caused by mixing the co de for a set of dep endability

proto cols with that of the application co de is itself a source of bugs A signicant savings

in and maintenance costs may b e realized if abstract application

indep endent sp ecications of dep endability proto cols are p ossible

We have develop ed a metho dology which allows the co de for a dep endability proto col

to b e sp ecied indep endently of the application sp ecic co de The metho dology has

b een implemented in an exp erimental kernel called Broadway Our reective mo del allows

compositionality of dep endability proto cols Comp ositionality means that we can sp ecify

and reason ab out a complex dep endability scheme in terms of its constituents Thus logi

cally distinct asp ects of a dep endability scheme may b e describ ed separately resulting in a

metho dology which allows dep endability proto cols to b e implemented as generic comp os

able comp onents

We employ reection as the enabling technology to allow mo dular sp ecication and

dynamic installation of dep endability proto cols Reection means that an application can

access and manipulate a description of its own b ehavior and execution environment The

actors representing such a description are called metalevel actors For our purp oses the

metalevel contains a description sucient to mo del the dep endability characteristics of

an executing application reection thus allows dynamic changes in the execution of an

application with resp ect to dep endability

The most general form of reection leads to interpretation and is costly For our pur

p oses we use a limited reective mo del in which each actor has three metaactors its

dispatcher its mail queue and its acquaintance list The acquaintances metaactor repre

sents the current state b ehavior of the actor The dispatcher and mail queue metaactors

implement the communication primitives of the runtime system so that the interaction b e

tween actors can b e mo died to change the dep endability characteristics of an application

Sp ecically a dispatcher metaactor is a representation of the implementation of an

actors transmission b ehavior When customized messages sent by the corresp onding base

actor are rerouted to the customized dispatcher An actors mail queue metaactor repre

sents the mail buer holding messages received by the actor If a customized mail queue

metaactor is installed all messages to the base actor are rerouted through it A cus

tomized mail queue may alter the order of messages to the base actor eg to enforce lo cal

synchronization constraints

A numb er of proto cols which increase dep endability of a system can b e expressed in

terms of a customized mail queue dispatcher and acquaintance list These proto cols include

two phase commit three phase commit primary backup full replication checkp ointing

and encryption Comp osition of dep endability proto cols is achieved by transparently manip

ulating the metaactors of the metaactors The resulting system allows not only dynamic

installation of generically sp ecied proto cols but their dynamic removal The limited form

of reection we use supp orts incremental compilation and increases execution time by only

a very small constant

INSET A Replicated Server

Client Client A B

Key: Eliminator Eliminator Dispatcher MailQueue Message send Causal Connection

Tagger Forwarder Tagger

Server Server

S1 S2

Figure A replicated service proto col Client A initially sends a message to Server S

1

The message is routed to the mail queue Broadcaster Broadcaster passes a copy of the

message o to S and sends a copy to S Both servers eventually reply and their replies

1 2

are tagged by the two Tagger dispatchers The Eliminator mail queue then passes only

one copy of the two messages onto A

To illustrate how Broadways reective architecture may b e used to supp ort dep end

ability proto cols we describ e the metalevel implementation of a replication proto col The

proto col involves replicating a server to resist crash failures

Our original system is a service with two clients Figure shows the result of installing

the metaactors for this proto col in the system A clone is made of the service and the

appropriate metaactors are installed at the service and at the clients The metaactors

are designed to manipulate generic messages As a result in combination with the sys

tems ability to clone actors this proto col is implemented transparently of the base actors

Furthermore since the proto col is generic it may b e reused with any application

For each client a metaactor a customized mail queue Eliminator is installed

at its no de This mail queue will eliminate duplicate messages from the copies of the server

For the server and its replicated copy customized metaactors are installed to handle the

transmission and reception of messages The dispatcher Tagger tags all outgoing messages

so that the clients using the mail queue Eliminator may eliminate duplicate resp onses

The mail queue Broadcaster copies to server S all messages sent to server S The

2 1

rep etition of messages is necessary to keep the state of the two servers consistent When a

message is sent to the service the get metho d of the customized metamail queue is invoked

instead When the service requires a new message to pro cess the Broadcaster mail queue

is sent a put message The co de for the Broadcaster actor class is shown b elow Notice

that the transparency of the proto col is preserved by the metho ds get and put These

metho ds manipulate entire messages never needing to insp ect the message contents

class Broadcaster

var S Base

Setup this actor

initcopyorig

S copy

Base orig

end

A new message is received

metho d getmsg

send S msg

myqueueenqueuemsg

end

end

The base actor requests a message

restrict put with myqueueempty

metho d put

send Base myqueuedequeue

end

end

In Figure the results of a sample message transaction are shown Note that additional

dispatcher metaactors are required to correctly handle messages that may b e sent by the

client after the server crashed but b efore that crash was detected To keep this example

simple these additional actors were not shown

Conclusions

Because of the costs of learning new programming languages and rewriting old co de the

conversion to new programming paradigms has b een slow This has led some observers to

downplay the imp ortance of research in programming languages At the same time there

is a p erception of a software crises as the cost of maintaining programs has escalated In

fact what exasp erates the problem is the use of old languages and

metho dologies which are insuciently expressive and provide little supp ort for software

maintenance

The acceptance of new programming paradigms is now likely to come more rapidly

There are two reasons for this First the cost of developing co de in newer languages often

outstrips the cost of maintaining old co de For example consider the increasingly deployed

ob jectoriented software technology The technology enables programmers to reduce devel

opment time by providing supp ort for design and to reduce software maintenance costs by

allowing them to incrementally mo dify their co de Second the increasing computational

p ower available and the ever lowering cost of concurrent computers implies that at least

p ortions of the co de need b e rewritten to take advantage of concurrent computers

The need for new programming paradigms is by no means the dominant force in parallel

or distributed computing research For example considerable eort has gone into the

development of parallelizing which attempt to extract parallelism from existing

sequential co de and then automatically determine the mapping to concurrent computer

architectures The parallelizing compilers approach suers from two limitations First

co de based on sequential algorithms cannot b e generally translated to the b est parallel

algorithms Second no general techniques can allow ecient placement and scheduling

strategies for arbitrary algorithms on a concurrent computer

The development of new programming paradigms should allow more complex programs

to b e written with less eort Furthermore it should make the expression of p otential

parallelism simpler However to b e practical new paradigms must not place unrealis

tic restrictions on expressiveness For example although purely functional or stateless

programming has some nice concurrency prop erties shared mutable state is an essential

requirement of distributed computing

Gains in programmer pro ductivity can only b e realized by the greater use of new ab

stractions and mo dularity mechanisms Mo dularity is gained by separating design concerns

co de for dierent purp oses should b e indep endently sp ecied and comp osed Abstraction

allows increased genericity and reuse Furthermore it raises the granularity of programming

by allowing co de to b e expressed in terms of more intuitive structures

We have discussed a numb er of ways in which mo dular construction of multicomp onent

concurrent systems can b e supp orted These include the use of constraints for expressing

co ordination patterns for over distributed ob jects ActorSpace Typ es for abstracting over

resource management strategies for groups of actors and metaprogramming for dep end

ability proto cols Although the resulting mo dularity allows concurrent programming to b e

simplied this is only a small part of the gain More imp ortantly the application indep en

dence provides a basis for constructing software rep ositories for example co de stored in

rep ositories can include sp ecications and implementations of constraints placement and

scheduling p olicies and dep endability proto cols The executable sp ecications may then

b e dynamically linked with dierent application co de without the need for reimplementing

any of them The average application develop er need not understand the details of the

representation rather she needs to know only the relevant prop erties of the abstraction

including prop erties such as the p erformance characteristics of certain access patterns

The concurrent programming language abstractions and mo dularity mechanisms we pro

p ose are certainly not a complete set They do however suggest ways of drastically reducing

software development and maintenance costs scaling up software systems and making it

feasible to use the p ower of parallel and distributed computing We b elieve that the suc

cessful application of these metho ds will further stimulate research in the development of a

new generation of realistic highlevel programming languages

INSET Furhter Reading

Following is a list of further readings

The Actor mo del was originally prop osed by Hewitt for example see and later

develop ed by Agha

C Hewitt Viewing Control Structures as Patterns of Passing Messages Journal of

Articial Intel ligence

G Agha Actors A Model of Concurrent Computation in Distributed Systems MIT

Press

A formal theory of actors including pro of techniques for establishing the equivalence of

actor systems app ears in

G Agha I Mason S Smith and C Talcott Towards a Theory of Actor Computation

In Third International Conference on Concurrency Theory CONCUR pages

SpringerVerlag August LNCS

Separation of representation and description of system and reasoning in terms of meta

ob jects are describ ed in

N Venkatasubramanian and C Talcott A MetaArchitecture for Distributed Resource

Management In Proceedings of the Hawaii International Conference on System Sciences

IEEE Computer So ciety Press January

The use of actors for message driven programming of multicomputers is describ ed in

W Athas and C Seitz Multicomputers MessagePassing Concurrent Computers IEEE

Computer pages August

A numb er of research eorts in ob jectoriented programming are describ ed in

A Yonezawa and M Tokoro editors ObjectOriented Concurrent Programming MIT

Press Cambridge Massachussets

ASTs generalize and abstract over Concurrent Aggregates

A A Chien Concurrent Aggregates Supporting Modularity in Massively Paral lel Pro

grams MIT Press

For actorbased such as Jmachine see

W Dally A VLSI Architecture for Concurrent Data Structures Kluwer Academic Press

S Matsuoka and A Yonezawa Analysis of Inheritance Anomaly in Ob jectOriented

Concurrent Programming Languages In G Agha P Wegner and A Yonezawa editors

Research Directions in ObjectOriented Programming MIT Press to b e published

Acknowledgments

The authors work is supp orted by the Oce of Naval Research ONR contract numb er

NJ by the Digital Equipment Corp oration and by joint supp ort from the

Defense Advanced Research Pro jects Agency and the National Science Foundation NSF

CCR The authors would like to thank contributors to the Op en Systems Lab ora

tory including Christian Callsen Shingo Fukui Chris Houck Shakuntala Miriyala Shangpin

Ren RK Shyamasundar Nalini Venkatasubramanian and Takuo Watanab e contributions

to the lab oratory The research describ ed in here has strongly b eneted from the rst au

thors discussions with and Carolyn Talcott among others

References

G Agha Concurrent Ob jectOriented Programming Communications of the ACM

Septemb er

G Agha and CJ Callsen ActorSpace An Op en Distributed

In Principles and Practice of Paral lel Programming Sigplan Notices To b e

published

G Agha S Frlund R Panwar and D Sturman A Linguistic Framework for Dy

namic Comp osition of Dep endability Proto cols In Dependable Computing for Critical

Applications III IFIP Transactions Elsevier Science Publisher

G Agha C Houck and R Panwar Distributed Execution of Actor Systems In

D Gelernter T Gross A Nicolau and D Padua editors Languages and Compilers

for Paral lel Computing pages SpringerVerlag LNCS

N Carriero and D Gelernter How to Write Parallel Programs A Guide to the

Perplexed ACM Computing Surveys Septemb er

Bjorn N FreemanBenson and Alan Borning Integrating Constraints with an Ob ject

Oriented Language In O Lehrmann Madsen editor Proceedings ECOOP LNCS

pages Utrecht The Netherlands July SpringerVerlag

S Frlund and G Agha A Language Framework for MultiOb ject Co ordination In

Proceedings of ECOOP Springer Verlag To app ear in LNCS

L H Jamieson Characterizing Parallel Algorithms In R J Douglass LH Jamieson

DB Gannon editor The Characteristics of Paral lel Algorithms pages MIT

Press

W Kim and G Agha Compilation of a Highly Parallel ActorBased Language In

U Banerjee D Gelernter A Nicolau and D Padua editors Proceedings of the Work

shop on Languages and Compilers for Paral lel Computing Yale University TR DCS

RR to app ear in LNCS SpringerVerlag

D C Luckham J Vera D Bryan L Augustin and F Belz Partial Orderings of Event

Sets and Their Application to Prototyping Concurrent Timed Systems In Proceedings

of the DARPA software Technology Conference April

B Shriver and P Wegner Eds editors Research Directions in ObjectOriented

Programming MIT Press Cambridge Mass

V Singh V Kumar G Agha and C Tomlinson Scalability of Parallel Sorting on

Mesh Multicomputers International Journal of Paral lel Programming April

Contents

Intro duction

Actors

Communication Abstractions

CallReturn Communication

Patterndirected Communication

Synchronization Constraints

Ob jectOriented Design

Separating Design Concerns

Synchronizers

Mo dular Sp ecication of Resource Management Policies

Customizing Dep endability

Conclusions

Top View