Three significant trends have under- scored the central role of concurrency in computing. First, there is in- creased use of interacting processes by individual users, for example, ap- plication programs running on X windows. Second, workstation net- works have become a cost-effective CONCURRENT OBJECT-ORIENTED

mechanism for resource sharing and distributed problem solving. For ex- ample, loosely coupled problems, such as finding all the factors of large prime numbers, have been solved by utilizing ideal cycles on networks of hundreds of workstations. A loosely coupled problem is one which can be easily partitioned into many smaller subproblems so that interactions between the subproblems is quite limited. Finally, multiprocessor tech-

CCYY”NlCITlCYICCT”EACCY/September 199O/Vol.33, No.9 125 I I I I I I I I I I I

nology has advanced to the point of led to different models of com- to rapid prototyping applications. providing supercomputing power at munication between objects. Object- a fraction of the traditional cost. oriented programming builds on the Patterns oi Concurrent At the same time, software engi- concepts of objects by supporting Problem Solvlng neering considerations such as the patterns of reuse and classification, Three common patterns of parallel- need for data abstraction to promote for example, through the use of in- ism in problems have been found in program modularity underlie the heritance which allows all instances practice (For example, see [8, 131). rapid acceptance: of object-oriented of a particular class to share the same First, pipeline concurrency involves the programming methodology. By sep- method. enumeration of potential solutions arating the specification of what is In the following section, we outline and the concurrent testing of these done (the abstraction) from how it is some common patterns of concur- solutions as they are enumerated. done (the imjhwrltation), the concept rent problem solving. These patterns Second, divide and conquer concur- of objects provides modularity neces- can be easily expressed in terms of rency involves the concurrent elabo- sary for programming in the large. It the rich variety of structures provid- ration of different subproblems and turns out that concurrency is a nat- ed by COOP. In particular, we dis- the joining of (some or all) of their ural consequence of the concept of cuss the actor model as a framework solutions in order to obtain a solution objects. In fact Simula, the first for concurrent systems’ and some to the overall problem. In divide and object-oriented language, simulated concepts which are useful in building conquer concurrency, there is no in- a simple form of concurrency using actor systems. We will then describe teraction between the procedures coroutines on conventional architec- some other models of objects and solving the subproblems. A third pat- tures. Current development of con- their relation to the actor model tern can be characterized as cooperative current object-oriented programming along with novel techniques for sup- probh-solving Cooperative problem- (COOP) is providing a solid software porting reusability and modularity in solving involves a dynamic complex foundation for concurrent comput- concurrent object-oriented program- interconnection network. As each ing on multiprocessors, Future gen- ming. The last section briefly out- object carries out its own computa- eration computing systems are likely lines some major on-going projects in tional process, it may communicate to be based on the foundations being COOP. with other objects, for example, to developed by this emerging software It is important to note that the ac- share the intermediate results it has technology. tor languages give special emphasis computed. An example of this kind The goal of this article is to discuss to developing flexible program struc- of a system is a simulation where the the foundations and methodology of tures which simplify reasoning about physical objects are represented by COOP. Concurrency)refers to the poten- programs. By reasoning we do not logical (computational) objects. tially parallel exe:cution of parts of a narrowly restrict ourselves to the Consider some canonical exam- computation. In a concurrent com- problem of program verification- ples illustrating different patterns of putation, the components of a pro- an important program of research parallelism. A simple example of gram may be executed sequentially, whose direct practical utility has yet pipeline concurrency is the prime or they may be executed in parallel. to be established. Rather our interest sieve. To generate all the prime Concurrency provides us with the is in the ability to understand the numbers, one could generate all flexibility to interleave the execution properties of software because of numbers and remove multiples of of components of a program on a clarity in the structure of the code. 2357I , I I-**, up to the largest prime single processor, or to distribute it Such an understanding may be computed thus far. As soon as a among several processors. Concur- gained by reasoning either infor- number is identified as prime, it is rency abstracts away some of the mally or formally about programs. added to the sieve and numbers are details in an execution, allowing us to The ease with which we can carry out also eliminated by testing for divisi- concentrate on conceptual issues such reasoning is aided by two fac- bility by this prime (see Figure 1). without having to be concerned with tors: by modularity in code which is The earlier stages of this particular a particular order of execution which the result of the ability to separate pipeline are a bottleneck because may result from the quirks of a given design concerns, and by the ability to many more numbers are divisible by system. abstract program structures which smaller primes. The linear pipeline Objects can be defined as entities occur repeatedly. In particular, be- can be improved by changing it to a which encapsula.te data and opera- cause of their flexible structure, actor tree with the numbers sent to dif- tions into a single computational languages are particularly well-suited ferent identically behaving objects, unit. Object models differ in how the each testing for divisibility by a given internal behavior of objects is IThe term Actor was introduced by Carl Hewitt at MIT in the early 1970s to describe the concept of (low) prime, and then merging the specified. Further, models of concur- reasoning agents. It has been refkd over the years results. This can be achieved by us- rent computation based on objects into a model of concurrency. It should be noted that ing demand-driven evaluation which our use of the term bears no relation to the languqe must specify how the objects interact, Actor-the latter being a commercial product in- dynamically creates context objects and different design concerns have troduced in the late 1980s. to filter the numbers for divisibility of

126 September 19901Vo1.33,No.9ICOYYUNICITlONSOFT"EICY I I I I I I I I I I I I I I I 111111111~ OBlBCT-OBlBN7BD DESIDN

the primes below it. Specifically, each number can create its own copies of the elements of the sieve as it goes along. This scheme provides tree pipelining for testing divisibility. I It should be observed that because Sieve ) 2 5 7 I a large number of unnecessary tests 3 are performed, the technique of gen- erating all numbers and then filter- ing is quite inefficient in the first place. An improved version would avoid generating multiples of the low primes, 2,3,5,.. (up to some prime). Athas describes the behavior of an algorithm designed precisely to do this [7]. Divide-and-conquer concurrency algorithms can often be expressed as functions. Arguments to a function are evaluated concurrently and their values collected to determine the foal result. Consider the problem of de- termining the product of a list of numbers [4]. We can represent the list as a tree as in Figure 2. The prob- lem can be recursively subdivided into the problem of multiplying two sublists, each of which is concur- (define tree-product rently evaluated, and their results are (lambda [tree] multiplied (see Figure 3). The prod- (if (number?tree) tree FlCURE q. A S/mp/e /W/me Sieve. (’ (tree-product (left-tree tree)) Numbers generated and successively tested (* (tree-product (right-tree tree)))))) for dlvlsiblllty by a linear plpellne of primes. The circled numbers represent numbers which are being generated. The boxed ItUmbWS are ellmlnatlng numbers which are not prime. New primes are added at the end of the sieve. 1 2 n n+l . . . FIGURE 2. A list of numbers to be mail queue I I I multiplied. The numbers are represented as leavesof a tree.

FBGURE 3. The code for multlplylng a list of numbers represented as a tree. In the X” -_ abovecode, a tree Is passed to tree-product dj -* Xn+l ’ \ which tests to see If the tree Is a number creates tasks,/ (I.e. a singleton). If so It returns the tree, I \ t otherwise It subdlvldes the problem Into two \ L specifies replacement reCUrSlIfe Calls. left-tree and right-tree \ are functions which pick off the left and task \,creates actors right branches of the tree. Note that the cl arguments to * may be evaluated concurrently. k 1 mail queue . . . FicuRE 4. A pictorial representation I of the behavior of an actor. when the actor processesthe nn communlcatlon, It deter- mines the behavior which will be used to process the n +I” communlcatlon. The mall address of the actor remains Invariant. The actor may alS0 Send communlcatlons to specific target actors and create new actors.

COMY”WlC*TIOWIOFT”E~.C~/Septe”,bcr 199o/vo1.33, No.9 I I I I I I I I I I

uct is then returned. The tree prod- ming (For example, see [l]): it ex- drawal request. In response, as soon uct program in Figure 3 looks very tends the dynamic resource creation as it has computed the new balance much like code in Lisp or Scheme ex- capability provided by function in the account, it is free to process the cept that the evaluation strategy is abstractions to concurrent computa- next request-even if other actions maximally concurrent. tion. The become primitive gives ac- implied by the withdrawal request In cooperative problem solving tors a history-sensitive behavior are still being carried out. To put it concurrency, intermediate results are necessary for shared mutable data another way, the concurrent specifi- stored in objects and shared by pass- objects. This is in contrast to a purely cation of replacement behaviors ing messages between objects. Simu- functional programming model and guarantees noninterference of state lation programs, where a logical generalizes the Lisp/Scheme/ML se- changes with potentially numerous object represents a physical object, is quential style sharing to concurrent threads running through an actor one application of this kind of con- computation. The send to primitive under a multiple-readers, single- currency. For example, the dynamic is the asynchronous analog of func- writer constraint. evolution of the paths of a number of tion application. It is the basic com- Concurrent computations can be bodies under the influence of each munication primitive causing a visualized in terms of event diagrams others’ gravitational fields can be message to be put in an actor’s mail- (see Figure 5). These diagrams were modeled as systems of cooperating box (message queue). It should be developed to model the behavior of objects. Another example of coopera- noted that each actor has a unique actor systems. Each vertical line, tive problem solving is blackboard mail address determined at the time called a &line, represents all the com- systems which allow collaboration of its creation. This address is used to munications received by a given ac- between agents through a shared specify the recipient (target) of a tor. The receipt of a communication work space. In an object-based sys- message. represents one kind of event. Another tem, the blackboard and the agents In the actor model, state change is kind of event is the creation of a new may be represented as systems of specified using replacement behav- actor represented by an open arc on objects. iors. Each time an actor processes a the top of a lifeline. Connections be- communication, it also computes its tween lifelines represent causal con- The ACtOr MOdeI behavior in response to the next nections between events. Pending A common semantic approach to communication it may process. The events, representing communica- modeling objects is to view the replacement behavior for a purely tions which have been sent but not behavior of objects as functions of in- functional actor is identical to the received, may be represented by ac- coming communications. This is the original behavior. In other cases, the tivation lines whose arrows note the approach taken in the actor model behavior may change. The change in message and the target. [21]. Actors are self-contained, in- the behavior may represent a simple teractive, independent components change of state variables, such as Control Structures of a computing system that com- change in the balance of an account, Concurrent control structures repre- municate by asynchronous message or it may represent changes in the sent particular patterns of message passing. The ba.sic actor primitives operations (methods) which are car- passing. Consider the classic exam- are (see Figure 4): ried out in response to messages. ple of a recursive control structure The ability to specify a replace- which illustrates the use of customers create: creating an actor from a ment behavior retains an important in implementing continuations. The behavior description and a set of advantage over conventional assign- example is adapted from [14] which parameters, possibly including ex- ment statements: assignments to a provided the original insight ex- isting actors; variable fix the level of granularity at ploited here. In a sequential lan- which one must analyze a system. By guage, a recursive formula is send to : sending a message to an ac- contrast, the replacement mecha- implemented using a stack of activa- tor; and nism allows one to aggregate changes tions. There is no mechanism in the and avoid unnecessary control flow sequential structure for distributing become: an actor replacing its own dependencies within computational the work of computing a factorial or behavior by a new behavior. units which are defined by reception&s concurrently processing more than [2]. Replacement is a serialization one request. These primitives form a simple mechanism which supports a trivial Our implementation of the factor- but powerful set upon which to build pipelining of the replacement ac- ial actor relies on creating a customer a wide range of higher-level abstrac- tions: the aggregation of changes al- which waits for the appropriate com- tions and concurrent programming lows an easy determination of when munication, in this case from the fac- paradigms [3]. The actor creation we have finished computing the state torial actor itself. The factorial actor primitive is to concurrent program- of an actor and are ready to take the is free to concurrently process the ming what the definition of a lambda next action. For example, suppose a next communication. We assume abstraction is to sequential program- bank account actor accepts a with- that a communication to a factorial

128 September1990/Vo1.33, No.9/COYYUNICATIOWSOFT"E~CW I I I I I I I I I I I I I I I I llllllsI- OUECT-OB1ENTED DDSIDN

includes a mail address to which the most of the processing required by portion to the magnitude of the com- value of the factorial is to be sent. In the request to a large number of ac- putation required. response to a communication with a tors, each of whom is dynamically There is nothing inherently con- non-zero integer n, the actor with the created. Furthermore, the number of current in the recursive algorithm to above behavior will do the following: such actors created is in direct pro- evaluate a factorial. Using the algo- l Create an actor whose behavior will be to multiply n with an integer it receives and send the reply to the mail address to which the factorial of n was to be sent. l Send itself the “request” to evaluate the factorial of n - 1 and send the value to the customer it created. One can intuitively see why the factorial actor behaves correctly, and can use induction to prove that it :kll sends communications actors does so. Provided the customer is sent the correct value of the factorial of I\ n-l, the customer will correctly evaluate the factorial of n. Moreover, the evaluation of one factorial does not have to be completed before the next request is processed; (i.e., the factorial actor can be a shared re- source concurrently evaluating sev- eral requests). The behavior of the factorial actor in response to a single initial request is shown in Figure 6. This particular function is not very complicated, with the consequence that the behavior of the customer is also quite simple. In general, the factorial behavior of the customer can be ar- bitrarily complex. The actor origin- ally receiving the request delegates

FIGURE 5. Actor event diagrams pro- [3,cl vide an abstract view of computation in a (m, W3d concurrent system. An actor is identified f with a vertical line which represents the linear arrival order of communications sent to that actor. in the above diagram, I, and k, represent two communications sent to > the actor 0. The communication k, arrives [2, ml before L,. in response to processing a com- munication, new actors may be created (m:*C% ml) (dashed lines) and different actors may be / sent communications (solid lines) which will > arrive at their target after an arbitrary but U,m’l finite delay. The bon represents a COmmUni- cation which has been sent but not yet (m:$(l, m’ processed. 6

elcu~~ 6. A recursive factor/al coor- putation. The computation is in response to a request to evaluate the factorial of 3. > Eath actor is denoted by a mail address and [O,m” 1 a behavior. The W’S represent the behavior of the dvnamicaiiv created customers. For euampie, the behavior vt3,cl sends I3*kl to the mail address c in response to the com- El munication k.

129 I II I I I I I I I I I

rithm in Figure 6, computation of a instance of Bat&Account. In this case for example: single factorial would not be any there is just a single state variable, (send-tomy-wifemy-account) faster if it were clone using an actor balance. The methods or communi- language as opposed to a sequential cation handlers associated with a would allow the actor whose mail ad- language. All we have done is repre- BankAccount follow. A method is dress is bound to my-wife to access sent the stack for recursion as a chain specified by listing a keyword repre- theactor my-account. of customers. However, given a net- senting the operation to be executed work of processors, an actor-based by the method, followed by a table Joln Contlnuatlons language could process a large num- that represents the content of the re- Divide and conquer concurrency ber of requests much faster by simply quest message (in this case a with- can often be naturally expressed by distributing the actors it creates draw-from message must specify an using a functional form which eval- among these processors. The factor- amount), and a body that defines uates its arguments concurrently. ial actor itself would not be as much how such messages are to be process- Implementation of such forms re- of a bott1enec.k for such com- ed. When used within the body of a quires the specification of a join putations. mutable generator, the form become continuation which synchronizes the Patterns of communications re- is used to specify the replacement evaluation of the different argu- presented in recursion, iteration, behavior of the instance of an actor ments. For example, the tree- divide and conquer, etc., can be created using the generator-the product program given in Figure 3 abstracted into linguistic forms generator itself does not change. can be expressed in terms of actor which automatically coordinate in- A new BankAccount may be primitives as shown in Figure 8. dependent computations. An impor- generated with an initial balance of The form (bar foo) represents an tant service provided by high-level 1000 by using the create operation as asynchronous message foo sent to actor languages such as Acore [20] is follows: bar. The form [xl x2. . . ] the generation and coordination of represents a data constructor whose elements xl, x2, . . . are concur- customers which are actors provided in (definemy-account a request message. A customer can (create rently evaluated. It should be noted that the two tree products in the do be sent a reply message when a request BankAccountlOOO)) is completed, or a complaint message body are also concurrently evaluated and their values are sent to new- if it is not possible to successfully We can make the communication cust, where new-cust is a history- complete the reqluest. handles (the keywords determining sensitive actor whose role is to store which method is to be performed) the first value it receives and multi- History-Sensitive visible by declaring them to be opera- ply the stored number with the sec- Behavior tions. The behavior of an operation Often it is necessary for an actor to ond number it receives in order to is to send to its first argument a mes- change its local state and to respond produce the final response that is sent sage containing itself and the rest of to more than one kind of message. to the customer which was specified the arguments it received. This For example, a bank account changes at the time of its first invocation. The allows object-oriented message- its behavior in response to processing form rlambda defines actors rather passing style and functional styles to an incoming deposit or withdrawal than actor behaviors (which are de- be freely mixed in an actor language. message. In order to define these fined by mutable). In the body of an The two styles serve as duals of each kinds of actors, a form called mutable rlambda, become specifies the re- other. is provided in the Rosette actor placement behavior of the actor itself. Requests may be issued to the new language developed at MCC by Thus rlambda is the actor analogue account by using the methods that Tomlinson and others in collabora- of lambda; it captures the history- are declared as operations: tion with the author. The mutable sensitive behavior of an actor using form is used to defme a generator ac- (deposit-tomy-account lOO) the form become. Note that in join- tor which creates actors using a * 'deposited100 cant, vl refers to the first message behavioral template. received by the actor-which could (withdraw-frommy-account78) correspond to either the product of The behavior of a simple bank ac- * 'withdrew 78 count may be defined using a mut- the left subtree or the right subtree. able expression (see Figure 7). The Subexpressions are evaluated con- The behavior of tree-product is generator actor that results from the currently. Thus, the computation of shown in terms of an event diagram mutable expression is bound to the a replacement behavior, done by the in Figure 9. When the tree-product symbol Bar&Account. The generator become command, is concurrent actor receives a list represented as a allows creation of instances via a with the computation of a response. tree containing ltree as its left subtree create expression. Following the key- The capability to access the account and rtree as its right subtree, it word mutable is a sequence of iden- may be passed to another actor, creates a customer, called a join con- tifiers for the state variables of an dynamically reconfiguring a system; tinuation, which awaits the computa-

130 I I I I I I I I I I I I I I I 1111111m1- OWDCT-O8XENTEDDEEIQ~

tion of the products of each of the two computations, and to do dynamic er- of evaluating the other subtree [4]. subtrees. The join continuation then ror handling. For example, if the Furthermore, we may want to flag er- proceeds to multiply the two num- number 0 is encountered, the join ror conditions such as data excep- bers and send the result to the orig- continuation can immediately return tions. If we require, for example, that inal requester. Because multiplication a O-without waiting for the results all the numbers in the tree be positive is commutative, we need not be con- cerned about matching the responses to the order of the parameters. If we were dealing with an operator which (define BankAccount was not commutative, we would need (mutable palance] to tag the message corresponding to [withdraw-from [amount] each argument and this tag would be (become BankAccount (-balance amount)) returned with the response from the (return ‘withdrew amount)] corresponding subcomputation. The [deposit-to [amount] replacement behavior of the join con- (become Bar&Account (+ balance amount)) tinuation would then depend on the (return ‘deposited amount)] order in which the results of the eval- [balance-query uation of arguments were received. (return ‘balance-isbalance)])) Because the semantics of concur- rency requires that the evaluation of the two invocations of tree-product be indeterminate, the behavior of (define tree-product join-cant cannot be expressed func- (rlambda [tree customer] tionally-despite the fact that the (if (number? tree) behavior of tree-product itself is (customer tree) functional. (let [newcust (join-cant customer)] In Figure 9, we provide the be- (do (tree-product (left-tree tree) new-cust) havior of the history-sensitive join (do (tree-product (right-tree tree) new-cust)))))) continuation explicitly. The advan- tage of explicit join continuations is (define join-cant [custromer] that they provide considerable flexi- (rlambda [vl] bility-they can be used to control (become (rlambda [v2] the evaluation order, to do partial (customer (’ vl v2))))))

FIGURE 7. A 8allk aCCt?Ullt. Tile behavior of a bank account Is described In a Tree-product oblectorlented terms using communication handlers (methods). The behavior Is history sensitive: each time a balance query is [Tsl made, a different response may be given by the bank account.

FIGURE 8. The tree-product ill tWlRS of primitive actors. The behavior of the dynamically creation loln continuation IS F!q@c& expllcltly shown. we use the form rlambda to lndlcate the definition of an actor which may have a replacement behavlor. The form d0 Is used to simply evaluate the two c+v1*v2 arguments concurrently.

FIGURE 9. rreeproducteventd The skeleton of actions taken bv the tree pro- (define join-cant [customer] duct In response to a message contalnlng the tree T= [ltree rtree] and the customer C. @lambda[vl] (case FIGURE 10. A joln contlnuatlon which ((zero? vl) (customer 0)) reacts appropriately to partial tompu- ((error? vl) (error-handler vl)) tatlons as they are completed. Note that (otherwise (become (rlambda [v2] error? Is an actor which returns a true or (customer (* vl ~2)))))))) false value depending on whether the argument It receives Is “acceptable.”

CCYYUNICITICIICCFTHEACMISeptember 1990/W%, No.9 I II I I I I I I I I

we may want to terminate the com- modeling such objects is equivalent It should be observed that the putation once we encounter a nega- to representing a fair merge. guarantee of message delivery in an tive number, In this case, we can Fair merges are a fundamental con- asynchronous communication model invoke an error handler to clean up cept in modeling concurrent systems; itself cannot be realized with cer- the data or take other appropriate ac- they allow one to abstract over differ- tainty-it can only be provided with tion. Figure 10 specifies the code for ent possible assumptions about the some level of confidence in a well- a join continuation with such error- relative speeds of processors, the engineered system. For example, in handling capability. scheduling of processes on proces- principle, an actor could produce In general, error handlers distinct sors, and the relative speed of the com- enough communications to exceed from the regular continuation struc- munication links. A model which the buffering capacity of the com- tures can be passed along. These made specific assumptions about munication network. This problem is error handlers provide non-nonfunc- such implementation-dependent fac- similar to that of implementing tional jumps which can appropriately tors may not be sufficiently abstract recursion using a stack: a recursion clean-up erroneous conditions that to be useful in reasoning about dif- may be prematurely terminated by may arise in the course of a computa- ferent possible implementations of a limitations of stack size. In a sequen- tion. Because this separates code with concurrent program (see, however, tial system, only bounded stacks are distinct purposes, programming with the discussion about coordinated ac- physically realizable but the bounds non-functional jumps supports tion in section entitled “Coordina- would vary with every specific im- greater modularity; in particular, it tion”). Using a semantics of fair plementation. gives us the ability to independently merge, one can reason about the specify and reason about the normal eventual behavior of a concurrent BUilUing ACtOr System5 and abnormal behavior of a program; reasoning about eventual Computation in a typical actor program. properties of a concurrent system is system is performed by the decisions analogous to reasoning about fixed and communications of many small NondetermlnJsm points in a recursion in sequential modules. Actor primitives provide a Although actors take a functional programming. very low-level description of concur- view of an object’s internal behavior The behavior of fair merge cannot rent systems-much like an assem- at any given point in time, actors can be represented by the standard sub- bly language. Higher-level constructs represent shared history-sensitive ob- stitution semantics grounded in the are necessary both for raising the jects. Consider the canonical exam- lambda calculus. A substitution sem- granularity of description and for en- ple of a bank account. The behavior antics requires that an expression capsulating faults. Such constructs of a bank account changes over time have the same value regardless of the delimit computational boundaries by as a function of the balance in the ac- context in which it is invoked-a con- aggregating a collection of events into count. By contrast, the purely func- dition violated by the shared bank organizational units characterized by tional programming approach, while account whose behavior changes as patterns of interactions. Simple ex- mathematically elegant, is insuffi- a function of the balance in the amples of patterns of interactions in- cient to represen.t structures in the account. Because the ability to clude control structures, synchronous real world of distributed computing: implement shared resources is so communication, and transactions. what makes a sha.red account mean- fundamental to a concurrent system, The following discussion elaborates ingful is that state change is visible to the actor approach is to integrate the on these examples and their use in many users. The values of functions ability to create modifiable, shared implementing actor systems. are, however, returned only to the structures into the programming caller or invoker of that function. model. Furthermore, the connec- Coorcllnatlon The need to implement syn- tions between objects may be dynam- A simple example of coordination is chronization between concurrent ically made or broken. Fair merges in that required in function calls which computations requires the ability to the actor model are implicit--they return a value to the (usually im- define an indeterminate, complete are captured by the guarantee of plicit) join continuation which is the merge of messages sent to an actor. message delivery which states that return address for the call. The crea- We call such a merge afair merge. A any message sent to an actor must tion of join continuations increases fair merge is complete because it eventually be received-i.e., after a the available concurrency in a func- merges message:; from every sender finite but arbitrarily long delay. The tion call. Thus, function calls are an and may not ignore any sender inde- guarantee of delivery does not specify example of a very simple two-party finitely and it is indeterminate be- the order in which messages may be interaction involving synchroniza- cause no particular order is specified received by an actor. In particular, tion between a message and an actor. for messages sent to the same object the sequence of messages from one A more complex example of a two- by different objects. Because shared actor to another need not be preserv- party interaction is synchronous history-sensitive objects interleave ed: this allows for the possibility of communication between two actors; messages sent by different objects, adaptive routing. such communication represents an

132 I I I I I I I I I I I I I I I I llllllm- OBlECT-OBIENTED DEBIDN

atomic interaction between two Multicomputers actors. Because there is no instan- everal kinds of concurrent computer architectures have been pro- taneous action at a distance in a posed. These architectures may be broadly divided into synchro- distributed system, synchronous Snous computers, shared memory computers, and multicomputers communication can be implemented (also called message-passing concurrent computers). Synchronous com- only by strongly constraining the puters, such as the Connection Machine, are suitable for data-parallel implementation. In particular, its computation. These computers are quite special-purpose and rather restrictive in their model of concurrency. We are primarily interested implementation requires a known in general-purpose computing-for this purpose, computers which sup- time bound on the potential com- port control parallelism are of greater interest. munication delay between the two Sharedmemory computers have multiple processors and provide a actors which can communicate global shared memory. For efficiency reasons, each processor also has synchronously. a local cache, which in turn creates the problem of maintaining cache The synchronous communication coherence. The shared memory computers that have been built typically model is critical to the problem of consist of 16 to 32 processors. Because large numbers of processors coordination in distributed systems. create increased contention for access to the global memory, this kind Specifically, it allows not only infor- of architecture is not scalable [lo]. mation to be shared but the meta- M&computers use a large number of small programmable computers (processors with their own memory) which are connected by a meesage- knowledge that the information has passing network. Multicomputers have evolved out of work done by been shared: if an actor X communi- Charles Seitz and his group at Caltech [9] and have been used to sup- cates information i synchronously port actor languages [I]. The network in multicomputers supports the with an actor Y, X knows that Y actor mail abstraction; memory is distributed and information is localized knows i and Y knows that X knows on each computer. Load balancing and maintaining locality of com- that Y knows i, etc., ad infinitum. munication simplified by using small objects which can be created and This state of knowledge is called destroyed dynamically, qualities which are characteristic of actor common knowledge; common knowl- systems. edge is essential for coordination Configurations of multicomputers with only 64 computers exhibit per- between agents who need to act in formance comparable to conventional supercomputers. It should be noted that machines based on the transputer are also multicomputers, concert (see, for example, [21]). In the but these computers use a model of computation based on com- above example, if X will not act un- municating sequential processes rather than the actor model. til X knows that Y will also act, then Multicomputers may be divided into two classes: medium-grained it can be shown inductively that nei- multicomputers and fine-grained multicomputers. ‘Ikro generations of ther actor will act unless they have medium-grained muiticomputers have been built. A typical first- common knowledge. The notion of generation machine (also called the cube or the hypercube because of common knowledge can be general- its communication network topology) consisted of 64 nodes and delivered ized to collections of an arbitrary 64 MIPS. Its communication latency was in the order of milliseconds. The number of agents. Common knowl- typical second-generation medium-grained multicomputer has 256 edge may be achieved through mech- nodes, can carry out about 2.5K MIPS and has a message latency in the order of tens of microseconds. The development of these machines anisms representing asynchronous continues. Third-generation machines are expected to be built over the communication which is guaranteed next five years and increase the overall computational power by two to be delivered within a specified orders of magnitude and reduce message latency to fractions of a bounded time interval. Note, how- microsecond [9]. ever, that there is no unique global However, the frontiers of multicomputer research are occupied by clock in a distributed system; thus work on fine-grained multicomputers. These computers realize an time delays must be expressed in idealized actor machine with fine-grain concurrent structure inherent relativistic terms (they may be in the functional form of an actor’s behavior; in turn the Actor model is bounded, for example, using a well-suited to programming them (for example, see (lo]). Two projects building experimental fine-grained multicomputers are the J-Machine distance metric). The problem of co- project by William Dally’s group at M.I.T. [ll] and the Mosaic project ordination raises a number of in- by Charles Seitz’s group at Caltech. The experimental prototype of the teresting linguistic and semantic Mosaic system will consist of 16,384 nodes and is expected to deliver issues and is an active area of 200,000 MIPS [9]. research in the author’s group. Obviously, Actor languages can be implemented on a number of com- Transactions are another example puter architectures such as sequential processors, shared memory of n-party interactions. In the context machines, and SIMD architectures. However, multicomputers are par- of actor systems, communications ticularly interesting because of their scalability characteristics. It should can be classified as requests or re- also be observed that actors can be directly supported on mu&computers sponses. Each request carries a cus- whereas implementing other programming paradigms on such com- puters may require their implementation in terms of some simple variant tomer to which a (unique) response of the actor execution model (For example, see [ 121). is to be sent. A transaction is defined

CODIMUWICITION5OFT~LACM/Scptembcr 199OWo1.33, No.9 133 I I I I I I I I I I

as all events intervening in the com- ient) of the communication. ed in a computation, for example bined causal and arrival ordering A problem with the use of event because of a shared subcomputation, between a given request and a re- diagrams is that they contain every the request reply structure still serves sponse to it. An event (processing of event and, in any realistic concurrent as a granularity control mechanism. a message) preced.es another event in system, there are simply too many Using the observatory, one can end the causal order ifthe second event is events. Using event diagrams to try up potentially weaving back and the result of processing the message to pinpoint an error can be harder forth through connected computa- corresponding to the first event. The than looking for a needle in a hay- tions looking for the source of error. arrival order represents the order in stack. Mechanisms are necessary to In order to address this kind of dif- which messages are processed by a structure events by delineating com- ficulty, Yonezawa’s group at the Uni- given actor. putational boundaries. One such versity of Tokyo has developed an Transactions delineate computa- mechanism is based on the concept alternative method for monitoring tional boundaries for error recovery of transactions. For our purposes, a programs based on collecting objects or for the alloca.tion of resources. transaction is delimited by events into groups (see [24]). Transactions have been used to sup- between a request message and a re- port debuggers in concurrent com- sponse message, where “between” is Resource ManaRement putation (see below) and have been in terms of a partial order defined by In a concurrent language, a problem proposed as a mechanism for deter- the transitive closure of causal and is often dynamically partitioned in- mining resource allocation policies at arrival orders on events. In addition, to subproblems. An application a high level (see the section entitled the transaction must obey the follow- developer needs to be able to specify “Resource Manargement”). ing properties: the allocation of resources to each dynamically created subproblem. Vlsuallzln~ Actor Programs 1. if a request generates subrequests, Because thousands of subcomputa- There are two important difficulties the corresponding subtransac- tions may be created, such allocation in monitoring and debugging con- tions must also be within the decisions need to be specified at a current programs. First, concurrency transaction high level of abstraction. Sponsors implies that pro,gram execution is 2. no subtransaction is shared with are actors which connect to the nondeterministic. Thus any given other transactions, (i.e., two in- underlying resource management execution trace of a program is not dependent transactions include system and are used to drive or throt- likely to be repeated. Second, in any the same subtransaction). tle a computational path. reasonable-sized aconcurrent system, 3. the first two conditions recursively For example, in a graph search the large number of objects and in- hold for the subtransactions. problem, examining a node may sug- teractions between the objects results gest examining a number of other in enormous data consisting of the Transactions can be used to pin- neighboring nodes. This process may large set of events and their relations. point errors in much the same way a be termed node expansion. All nodes This is further complicated by the microscope is used to pinpoint areas which are candidates for expansion fact that since there is no unique or by recursively focusing on smaller may not be equally promising can- true global state in a distributed regions which contain the potential didates; thus a sponsor is needed to system, the observations of on-going point of interest (see Figure 11). In specify how much resource to activity in a system themselves suffer case of a transaction which does not allocate to a node expansion. Fur- from nondeterminism due to the meet its specification, one can look at thermore, an allocation decision may observer’s frame of reference. its subtransactions to determine need to be dynamically modified as One approach to monitoring com- which of these subtransactions may a computation proceeds. In order to putations is to retroactively recon- be causing the observed error. In determine how much resource to give struct relations between events which standard usage, the term transaction to a node expansion in a graph search have already occurred; this can be ac- also implies a requirement of atom- algorithm, a sponsor may use a given complished by constructing event icity, (i.e., either all events that are sponsorship algorithm which diagrams of the sort described in part of a transaction occur or none of measures the goodness of a target Figure 5. Each actor records the them occur). While the atomicity re- expansion. The sponsor may also be communications. it has received in quirement is useful for delimiting a function of current resources which the order it receives them and the ac- boundaries for error recovery, it is are available for the computation. tions it takes in response to those often too strong for a general- A decision about whether to per- communications -namely, the com- purpose programming language. form a subcomputation is thus speci- munications it sends out and the ac- A debugging system based on this fied independently of how the tors it creates. The sending and mechanism has been implemented computation itself is to be carried receiving events are linked by look- by Manning at MIT. The system is out. Furthermore, a sponsorship ing at the recordings in the actor called the observatory [19]. When the algorithm does not need to specify which was the target (specified recip- transactional structure is not preserv- details about how to utilize physical

134 September 199WVol.33, No.9/COYYUNICITIONIOFT”EICN I I I I I I I I 1 1 I I I I I 111111111- OUECTOBlEN7ED DEBIQW

resources, such as memory or pro- cessing power, and does not need to make assumptions about their asso- subtransaction ciated costs. It simply provides very I I high-level control over the relative rates or extent of unfolding of par- Request Response 1 ticular computational paths. 1_ In many computations, it is natural for subcomputations to be shared between a number of in- dependent computations. A number of messages may be merged before I subtransaction being processed; for example, a number of requests to a money market account may be merged into a single subcomputation involving FIGURE ‘I 9. A TransaCtIOnal StrUCtUre.EaLh request generates a unique response. the trading of a set of stocks. In this case, explicit policies must be eled by actors, much as functions can complicates programs by discourag- developed to merge resources and, describe the semantics of procedures ing the use of functional compo- conversely, to assess computation in sequential programming. nents when such components are costs. A traditional sequential process adequate. model allows arbitrary control struc- An alternative approach to pro- Other MOdeIS 09 ObJeCtS tures to be specified within the body viding efficient execution on concur- A number of COOP language models of a given object. The traditional rent architectures is to throttle the unify a declarative view of objects as model also encourages sequencing of concurrency in an inherently concur- abstract data types with a procedural actions which is potentially un- rent language, using translators view of objects as sequential pro- necessary from the point of view of which are optimized for a particular cesses. In this model, each object is a understanding the concurrency in- architecture. This is an area of active sequential process which responds to herent in the logic of a program. An research in actors (as well as in the messages sent to that object. Every advantage of a process model is that concurrent implementation of de- object may execute its actions con- it allows the explicit specification of clarative languages such as func- currently. The number of objects state change using a familiar one step tional programming languages and which have a pending message to at a time assignment to variables; the the so-called concurrent logic process (i.e., are potentially active) at variables are, of course, encapsulated languages). a given time during the execution of within a given object. Another ad- a program is called its concurrency vantage is that a programmer can Inherent Concurrency index at that time. In particular, the optimize the size of the sequential There are some basic language concurrency index is limited by the processes to match the optimal size of design decisions involved in pro- total number of objects in a system at processes on a given concurrent ar- viding a notation to specify the a given time, although new objects chitecture. Thus, the process size can behavior of objects; these decisions may be created dynamically. The be determined by a programmer as affect what kind of concurrency can concurrency index may be increased a function of architectural charac- be extracted from the behavioral by invoking objects asynchronously teristics such as the costs associated description of an object. In par- -thus activating other objects. with process creation, context switch- ticular, two styles of expression Languages which use a process ing and communication. evaluation can be identified: model of objects include ABCL [24], On the other hand, the sequential POOL [5], Concurrent Smalltalk process model of objects has at least Call/Return Style. Subexpressions (see [25]) and BETA (see [22]). Some three disadvantages. First, sequential in the code are evaluated (possibly other languages realize a simple processes which are optimal on one concurrently) and their values are variant of the Actor model: the body architecture may not be so on substituted before proceeding to the of an object is executed sequentially another with different characteristics. enclosing expression. and assignments may be used within Second, because not all state change Customer-Passing Style. Subex- the body but messages are buffered is due to the logic of an algorithm, it pression’s evaluations and the join and may not be received during “in- becomes harder to reason about the continuations’ creation are initiated termediate” states of an object. Se- parallelism in a particular algorithm. concurrently. The object is then free mantically, the behavior of all COOP Finally, because assignments of va.l- to accept the next message. A join languages, whether they use a pro- ues to variables are frequently used continuation takes the results of cess model or otherwise, can be mod- in the sequential process model, it subexpression evaluations and car-

CC*IY”WICITICII~CFT”LACLI/September 199O/Vol.33, No.9 I 1 I I I I I I I I

ries out the rest of the computation (call it second) to that part of the list. have a pending message. Further- specified by the original computa- If the length of the list is not known, more, message delivery is assumed to tional thread provided in the object’s it would take an extra n step to deter- take one time step. These time steps behavior. mine it. are called sweeps. Notice that a dif- The customer,-passing style sup- After a list is split, the two sublists ficulty with this algorithm is that it ported by actors is the concurrent can be concurrently sorted using requires roughly n processors to sort generalization of continuation- mergesort, and the results merged by a list of n numbers. However, most of passing style supported in sequential successively comparing an element these processors would be idle much languages such as Scheme? In case of from each of the two lists and picking of the time (as far the execution of the sequential systems, the object must the smaller one. The next element mergesort algorithm is concerned). have completed processing a com- from the list to which the lesser ele- In practice, the processing corre- munication before it can process ment belongs is then used in the next sponding to a sweep will be delayed another communication. By con- comparison. Thus, given two sorted until all the processing in the trast, in concurrent systems it is lists, merge produces a sorted list previous sweep can be completed. In possible to process the next com- containing elements in both lists. It other words, the concurrency index munication as soon as the replace- should be noted that this merge pro- curve plotted as a function of steps ment behavior for an object is cedure has nothing to do with the needs to be truncated at the max- known. concept of merge in concurrency imum number of processors avail- Note that the ability to distribute which represents an interleaving of able. Thus, executing the algorithm work in systems using call/return all incoming messages discussed on a real parallel computer will take style, those using customer-passing earlier. at least a time factor which equalizes style, and those that do or do not se- The mergesort algorithm can be the areas under the two concurrency quence actions vvithin objects, may expressed as in Figure 12. index curves (the truncated curve be identical. For example, the lan- The form let* represents multiple and the curve assuming a sufficiently guage Cantor developed at Caltech, (possibly recursive) let bindings and large number of processors). uses sequential execution of code in first-half and second-half return the The total time efficiency of the body of an object. Cantor has the respective halves of the list and their mergesort in the presence of a limited full power of actor languages; it sup- lengths. As recursive calls are made, number of processors can be im- ports dynamic creation of objects, the list is split until we have proved by the following observation: asynchronous message passing be- singletons. Each split requires half Because the beginning element and tween objects, and atomic replace- the number of operations of the length of the first half of the list are ment behaviors. In case of primitive previous. As in a sequential merge- known, the first half of the list is actor actions, typically asynchronous sort, the total number of operations determined even as the first element message sends, sequencing actions is O(n log n); however, the concur- of the second half is being com- within an object causes minimal rency index doubles each time a split puted.’ Thus, one can start sorting delay; the time required for these ac- is made. Thus the splits can poten- the first half of the list concurrently tions is fairly small and the resulting tially be executed in O(n) time- with computing the first element of activity is concurrent. However, given a sufficient number of pro- the second half of the list. The when arbitrarily complex expressions cessors and assuming constant over- algorithm in Figure 14 provides a are to be evaluated, unnecessary se- head. Initially there are n/2 merges skeleton of how this can be done. quential dependencies can create involving only two elements and Figure 15 plots the expected ideal significant bottlenecks. these can be carried out concurrently. behavior of this algorithm (as Consider the concurrent imple- The final step involves a single merge simulated by the Rosette system). It mentation of the mergesort algo- of two lists of roughly n/2 elements. gives the concurrency index as a rithm. Assume we have a linked list The merges takes O(n) time since in function of the number of sweeps. of numbers which we want to sort. Of the final merge one has to walk down Note that the processors are more course, a linked list is a (very) sequen- the two lists doing comparisons. uniformly busy and the maximum tial data structure; we are using it Following Athas [17], Figure 13 number used is only a small fraction here for illustrative purposes only. A gives the concurrency index (CI) for of the number of elements in the list. linked list can be split in n/2 steps the mergesort algorithm executed on The reason for the more uniform where n is the length of the list, pro- 1000 elements. To simplify counting concurrency index is that the concur- vided that n is known-essentially we execution steps, we assume all pro- rency index builds up much more have to walk the: pointer links to the cesses are run synchronously- rapidly as more mergesorts are middle of the list and create a pointer although the algorithm has no syn- triggered. chronous processing requirement. The ease of expressionin an inher- Each interval in the x-axis of the *It is interesting to note that Scheme itselfwas in- spired by an attempt to understand the concept of diagram represents the processing of 3This observation was communicated to rhe author actors as it was first proposed [I]. a single message by all actors which by Chris Tomlinson.

136 I I I I I I I

ently concurrent language simplifies chronous communication tits natur- message may be to return 8 to the noting the data dependencies which ally with the use of a single active customer c which has been supplied are simply expressed as synchroniza- computational thread which weaves in the message. It is often convenient tions implicit in function calls. On through different objects which are to assume that the value of a subex- the other hand, it is possible to ex- invoked and return a value to their pression will be automatically press the same code in terms of se- caller. substituted for the arithmetic subex- quentially executed primitive actor In the context of concurrent com- pression when the expression is eval- bodies-without any meaningful loss puting, however, synchronous com- uated. Such a notation abstracts of speed. The second case requires munication reduces the number of from the explicit synchronization that, instead of waiting for an arbi- objects that may be potentially active which must be implemented in terms trarily large number of objects to at any given point in time. If an asyn- of asynchronous message-passing. execute, the dynamic creation of a chronous communication model is The difference between implicit and number of context objects be explic- used, as in the primitive Actor model explicit synchronization is similar to itly specified to carry out the subcom- (see section entitled “The Actor the difference between an assembly putations. In an inherently high-level Model”), then customers representing language and a (sequential) pro- actor language, this work is simply return addresses must be explicitly gramming language with expres- transferred to a compiler. supplied. For example, the behavior sions. The constructs of the actor of an actor x in response to a [ + 3 c] model are primitives which can Communlcrsf Ion and CoorUlnaflon Because there is no instantaneous ac- tion at a distance, the interactions (define mergesort between components of a distributed (rlambda [list ten] system must be built in terms of (if (= len 1) asynchronous communication. In an list asynchronous communication (let* [first(first-half list ten)] model, a sender is free to take further [second (second-half list len)]] action after dispatching a given mes- (merge (mergesort first) (mergesorl second)))))) sage. This is in contrast to sequential object-oriented languages such as Smalltalk which use synchronous communication: in this case, a sender waits for a response before Concurrency Index for continuing its execution. Syn- Merge Sort

FIGURE 12. A C0M/frent IIN?rgeWf example. The actor checksto see if the length of the list is I, if so returns the single number ia the list. Otherwise it subdivides the list into two halves which are bound to first and second. These two halves are sorted using two concurrent recursive calls to mergesort and the results sorted lists are merged using the actor merge. The es- ample is discussed in 171. Note that the let bindings are evaluated before the let body.

FIGURE 13. COnCUMe/Jcy //f&I for a mergeSoft of 1000 elements. NOtlCa that 3000 the vertical anis of the graph is on log scale Steps (from 173). The horizontal anls represents time steps (sweeps) In which every actor Processes a single message. This synchronous Processing assumption is used only to simplify counting steps and Is not required by the algorithm. The first 1000 steps in- (define mergesort UOlUe determining the length of a given list (tlambda [list len] by walking through it. (if (= len 1) list FacuReu u4. code for a concurrent (merge (mergesort (first-half [list len])) mergesort with with fewer data dependen- ties. Note that the sequential let has been (mergesort (second-half [list Ien])))))) removed.

CCMMUNlCATlCRI CFT”E ACM/September 1990/W 33, No.9 137 I II I I I I I I I I

be used to build higher-order (i.e., internal loops are prohibited ObJect-Oriented concurrent procedural and data within actors). Thus the interaction ProgrammIng abstractions. problem is transferred to another Object-oriented programming sup- level: the number of interactions be- ports the reusability of code, thus tween independently triggered com- supporting an evolutionary pro- Reasoning putations can again be large as an gramming methodology, and it pro- about ObJeCt actor interleaves messages triggered vides modularity in programming, Behavior by distinct requests from different thus allowing a separation of design Because the state of the components senders. This potential disadvantage concerns. These powerful aspects of of a system is constantly changing, it is mitigated by two factors: first, ac- object-oriented programming can be is generally impossible to predeter- tor languages encourage greater use utilized in concurrent programming mine precisely what state a particular of functional components which are by providing mechanisms such as in- component will be in when another referentially transparent and easy heritance and reflection. This section component attempts to interact with to reason about. Second, actor lan- discusses the basic constructs which it. In models of concurrency based on guages use a customer-passing style can be used to build these mechan- communicating sequential processes, to separate the continuation repre- isms and points to some interesting the effect of a m,essage received at senting an actor’s future behavior research issues. all potential entry points within a and the continuation of a computa- process must be considered. In a tional thread. In other models of con- Inheritance shared variable model, even more current objects, multiple entry points A powerful feature of object-oriented interactions are possible as different are possible when synchronous com- languages is inheritance. Inheritance information may be written into munication is used. However, even in was introduced in Simula primarily each shared variable of a process some of these models, the target ob- as a organizational tool for classifica- by any other process, creating an ject is invoked at a single entry point tion. In Simula, objects can be de- exponential number of possibilities and provides a response to the caller fined as members of a class and, as a for interaction. Each interaction at the point of the call. consequence, share procedures that corresponds to a different indeter- A second advantage of COOP is are applicable to all members of the minate execution of the system. the locality properties in the model. class; note that members of a class Actors encapsulate operations so .4n object may send a message only may themselves be classes. Class- that they may be externally invoked to those objects it knows about [15]. based sharing naturally promotes at only at one entry point. This can The axioms governing which objects modularity in the code by putting all be achieved by breaking up a sequen- are known to an object are called the code common to a number of ob- tial process into a number of smaller locality laws; these laws were jects in one place. Modifying and independent objects; in fact, it is developed in the context of the Actor debugging programs is simplified by sometimes possible to use formal model by Hewitt and Baker at MIT. making changes to a class whose transformation rules to automatically Locality laws further restrict the behavior in turn is visible to all its decompose processes into objects number of possible interactions be- members. Organization of objects with a single entry point as proposed tween objects which have to be con- using classification incorporates ob- by Shibayama at the Tokyo Institute sidered in a given system. Locality jects into a tree-like structure and can of Technology (see [24]). Intermedi- laws make it possible to model open provide clarity in knowledge repre- ate states of an actor are invisible to systems, (i.e., evolving systems which sentation-as experience in chem- the outside, and actors may not be are open to interaction with their en- istry (periodic table) and biology interrupted in su.ch states. Because vironment). Because the outside en- (taxonomy of species) has shown. interactions between intermediate vironment is dynamically changing Inheritance essentially makes the states of two actors need not be con- and may contain unknown elements, code in one object (a class) visible to sidered, such decomposition pro- its behavior cannot be completely another object (a member). Code motes modularity; specifically, it can predicted. Therefore an open sharing leads to namespace manage- reduce the complexity of invariant systems model must allow local ment issues-the same identifier properties to be established in order reasoning about a module in dif- may be bound to procedures and to reason about the behavior of a pro- ferent possible contexts. By regu- parameters in an object and in its gram. The distinction between actors lating interactions with other actors, class. Object-oriented languages dif- and procedures with multiple com- locality laws make such local reason- fer regarding how such name con- munication entry points can be ing feasible. In particular, because flicts are handled. In Simula, appreciated by considering their the constituents of the distributed superclass identifiers are renamed; analogy to the difference between system will not be known to a single this essentially provides static bind- procedure calls and unrestricted object in the system, (global) broad- ings which are resolved lexically. goto’s. casting is not generally a meaningful Simula also provides for a virtual The behavior of an actor is atomic, construct in open systems. declaration which allows identifiers I I I I I I I

representing variables (but not those ,m- bound to methods) in a superclass to m- be visible in a subclass-a case of dynamic scoping. Virtuals are used to support incomplete specifications which are to be added to by the subclasses. While Simula was not designed to support concurrency, one of the designers of Simula, Kristen Nygaard at the University of Oslo, 300, ,3L,step has developed in collaboration with Ole Madsen at Arhus University and FIGURE q5. Mergesort with fewer dependencies. The concurrency index for a mergesort of 1000 elements is plotted. In this case, only approrlmately 22 processors are others, the language BETA which ex- used at any given time. Note that the X-a& begins at 1001 to IgnOre the SeqUenthI walk plicitly supports concurrent pro- through the list required to determine its length. gramming. Concurrency in BETA is obtained by explicitly specifying al- and these mechanisms can coexist in network are all implicit. As one ternation or multisequential execu- the same system. This proposal is an moves from a higher-level language tion (see the chapters on BETA in extension of the work of Jagannathan to its implementation language, a PW. on first-class environments [16]. number of objects are given concrete By contrast, Smalltalk takes a The interaction of concurrency representations and can be explicit- more operational view of in- and inheritance raises a number of ly manipulated at the lower im- heritance. Conflicts in identifiers are interesting issues. For example, re- plementation level. For example, the resolved dynamically to provide placement behaviors in actors are join continuation actor in Figure 8 is greater flexibility. This emphasis led spectfied atomically. If inheritance is implicit in Figure 3. When the join to its use primarily as a program- defined in terms of a message- continuation is made explicit, it can ming method to support sharing and passing protocol between one object be usefully modified, as we showed in reusability of code and data, rather and another, the task of determining Figure 10. than as a mechanism for classilica- a replacement may be naturally dis- The dilemma is that if a very low- tion. A good discussion of a number tributed as part of the replacement level language is used, the advantages of issues relating to inheritance and behavior is determined locally and of abstraction provided in a high- object-oriented programming can be part in a different object. This is not level notation are lost. Alternately, found in [22]. (See, specifically the an issue in a sequential language like the flexibility of a low-level language classification of object-oriented lan- Smalltalk which uses synchronous may be lost in a high-level language. guages developed by Peter Wegner at communication with a single active Moreover, although it is possible for Brown University.) Related mech- thread: parts of the state of an object a low-level program to have a model anisms include classless schemes are updated through assignments of its own behavior (for example, as such as dynamic inheritance and made by the object and other parts of in the case of a Universal Turing delegation. the state may be assigned by its class. Machine), this need not always be One proposal, advanced by Jagan- Another complication which arises in the case. A reflective architecture ad- nathan and the author, is to allow concurrent object-oriented languages dresses this problem by allowing us to programmers to define different with inheritance mechanisms is the program in a high-level language possible inheritance mechanisms in interaction between inheritance and without losing the possibility of a single linguistic framework [17]. synchronization constraints. This representing and manipulating the Associated with an object is a local has been a very active area of re- objects that are normally implicit environment which provides bind- search and a number of solutions [18]. Reification operators can be ings for the identifiers in that object. have been proposed (for example, see used to represent at the level of the The idea is to provide the ability to [231); we discuss this interaction application, objects which are in the reify environments (see the discus- briefly in the next section. underlying architecture. These ob- sion in the following section and to jects can then be manipulated like explicitly manipulate them as first ReFIecClon any other objects at the higher ap- class objects). For example, two en- In the normal course of execution plication level. Reflective operators vironments may be composed so as of a program, a number of objects are may then be used to install the to shadow the bindings in one object implicit. In particular, the interpreter modified objects into the underlying (for example, a class) with the bind- or compiler being used to evaluate architecture. Reflection thus provides ings in another object (for example, the code for an object, the text of the a causal connection between the an instance). By using different pos- code, the environment in which the operations performed on this repre- sible compositions, distinct inheri- bindings of identifiers in an object are sentation and the corresponding ob- tance mechanisms can be obtained evaluated, and the communication jects in the underlying architecture.

CCIY”I(ICIIT,CIISOFTREliCMISeptcmber 199OWol.33, No.9 139 I I I I I I I I I I I

In the example of a tree prod- the target. The original sender is then In the POOL object model, each ob- uct, the program can be expressed free to continue its processing. While ject has a body, a local process, which in a high-level language as a func- theoretically feasible, this solution starts as soon as the object is created tional product expression. However, can be inefficient: it may increase the and executes in parallel with the when needed, its join continuation traffic in the communication network bodies of all other objects. Sending can be dynamically reified and a new as a sender repeatedly tries to and receiving of messages is indi- join continuation actor can be in- transmit the message to an unavail- cated explicitly within this body in stalled to perform the necessary able recipient. such a way inside every object every- synchronization. Another solution, used in actor thing proceeds sequentially and de- In a COOP system, the evaluator systems, is to let an object explicitly terministically. In industrial of an object is called its meta-object. buffer incoming communications partnership with Philips, the project Reflective architectures in COOP’s which it is not ready to process (i.e., has also designed a language-driven have been used to implement a num- selective inre&ivi~). For example, an architecture called DOOM (Decen- ber of interesting applications. For actor may need to process messages tralized Object-Oriented Machine). example, Watanabe and Yonezawa in the order in which they are sent by DOOM is a parallel machine con- (see [24]) have used it to separate the a given sender. However, because of sisting of a number of processors (100 logic of an algorithm from its sched- adaptive routing, the order in which in the present prototype), each with uling for the purposes of a simula- messages arrive may be different its own private memory, and con- tion: in order to build a virtual time from the order in which they were nected via a packet-switching net- simulation, messages are time- sent. In this case, messages which ar- work. Many small and several stamped by the meta-object and sent rive out of sequence can simply be medium-to-large applications have to the meta-objec:t of the target which buffered until their predecessors have been written in POOL. Example ap- uses the time-sta.mp to schedule the arrived. plication areas are databases, docu- processing of a message or to decide The insensitive actor approach has ment retrieval, VLSI simulation, ray if a rollback is required. Thus the an important deficiency: it fails to tracing, expert systems, and natural code for an individual object need separate the question of what order language translation. only contain the logic of the simula- a given set of tasks can be executed Another large ESPRIT project, tion, not the mechanisms used to in-i.e., the synchronization con- called ITHACA (Integrated Toolkit carry out the simulation; the specifi- straints from the question of how for Highly Advanced Applications), cation of the mechanisms is separ- those tasks are to be executed-i.e., is working to produce an object- ated into the me:ta-objects. the algorithmic structure of actions to oriented application development One applicat:ion of reflection in be taken. Such a separation would environment including a concurrent actor-based systems is to address support local reasoning about feasi- object-oriented language and the problem of synchronization con- ble actions. In the Rosette language, database, a Software Information straints, (i.e., c:onditions limiting Tomlinson and Singh proposed a Base (which will store a large collec- which communications an actor in a reflective mechanism which reifies a tion of classes), a set of tools for given state is able to process). For ex- mail queue for an actor and modifies browsing, querying and debugging ample, a bounded buffer which is full the queue’s behavior by making it classes, and tools to support interac- cannot service requests to enqueue. sensitive to enabledness conditions tive application construction from In some COOP languages, synchro- which capture the synchronization reusable classes. ITHACA is a nous communication is used to en- constraints of the actor to which the 5-year, 100 man-year/year (12 million force synchronization constraints; queue belongs 11231. ECU/year) project led by Nixdorf, the recipient refuses to accept com- with Bull, Geneva, and three other munications which it is not in a state European partners. An academic to process. This solution, while quite The development of architectures partner in the project is a group simple, can reduce the amount of and systems based on the COOP under the direction of Tsichritzis at concurrency available in a system by model is an active area of research the University of Geneva. requiring suspension of the execution around the world. We briefly de- Japan’s Ministry of International of actions by a sender until the recip- scribe a few of these efforts. This Trade and Industry recently an- ient is ready to accept the message list is by no means complete but nounced that it will support a Coop- -even if the sender’s future behavior gives a flavor for some of the work erating Agents Project based on does not depend on whether the mes- under way. COOP. Initial funding level for this sage has been delivered. In Europe, several large-scale ef- seven year project is estimated to be One approach to increasing con- forts are under way. Under the aus- $35 million. Yonezawa at the Uni- currency in the synchronous com- pices of ESPIRIT, de Bakker, verity of Tokyo, whose group devel- munication model is to dynamically America and others have worked on oped ABCL (an actor-based create a new object which attempts to the definition of a parallel object- concurrent language), is an academic synchronously communicate with oriented language called POOL [5]. leader of this effort. A focus of this

140 September 199O/Vol.33, No.9ICOMYUWICATIOWS OFTREAOM project is to apply the work on actors Europe (PARLE ‘89), ool. ZZ: Parallel Lan- 17. Jagannathan, S. and Agha, G. In- to coordination technology, lunges, LNCS366. Springer-Verlag, New heritance through Reflection. Tech. Rep., In the United States, where a York, 1989. Dept. of Computer Science, Univ. of II- 4. Agha, G. and Jagannathan, S. Reflection linois at Urbana Champaign, 1990. To be foundation for COOP was provided in concurrmt systmw: A model of comment con- published. by the work of Carl Hewitt and asso- tinuationr. Tech. Rep., Dept. ofComputer 18. Maes, P. Computational Reflection. ciates at MIT, a number of smaller Science. Univ. of Illinois at Urbana Ph.D. dissertation, Vrije University, groups are developing COOP sys- Champaign, 1990. To be published. Brussels, Belgium, 1987. Tech. Rep. 87-2. tems. Hewitt’s group in particular is 5. America, P. Issues in the design of a 19. Manning, C. Traveler: the actor obser- focusing on open information sys- parallel object-oriented language. Formal vatory. In Pmceeding-s of European Confumce tems and artificial intelligence appli- Aspects Computing, 1, 4 (1989), 366-411. OR Object-Oriented Programming. Springer cations. Ken Kahn, Vijay Saraswat 6. Annot, J.K. and den Haan, P.A.M. Verlag, New York, January 1987. Also ap- and others at Xerox PARC are work- POOL and DOOM: The object-oriented pears in Lecture Notes in Computer ing on a high-level actor program- approach. In Parallel Computers: Object- Science, vol. 276. ming language called Janus. The Oriented, Functional, Logic, PC. Treleaven, 20. Manning, C. Introduction to program- Ed. Wiley, 1990, pp. 47-79. ming actors in acore. In Towardr Open Zn- author’s group at the University of 7. Athas, W. Finegrain concurrent comptiationr. formation Systenu Science. C. Hewitt and G. Illinois at Urbana-Champaign is Ph.D. dissertation, Computer Science Agha, Ed., MIT Press, Cambridge, currently working on programming Dept., California Institute of Technology, Mass, to be published. abstractions and language models for 1987. Also published as Tech. Rep. 2 1. Moses, Y. Knowledge in a distributed enoiron- dependable concurrent computing. 5242:TR:87. merit. Tech. Rep. STAN-CS-86-1120, Finally, the United States has a 8. Athas, W. and Boden, N. Cantor: An Ac- Computer Science Dept., Stanford considerable lead in innovative tor Programming System for Scientific University, 1986. multicomputer architectures in- Computing. In Proceedings of the NSF 22. Shriver, B. and Wegner, I?, Eds. Research spired by language models closely Workshop on Object-Based Concurrat Pmgmm- Directions in Object Oriented Programming. tied to concurrent object-oriented ming, G. Agha, P. Wegner, and A. MIT Press, Cambridge, Mass., 1987. Yonezawa, Eds. ACM, N.Y., April 1989. 23. Tomlinson, C. and Singh, V. Inheritance programming. pp. 66-68. Special Issue of SIGPLAN and Synchronization with Enabled Sets. Notices. In Proceedings of OOPSLA-89, (1989). To be Acknowledgments. 9. Athas, W. and Seitz, C. Multicomputers: published. ACM, New York. I wish to thank Julian Edwards, message-passing concurrent computers. 24. Yonezawa, A., Ed. ABCL: An Object- Suresh Jagannathan, Carl Manning, IEEE Cornput. 9, 23 (August 1988). Oriented Concurrent System. MIT Press, Jeff Mantei, Satoshi Matsuoka, 10. Dally, W. A VLSZArchitecturefor Concurrent Cambridge, Mass., 1990. Oscar Nierstrasz, Vipin Swarup, Data Structures. Kluwer Academic Press, 25. Yonezawa, A. and Tokoro, M., Eds. 1986. Object-Oriented Concurrent Programming. Peter Wegner, Rebecca Wirfs-Brock 11. Dally, W. The J-Machine: $&zm Supportfor MIT Press, Cambridge, Mass., 1987. and Akinori Yonezawa for helpful Actors. In Towards Open Znformation Systems CR Categories and Subject Descriptors: comments on drafts of this article. Science, C. Hewitt and G. Agha, Eds. (3.1.2. [Processor Architectures] Multiple The ideas expressed in this article M.I.T. Press, Cambridge, Mass., to be Data Stream Architectures; D.1.3. [Program- have benefitted from interactions published. ming Techniques] Concurrent Program- with a number of other individuals 12. Dally, W. and Wills, D. Universal mech- ming; D.3.1. [Programming Languages] including Carl Hewitt, Alan Perlis, anisms for concurrency. In Proceedings of Formal Definitions and Theory-semantics, Mark Scheevel, Vineet Singh, Parallel Architectures and Languages Europe syntax; D.3.3. [Programming Languages] Carolyn Talcott, Chris Tomlinson (PARLE ‘89). k51. ZZ: Parallel Languages. Language Constructs and Becky Will. The work described Springer-Verlag, New York, 1989. LNCS General Terms: Design, Methodology 366. pp. 19-33. in this article has been made possible Additional Key Words and Phrases: Actor 13. Fox, G., Johnson, M., Lyzenga, G., Otto, in part by a Young Investigator model, concurrency, concurrent program- S., Salmon, J., and Walker, D. Soloing ming structures, multiprocessor architectures, Award from the Office of Naval Probloblemr on Concurrent Processors Vol I, object-oriented programming, programming Research (ONR contract number General Techniques and Regular Problems language theory N00014-90-J-1899) and in part by an Prentice Hall, Englewood Cliffs, New About the Author Jersey, 1988. Incentives for Excellence Award from GUL AGHA is assistant professor and direc- 14. Hewitt, C. Viewing control structures as the Digital Equipment Corporation. tor of the Open Systems Laboratory in the patterns ofpassing messages.J. Artif Zn- Department of Computer Science at the References tell. 8, 3 (June 1977) 323-364. University of Illinois at Urbana-Champaign. 15. Hewitt, C. and Baker, H. Laws forcom- 1. Abelson, H. and Sussman, C.J. Structure His current research interests include concur- municating parallel processes. In 1977 andlntnpretation ofComputer Programs. MIT rent programming languages, open system ZFZP Congress Proceedings, IFIP (August Press, Cambridge, Mass, 1985. architecture and coordination technology. 1977) pp. 987-992. 2. Agha, G. Actors: A Model of Concurrent Author’s Present Address: Dept. ofcomputer 16. Jagannathan, S. A Programming Lan- Computation in Distributed Systems. MIT Science, 1304 W. Springfield Ave., University guage Supporting First-Class, Parallel Press, Cambridge, Mass., 1986. of Illinois at Urbana-Champaign, Urbana, IL Environments. Tech. Rep. LCS-TR 434, 3. Agha, G. Supporting multiparadigm pro- 61801. agha@?cs.uiuc.edu. gramming on actor architectures, In Pro- Massachusetts Institute of Technology, ceedings of Parallel Architectures and Languages December 1988. @ 1990 ACM OOOl-0782/90/0900-0125 $1.50

COYLIUNICITIONSCFT”ElCY/Septembcr 199O/Vol.33, No.9