Norms As Mental Objects -- from Normative Beliefs to Normative Goals

RosariaConte* Cristiano Castelfranchi*

set up the value of the resources 1. Introduction exchanged, that is, what each is entitled to expect from others in retum, given the The study of norms (Ns) is resources invested. Without a norm of recent but growing concern in the AI reciprocation, such expectation would be field (cf. [MOS92];[SHO92]) as well easily disconfirmed when social within other formal approaches to social exchange, as often happens, does not action, such as Game Theory (where occur simultaneously. The utilitarian theories of Ns began to appear long agent would spontaneously respect the before, cf.[LEW69]; [ULL77]). The agreement only if the partner is regarded normative is an essential dimension of as stronger. Otherwise, he would get social life. An explicit model of such away with no costs, in a word, he would dimension is crucial for any theory of cheat. social action. Notions such as "social Analogously, joint intentions and commitment" (cf. [GAS91]; [COH90]; teamwork arise from true negotiation etc.) "joint intentions", "teamwork" and create rights based on commitments. [COH91], "negotiation" [DAV83], Finally, roles are but sets of obligations "social roles" [WER89] etc. will not and rights. escape a purely metaphorical use if their A norm-abiding behavior need normative character is not accounted for. not be based on the cognitive processing Indeed, one’s committing oneself of Ns (it might be simply due to before someoneelse causes the latter to imitation), nor does a normative mind be entitled, on the very basis of necessarily produce a behavior commitment, to expect, control, and corresponding to Ns (Ns are operating in exact the action one has committed the mind even if the agent’s final oneself to. Social commitment is a decision is to transgress against them). customary, if not the most frequent, To model the normative reasoning does source of entitlements in everyday not imply to model a norm-abiding interaction: first of all, a promisecreates system. However, a large portion of an entitled expectation in the addressee, autonomous agents’ normative behavior namely a belief that the promised course is allowed by a cognitive processing of of action will be realized. As a Ns. Therefore, Ns are not only an consequence, the addressee feels entitled essentially social but also a mentalistic to exact the action expected, that is, to notion. Ns are indeed a typical node of control its occurrence and react if it is the micro-macrolink, of the link between not performed. In particular, the individual agents and collective addressee is entitled to express a public phenomena, minds and social structures. protest and receive the consent and A view of normative systems as support of the community of witnesses "emerging properties" of the social before whomthe action of commitment systems, improving coordination within took place. the social systems themselves (cf. Furthermore, there is no true [MOS92])is not sufficient. negotiation without control for cheaters On one hand, that of and norm of reciprocation (of. coordination is only one of the [GOU60]). Whensocial agents negotiate functionalities of Ns. Amongothers, an exchange of resources, they actually consider the function of identification, (the good manners, for example, improve identification rather than coordination: on their grounds, in- SBSP- Social BehaviorSimulation Project, Istituto di Psicologia,CNR, V.le Marx15, 1- groups are allowed to be identified, etc.); 00137Roma, Italy. that of protecting the single’s interests Tel: (06) 8292626,Fax: (06) 824737,e-mail: beyond its own will (as happens with the PSCS~IRMKANT.BITNET norms prescribing to fasten seat belts, to wear helmets when riding motorcycles

40 etc.) Furthermore,there are at least two and someinitial solutions proposed. types of norms of coordination: those The problemsare the following: whichare aimedat improvingthe agents’ performances(like the traffic norms), - Howare Ns represented in the and those which are intended to reduce agents’ minds? Should they be displayed aggression (like the normof seen as a specific mental object, reciprocation). and if so, whichone? On the other hand, the - Whichrelation do they bear with emergenceparadigm gives an account of beliefs and goals? Howcan they conventional norms and conforming regulate the agents’ behaviors? behaviors (cf. [BIC90]), but leaves - Howis their prescriptive (and not unexplainedthe prescriptive character of simply conventional) character norms. Within the game-theoretic (that is, a more or less explicit paradigm,a social normis defined as an request and the corresponding equilibrium, a combinationof strategies duty) expressed? such that "each maximizeshis expected - Whydoes an autonomous agent utility by conforming,on the condition complywith norms, thus fulfilling that nearly everybodyelse conforms to others’ expectations? the norm." (cf. [BIC90: 842]) Social norms are defined as behavioral 2. Thenormative coin. regularities which emerge from the strategic agents’ choices to conform. What is a norm in an agent’s This is an account of the spreading of mind?This question leaves aside another certain behaviors over a population of fundamental question, namely what is a strategic agents, in a word, a modelof N tout court. Althoughprioritary, wewill social conventions. However, such a not face this question here (however,see model does not account for a crucial [CAS91]).Therefore, we will not try to aspect of the normative mechanism, answer the question what the "external" which plays a role in the spreading of side of Ns is, namely, how Ns are normative behaviors, namely the encountered in social life, which normativerequest. Whatis lacking in the functionalities they have, whichfeatures game-theoretic definition define de facto (independent of agents’ aforementioned is a reference to the beliefs) a normativesource, etc.. agent’s will that others conformto the Here, we will focus exclusively norms. A social norm, indeed, is such if on the "internal" side of Ns, that is: it is associated at least with a general Whichcognitive role do Ns play in the wantthat it be observed. Such a wantis agents’ minds and in which format are usually conveyed in many ways: from they represented? Whatkind of mental simple expectations (note that the term object is a N? Whatis a normativesource expectation is often used to refer to a in the agents’beliefs? hybrid mental object, namely a There are at least two distinct goal/belief: if you expect the weather ways in which norms can be tomorrow to be sunny and warm you implemented on a computer system: as both believe and wish it to be so), to built-in functioning rules and constraints explicit commandsand requests; from (like productionrules) or as explicit and implicit disapproval of transgressors, to specific mentalobjects (i.e. obligations, explicit reproachetc.. duties, etc.) distinct from, say, goals and Therefore, here it is intended to beliefs. Of course, there might be draw the attention on both the following intermediate solutions: for example,and aspects: in line with the game-theoreticview, one might think of noms as a way of ¯ the prescriptive character of Ns, describing the behaviors of strategic that is, their role in controlling and agents in interaction: no specific regulating the behaviors of agents normativeobject is implementedin those subject to them; agents’ mindsalthough cognitive action is allowed. A further alternative consists ¯ a need for a mentalistic notion of of implementingNs just as goals, namely Ns allowing cognitive agents to final ends: In this case, cognitive agents becomenormative agents as well. Ns would be allowed to choose amongtheir are not yet sufficiently characterized (competing) goals (instead of simply as mental objects (cf [SHO92]).Here, applying procedures and routines) but some crucial problems concerning they wouldtreat Ns as any other goal of the mentalnature of Ns will be raised theirs. At least two undesirable

41 consequences seem to derive from the the future behavior of the agents subject latter eventuality: the observance of Ns to them, their addressees (As). would come to depend exclusively upon To this end, they ought to give the agents’ subjective preferences, and rise to some new goal in an A’s mind there would be no social control and (cfr. [CON91b]): for autonomous agents influencing. To implement norms as to undertake (or abstain from) a course specific mental objects, although a costly of action, it is not sufficient that they alternative, is required in a cognitive knowthat this course of action is wanted modelling approach to social agenthood, (by someone else) (cf. [ROS88]; in which the present work is framed. [GAL90]). It is necessary that these Moreover, the cognitive regulatory agents have the goal to do so. Norms endowment of the human species has may act as a mechanism of goal- undoubtedly profited by the explicit generation. Indeed, they represent a representation of norms. A crucial but powerful mechanism for inducing new difficult task, beyond the scope of the goals in people’s minds in a cognitive present work, would then be to explore way. Howis this possible? these advantages and confront them with At first, Ns need being those of all alternatives considered. represented as beliefs in an A’s mind. Let Our language draws on us start from beliefs about requests. A [COH90]’smodel for describing rational simple request, before succeeding and interaction. For the readability of the producing acceptance on the side of the paper, let us provide the semantics of the required person, is nothing but a belief, main predicates used and the axiom namely a belief about someone’s will. (A1) here called the axiom of goal- Such a belief can be expressed as generation (for further analysis, see follows: [COH90], [CON91a], [CON91b]): (BELx (GOALy (GOALx p))) (1) a: action; p andq: states of the world; where agent x believes someone else, v: a worldstatepositively valued; agent y, has the goal that x has the goal x ,y, andz single agents; that p. Morespecifically, x believes that (BELxp): x hasp as a belief what y requires of her is to do an action (GOALx p): x wantsthat p is true at some planned for p: pointin the future. (OUGHTp): an obligationconcerning any givenproposition p; (BELx (GOALy ((DONEx a))). (2) (DONEx a) = def (DONEa) A (AGT action a has beendone by Now,two questions arise here: agentx; Op p is true at somepoint in the 1. what is the difference, if any, future. betweenthis type of belief, that is, a ((GOALxp)^(BELx(q~Op)))~ belief about an ordinary will, and a O(GOALx q) belief about a normative or ifx wantsp andbelieves that prescriptive will? Is a N always if q thanp will follow,then x represented in people’s minds as an will wantthat q as well. expression of some particular (OBTAINx p)=def (GOAL x p) AO external will? (BELx p)) x obtainsp iff p is a goal 2. how do we go from a belief about of x’s andlater it will be a normative will to the goal to true andx will believeso. complywith it, that is, to a normative (GOAL-CONFLx y p q)=def goal? (GOALx p) A (GOALy ^(p vq) Starting from the former goalswhich consist of question, one could say, that normative incompatiblepropositions are beliefs are beliefs about a general will in conflict. affecting a class of agents. This equals to saying that there is a class of agents 2.1. Thenormative belief. wanting some of them to accomplish a given action. This view is interesting and Ns are generally considered as fits rather well the DAI field, where prescriptions, directives, commands. attention is now increasingly paid to Even when expressed in any other type shared mental states and collective action of speech act they are meant to "direct" ([GRO90]; [LEV90]; [COH91], etc.; for

42 review and further analysis, see coincide with the whole superset Yi. In’ [RAO92]). However, it presents two the latter case, the normative "will drawbacks. coincides with the group’s will. But this is First, it relies upon a notion of not necessary. Suffice it to say that, in a collective or group’s will as yet normative belief, a subcomponent fundamentally distributive. In [RAO92], (individual or sociall) of the group for instance, a social agent’s wants and mentioned to issue a request. In x’s beliefs are defined as the conjunction of belief, what makesa request normative is goals and beliefs of the group’s the very fact that a given z is believed to members. The authors contrast this with want that Yi have an obligation to do a. the opposite view (held, for instance, by To be noted, what is here [SEAg0])of social entity as "irreducible" proposed is a notion of abstract to their members.An alternative to both obligation, one which goes beyond any views exists, namely to say that personal (or supposedly such) will. In individuals form a collective entity if coercion, for example, the coercive agent they objectively depend on one another does not (nor is believed to) have the to achieve one and the same goal or goal that the coerced agent believes in an interest (the latter being defined in abstract obligation to do something. All [CON91a]as a worldstate neither wanted he needs is to persuade other that she is nor believed by the agents involved forced (namely threatened) to do what which nonetheless implies the future the coercive agent wants. On the achievement of (one o0 their goals). contrary, a normative will is believed to Furthermore, the group may achieve its create a mental state of abstract common goal, or realize its common obligation, independent of any personal interest, by distributing tasks amongits want or need. In other words, z is not members. However, the task assignment (believed to be) happy with yi’s doing a may be accomplished by a specialized in virtue of z’s personal request (be it subcomponent of the group. In other coercive or not). By default a normative words, the notion of group will can be will is one which wants you to have an "reduced" to the mental states (of some) obligation to do something, and not of its members without necessarily simply the corresponding goal. supposing shared goals, or even joint Furthermore, a normative will is intentions, usually believed in turn to be grounded Secondly, and moreover, a on norms, to be norm-based. More normative belief seems to be grounded specifically, a strong sense of normative on something more than a general will. will occurs whenthat will is characterized In particular, what seems to be implied is as held to issue Ns: a notion of obligation, or duty. In this perspective, let us express the general (N-BEL x Yi a)~ ( 5 form of a normative belief as follows: :3z(BEL x(OUGHT(GOALz (OUGHT(DONEYi a))))) (N-BELx Yi a) =def A weaker or milder meaning is that of ( Ai.I~,(BEL x (OUGHT(DONE Yi a))))) entitled, or legitimate will, a legitimate goal being defined as follows: where (OUGHT(DONE Yi a)) stands for an obligation for a set of agents Yi to do (L-GOALx p ) =def (6) action a. The question is: what relation Vy3q(GOAL-CONFLx y p q) does (3) bear with belief type (2)? This relation seems possible thanks to the (OUGHT--~GOAL y q)) notion of normative will. In other terms, in words: p is a legitimate goal of x’s iff a normative belief implies a belief about forall agents y that happen to have a the existence of a normativewill: given goal q conflicting with p, y ends (N-BEL x Yi a)~ (4) up with having an obligation to give up 3z(BEL x (GOALz (OUGHT(DONE Yl a)))) 1 As in [RAO92],the definition of a social entity where z belongs to a higher level set of should be recursive. Therefore, what has been said agents yj’ of which Yi is a subset: An with regard to the group at large applies to its agent x has a normative belief about subcomponentsas well. In case z in turn is a action a if x believes that someonewants multi agent subcomponent, its will might be that it is obligatory for Yi to do a. This is shared or not amongits members.Its task (say, to legiferate) might be accomplished in such a way a minimal condition, since z might even that not all membersshare the same goals. q. In other words, a legitimate goal is society’s roles does not influence in any watched over by a norm. In this sense, relevant way her decisions. What is rights and legitimacy are said to give neededfor an agent to regard herself as assistance to, and go to the rescue of, subject to, addressedby, a given N? those who cannot defend themselves against aggression and cheat. 3.1. Thepertinence belief. Consequently, an entitled will is that which a norm protects from any First another belief is needed, conflictinginterest. namely a pertinence belief: For x to To sumup, for an agent to have believe that she is addressed by a given a normativebelief is sufficient to believe N, x needs to believe that she is a that there is an obligationfor a givenset memberof the class of As of that N: of agents to do a given action. At a more careful examination, however, the (P-N-BEL x a) =def (7) obligation is believedto implythat: (Ai=l,n (N-BELx Yi a)) A (Vj.I,n(BEL x (x - a givenaction is prescribed,that is, where P-N-BEL stands for normative requested by belief of pertinence. Now,x’s beliefs tell her not only - a norm-basedwill, be it held to that there is an obligation to do action a, issue that request, or simplyentitled but also that the obligation concerns to do so. preciselyherself. Of course, a N-belief does not 3.2. Thenormative goal. implythat a deliberate issuing of a N has in fact occurred. Social normsare often Still, (7) is not much less set up by virtue of functional unwanted "descriptive" than (3). Wedo not see any effects. However,once a given effect is normativegoal, yet. believed to be a social norm,an entitled First, let us express a N-goalas will is also believedto be implied,if only an anonymous one ("You are follows: wanted/expectedto (not) do this...", "It (N-GOALx a) =def generally expectedthat...", "This is done (P-N-BEL x a) A (GOALx (DONEx SO...", etc.). This equals to saying that the A normative goal of a given present modelof N-beliefs is recursive. agent x about action a is therefore a goal A request is believedto be normativeif it that x happensto have as long as she has ultimately traces back to some norm. a pertinence normative belief about a. This is not to say that Ns are Ultimately, x has a normativegoal in so "irreducible" objects. Of course, the far as and because she believes to be origins of Ns call for an explanation subject to a N. Therefore, a N-goal which unavoidably brings into play the differs, on one hand, from a simple communityof agents, their interests and constrain which reduces the set of their interactional practice (cf. actions available to the system (cf. [ULL77]). However, in the agents’ [SHO92]),and, on the other, from other representations there is no need for ordinary goals. keeping a record of such history. In the With regard to behavioral agents’ beliefs a N is alwaysrepresented constrains, a N-goalis less compelling: as a legitimate, even a norm-driven, An agent endowed with N-goals is prescription. The present modeltries to allowed to compare them with other give an accountof this evidence. goals of hers and to someextent freely choose whichone will be executed. Only 3. Theroute of Ns in the mind. if endowedwith N-goals an agent may legitimately be said to complywith, or Turning to question 2. raised violate, a N. Onlyin such a case, indeed, above, a normativebelief is only one of she maybe said to be truly normative. the ingredients of normativereasoning. With regard to ordinary goals, a Norms,indeed, are hybrid configurations N-goal is obviously more compelling: of beliefs and goals. Actually, as defined whenan agent decides to give it up, she so far, a normative belief is only knowsshe both thwarts one of her goals descriptive: it does not "constrain" or regulate the believer and his decisions. Indeed, an observer’s description of a

44 and violates a N 2 There seem to be several ways of Now, the question is: Howand accounting for the process leading to why does a N-belief come to interfere normative goals (N-goals) as well with x’s decisions? What is it that makes several alternative waysof constructing a her "responsive" to the Ns concerning N-agent. There also seems to be a her? What is it that makes a normative correspondence between the process belief turn into a normative goal? from a belief about an ordinary request to the decision of accepting such a 3.2.1. Goal- and norm-adoption. request, which we called (el. [CON91b]) goal-adoption, and the process from a N-belief to a N-goal, which by analogy will be called here norm-adoption. Goal-Adoption Norm-Adoption 1. Conditional Action Slavish Automatic

(BELx (GOALy (DONE x a))) (P-N-BELx a) D 0(IX)NE 0 (DONExa) 2.Instrumental Adoption Self-interested Utilitarian thanks to (AI) Vp3q.(BELx ((OBTAIN y Va..qp.((P-N-BELx a) 0(OBTAINx q)))D (BELx ((DONEx 0(GOALx (OBTAINy p)) O(OBTAINx p))))D 0(N-GOALx a) 3.Cooperative Adoption Co-interested Value-driven thanks to (AI)) Vp3q.(BELx((OBTAINyp)D 0q))~ Va3q.((P-N-BELxa)A 0(GOALx (OBTAINy p)) (BELx ((DONEa) ~ Oq)))D O(N-GOALx a) with q’s being (in x’s beliefs) commonlywanted by x and y’ with q’s being(in x’s beliefs) worldstate positively value by both x andthe normativesource: (BELx (q = v(xz))) withv standingfor anyvalue. 4. Terminal Adoption Benevolent Kantian

(Ay=l,n (GOAL (Ax=I,n(N-GOALn( x))) (OBTAINy py))) withax beingthe set of N-actions withpy beingthe set of y’s goals. requiredof x Table 1: The route of Ns in the mind.

In situation 1. (conditional goal is fired. Now,this is a rather cheap action), we find some sort of production solution: no reasoning and autonomy are rule: in goal-adoption (G-A), anytime allowed. It is simple machinery that request is received by a system endowed could be of help in cutting short some with such a rule, a goal that a be done is practical reasoning, but is insufficient as fired. Analoguously, in N-adoption (N- far as the modelling of normative A), anytime a N-belief is formed a N- reasoning is concerned. However, such a rule seems to account for a number of real-life situations. Think, as far as 2 Intuitively, she givesup boththe expected slavish G-Ais concerned, of the habit of consequencesof the action prescribed(any giving instructions when asked by worldstatesupposedly convenient to the agentor passengers, and in the case of automatic otherwisepositively valued) and in addition N-A, of the routine of stopping at the sustains the costs of N-transgression.Although red light (of course, in situations 1, it is required,a formaltreatment of bothaspects is beyondthe scopeof this work. hard to differentiate G-A from N-A).

45 In situations 2 (instrumental low expected chances of being caught adoption), greater autonomy is allowed: red-handed, etc.), what is likely to occur adoption is subject to restrictions. In G- is a confrontation with other As of a A, on the base of this rule, x will self- given N. As known, a high rate of interestedly adopt only those of y’s goals transgressions observed discourages which x believes to be a sufficient one’s compliance. Viceversa, and for the condition for x to achieve some of hers. samereason, it is possible to showthat if Typically, but not exclusively, this rule one has complied with a given N, one depicts situations of exchange. An will be likely to influence other agents to utilitarian N-Arule says that forall Ns, x do the same (normative equity). Indeed, will have the corresponding N-goals if it can be argued (cf. [CON92]) that she believes she can get something out normative influencing plays a rather of complying with them. (Think of the relevant role in the spreading of observance of Ns for fear of punishment, normative behavior over a population of need of approval, desire to be praised, autonomous agents. etc..) Cooperative, or cointerested, 4. Conclusive remarks and future goal adoption occurs whenever an agent research adopts another’s goal to achieve a commongoal. N-adoption is cooperative In this paper, the necessity of a whenit is value-driven, that is, whenthe cognitive modelling of norms has been agent autonomously shares both the end argued. It is proposed to keep distinct of the normand the belief that the latter the normative choice from any norm- achieves that end. This type of N-Acan like behavior, that is that behavior which be seen as some sort of moral appears to correspond to norms. Such a cooperation since the effect of the norm difference is shown to be allowed only is shared (in the N-addressee’s beliefs) thanks to a theory of norms as a two- by the aaddressee and the normative fold object (internal, that is, mental and source. external, or societal). The last situation is terminal Some instruments, still rather adoption. This is not a rule, but a meta- tentative, have been proposed for a goal which is defined, in the case of G- formal treatment of the "internal side" of A, as benevolent (x is benevolent with Ns. In particular, a view of Ns as a regard to y whenshe wants the whole set complex mental object has been of y’s goals to be achieved), and, in the attempted. This object has been shown to case of N-A, "Kantian .... (x wants to consist of other more specific observe the whole set of Ns addressing ingredients, namely goals and beliefs. herself as ends in themselves). Two notions of normative belief and In situation 1, the rule is a typical goal have been provided and discussed, production rule. Its output is an action. and aspects of the process of norm- In situations 2 and 3, the rules output adoption examined and confronted with some specific goals. In the case of N-A, the process of adopting another agent’s the agent ends up with a new type of goals. goal, namely a normative goal. As seen at the beginning, this Acknowledgement implies x’s belief that she is requested to do a by a normative will. But it implies Wewould like to thank Dr. Gianni Amati two further beliefs as well, namely that for his helpful reading the paper. the normative source is not acting in its own personal interests; and that other agents are subject to the same entitled References request (in a normative belief, a set of norm addressees is always mentioned). [BIC90] Bicchieri, C Norms of Now,these further aspects play a relevant cooperation. Ethics, 100, 1990, 838- role, especially within the process 861. leading from N-goals to N-actions. A N-goal, in fact, is not sufficient [CAS91] Castelfranchi, C. & Conte, R. for an agent to comply with a N. Several Problemi di rappresentazione mentale factors occurring within the process delle norme. Le strutture della mente leading from N-goals to N-actions might normativa, in R. Conte (ed.) La norma. cause the agent to abandon the goal and Mente e regolazione sociale. Roma, transgress against the norm. Amongthe Editori Riuniti, 1991, 157-193. others (more urgent conflicting goals;

46 [COH90]Cohen, P. R. & Levesque, H. Artificial Intelligence (AAAI-90), 1990, J. Persistence, Intention, and 94 -99. Commitment, in P.R Cohen, J. Morgan & M.A. Pollack (eds.) Intentions in [LEW69] Lewis, D. Convention. Communication. Cambridge, MA, MIT, Cambridge, MA. Harvard University 1990. Press, 1969.

[COH91] Cohen, P. & Levesque, H.J. [MOS92] Moses, Y. & Tennenholtz, M. Teamwork.. TR-SRIInternational, 1991. On Computational Aspects of Artificial Social Systems. Proc. of the 11th DAI [CON91a]Conte, R. & Castelfranchi, C. Workshop, Glen Arbor, February 1992. Mind is not enough. Precognitive bases of social action. In N. Gilbert (ed.), [RAO92]Rao, A. S., Georgeff, M.P., & Proceedings of the Simulating Societies Sonenmerg, E.A. Social plans: A Sympsium ’92, London, UCL Press, in preliminary report, in E. Wemer& Y, press; TR-IP-PSCS,1991. Demazeau(eds.) Decentralized AI - 3. North Holland, Elsevier, 1992. [CON91b] Conte, R., Miceli, M. & Castelfranchi, C. Limits and levels of [ROS88] Rosenschein, J.S. & cooperation. Disentangling various types Genesereth, M.R. Deals among rational of prosocial interaction. In Y. Demazeau agents, in B.A. Huberman (ed.) The & J.P. Mueller (eds.) Decentralized AI-2. ecology of computation. North-Holland, North-Holland, Elsevier, 1991, 147-157. Elsevier, 1988. [CON92]Conte, R. & Castelfranchi, C. [SEA90] Searle, J.R. Collective Minds and Norms: Types of normative intentions and actions, in P.R. Cohen, J. reasoning. In C. Bicchieri & A. Pagnini Morgan, & M.E. Pollack (eds.) (eds.), Proceedings of the 2nd Meeting Intentions in communication. on "Knowledge, Belief, and Strategic Cambridge, MA,MIT Press, 1990. Interaction", Cambridge University Press, in press; TR-IP-PSCS,1992. [SHO92] Shoham, Y. & Tennenholtz, M. Onthe synthesis of useful social laws for [DAV83] Davies,R. & Smith P.G. artificial agent societies. Proc. of the Negotiation as metaphor for distributed AAAI Conference, 1992, 276-281. problem-solving. Artificial Intelligence, 20, 1983, 63-109. [ULL77] Ulman-Margalit, E. The emergence of norms. Oxford, Clarendon, [GAL90]Galliers, J.R. The positive role 1977. of conflict in cooperative multi-agent systems, in Y. Demazeau, & J.P. MueUer [WER89]Wemer, E. Cooperating agents: (eds) Decentralized AI. North-Holland, A unified theory of communication and Elsevier, 1990. social structure, in M. Huhns and L. Gasser (eds.), Distributed artificial [GAS91] Gasser, L. Social conceptions intelligence, Vol. 2, Kaufmann and of knowledge and action: DAI Pitman, London, 1989. foundations and open systems semantics. Artificial intelligenc, 47, 1991, 107-138.

[GRO90] Grosz, B.J. & Sidner, C.L. Plans for discourse, in P.R. Cohen, J. Morgan, & M.E. Pollack (eds.) Intentions in communication. Cambridge, MA,MIT Press, 1990.

[GOU60] Gouldner, A. The norm of reciprocity: A preliminary statement. American Sociological Review, 25, 1960, 161-179. [LEV90] Levesque, H.J., Cohen, P.R., & Nunes, J.H.T. On acting together. Proc. of the Eighth National Conference on