Regulative and Constitutive Norms in Normative Multiagent Systems

Guido Boella Leendert van der Torre∗ Dipartimento di Informatica CWI Amsterdam & Delft University of Technology Universita` di Torino - Italy The Netherlands e-mail: [email protected] e-mail: [email protected]

Abstract which specify the ideal behavior of agents. Searle (1995) ob- serves that to describe the construction of social reality one In this paper we introduce a formal framework for the con- needs, besides regulative norms like obligations, prohibi- struction of normative multiagent systems, based on Searle’s tions and permissions, also what he calls constitutive norms, notion of the construction of social reality. Within the struc- ture of normative multiagent systems we distinguish between which define that something counts as something else for a regulative norms that describe obligations, prohibitions and given institution. We are interested in formal models of how permissions, and constitutive norms that regulate the creation the normative system creates institutional reality and regu- of institutional facts as well as the modification of the norma- lates the changes that the agents of the system can perform tive system itself. Using the metaphor of normative systems by means of constitutive rules. The research questions we as agents, we attribute mental attitudes to the normative sys- address in this paper are as follows. tem. In particular, we formalize regulative norms as goals of the normative system, and constitutive norms as beliefs of the 1. How to define a formal framework for normative multia- normative system. Agents reason about norm creation using gent systems including regulative and constitutive norms? recursive modelling. 2. How to reason about modifications of the normative sys- tem in this framework? Introduction 3. How to play games and other behaviors in this normative multiagent system, including violations of norms? Normative multiagent systems are “sets of agents (human or artificial) whose interactions can fruitfully be regarded as The challenge of the framework is to balance the first and norm-governed; the norms prescribe how the agents ideally the last question, that is, to balance on the one hand logical should and should not behave. [...] Importantly, the norms techniques to describe the agents, the norms, the institutional allow for the possibility that actual behavior may at times structures, et cetera, and on the other hand game theoretic deviate from the ideal, i.e., that violations of obligations, or techniques to describe the games the agents play. We pro- of agents’ rights, may occur” (Jones & Carmo 2001). Many ceed from the logical perspective to describe the static struc- theories and applications of multiagent systems such as elec- ture of a normative multiagent system, and only consider a tronic commerce, virtual communities, theories of fraud and limited set of games. We use input/output logic (Makinson deception, of trust dynamics and reputation, et cetera, can & van der Torre 2000; 2001) to reason about mental attitudes fruitfully employ the notion of a normative system regulat- in the normative multiagent system. We do not incorporate ing an agent society. For example, norms allow to regu- an action logic, but we define the games of the agents in late systems of heterogeneous agents where the absence of terms of decision variables. a central design prevents enforcing a given behavior by con- As a running example, we consider an agent who likes straining the architecture. In earlier work we have studied to cultivate some crop, and believes that it is sanctioned if it the representation of norms and their role in reasoning. One does not own the field. Since putting a fence around the field question is whether norms require an explicit representation, counts as making the field its property, its optimal decision and if they do, whether they should be represented as prim- is to build a fence and cultivate the crop. Moreover, if a sec- itive entities or in terms of other explicitly represented no- ond obligation to have an authorization of the land register tions like beliefs and goals of agents. Another question is is created when it becomes an estate owner, and the agent’s how agents take decisions when they are subject to norms. applying for registration counts as being authorized accord- Most formalizations of normative systems, including our ing to some constitutive norm, then the optimal decision of own previous work on normative multiagent systems, iden- the agent includes applying for the registration. tify norms with obligations, prohibitions and permissions The paper is organized as follows. We first introduce the normative multiagent system, regulative and constitutive ∗Supported by the ArchiMate research project. norms, and games. Thereafter we use constitutive norms to Copyright °c 2004, American Association for Artificial Intelli- regulate modifications of the normative system. gence (www.aaai.org). All rights reserved.

KR 2004 255 The construction of social reality Methodology We introduce a formal framework for the construction of There are many formalisms to describe the evolution of nor- normative multiagent systems, based on Searle’s notion of mative multiagent systems. In principle, it can be formalized the construction of social reality. Searle (1969) argues that by for example state transition systems like dynamic deon- there is a distinction between two types of rules. tic logic, action languages as developed in artificial intel- ligence, or generalizations of classical decision theory like “Some rules regulate antecedently existing forms of Markov decision processes. In this paper we proceed from behaviour. For example, the rules of polite table be- the logical perspective, and we only consider a limited set of haviour regulate eating, but eating exists independently games. More precisely, we model normative multiagent sys- of these rules. Some rules, on the other hand, do not tems as detailed but static rule-based structures, we model merely regulate an antecedently existing activity called games by a simple protocol that only contains a finite se- playing chess; they, as it were, create the possibility quence of agents making a move, and we specify decision of or define that activity. The activity of playing chess problems by an initial normative multiagent system and a is constituted by action in accordance with these rules. protocol. The behavior is a sequence of decisions of the The institutions of marriage, money, and promising are agents as specified by the game protocol, and the effects of like the institutions of baseball and chess in that they these decisions are updated normative multiagent systems as are systems of such constitutive rules or conventions” well as other effects. (Searle 1969, p. 131) Moreover, to formalize games and other behaviors in the Within normative multiagent systems we distinguish be- normative multiagent system, we attribute mental states to tween regulative norms that describe obligations, prohibi- agents as well as to normative systems, thus we model them tions and permissions, and constitutive norms that regulate as agents too. This has been proposed by Boella and Lesmo the creation of institutional facts like property, marriage and (2002) and may be seen as an instance of Dennett’s inten- money, as well as the modification of normative system it- tional stance (Dennett 1987). The main reason is that it fa- self. Constitutive norms are introduced in our normative cilitates the specification of games in which agents take the multiagent systems for the following three reasons. (autonomous!) normative system into account. For example, First of all, regulative norms are not categorical, but con- an agent considers whether its actions will lead to a sanction ditional: they specify all their applicability conditions. In of the normative system. The advantage of the approach case of complex and rapidly evolving systems new situations is that standard techniques developed in decision and game arise which should be considered in the conditions of the theory can be applied to normative reasoning. Moreover, norms. Thus, new regulative norms must be introduced each the use of the agent metaphor is also useful to describe the time the applicability conditions must be extended to include structural relations between agents playing roles in a norma- new cases. In order to avoid changing existing norms or tive system like legislators creating norms, judges counting adding new ones, it would be more economic that regulative behavior as violations and associating sanctions, policemen norms could factor out particular cases and refer, instead, to enforcing sanctions, and citizens signing contracts. For ex- more abstract concepts only. Hence, the normative system ample, the normative agent may contain the role of a legis- should include some mechanism to introduce new institu- lator, a judge and a policeman. Finally, obligations of the tional categories of abstract entities for classifying possible agents can be formalized as desires or goals of the norma- states of affairs. Norms could refer to this institutional clas- tive agent. This representation may be paraphrased as “Your sification of reality rather than to the commonsense classifi- wish is my command”, because the desires or wishes of cation (Breuker, Valente, & Winkels 1997): changes to the the normative agent are the obligations or commands of the conditions of the norms would be reduced to changes to the other agents. The goals of the normative system describe the institutional classification of reality. ideal behavior of the system. Likewise, constitutive norms Second, the dynamics of the social order which the nor- can be formalized as beliefs of the normative agent. This is mative system aims to achieve is due to the evolution of the explained in detail later in this paper. normative system over time, which introduces new norms, The application of the agent metaphor is in this paper only abrogates outdated ones, and, as just noticed, changes its in- a useful technical trick, but we note that it can also be ex- stitutional classification of reality. So the normative system plained from a more philosophical point of view. In par- must specify how the normative system itself can be changed ticular, it is inspired by the interpretation of normative sys- by introducing new regulative norms and new institutional tems as dynamic social order (Boella & van der Torre 2003a; categories, and specify by whom the changes can be done. 2004). According to Castelfranchi (2000), a social order is Third, the dynamics of a normative system includes the a pattern of interactions among interfering agents “such that possibility that not only new norms are introduced by the it allows the satisfaction of the interests of some agent A”. agents playing a legislative role, but also that ordinary agents These interests can be a shared goal, a value that is good for create new obligations, prohibitions and permissions con- everybody or for most of the members. But the agents at- cerning specific agents. This activity is particularly impor- tribute to the normative system, besides goals, also the abil- tant in applications for e-commerce where it is necessary ity to autonomously enforce the conformity of the agents to to model contracts which introduce new normative relations the norms. To achieve its goal the normative system forms among agents, like the duty to pay a fee for a service (Del- subgoals to consider as a violation the behavior not conform- larocas 2001; Neal et al. 2003). ing to obligations, and to sanction violations.

256 KR 2004 Structure of normative multiagent system • an agent description AD : A → 2X∪B∪M is a total func- The conceptual model of the normative multiagent system is tion that maps each agent to sets of variables (its decision visualized in Figure 1, in which we distinguish the multia- variables), beliefs, desires and goals, but that does not gent system (straight lines) and additions for the normative necessarily assign each variable to at least one agent. For system (dotted lines). Following the usual conventions of, each agent b ∈ A, we write Xb for X ∩AD(b), and Bb for for example, class diagrams in the unified modelling lan- B ∩ AD(b), etc. We write parameters P = X \ ∪b∈AXb. guage (UML), 2 is a concept or set, — and → are asso- • a priority relation ≥: A → 2M × 2M is a function from ciations between concepts, and −−£ is the “is-a” or subset agents to a transitive and reflexive relation on the power- relation. The logical structure of the associations is detailed set of the motivations containing at least the subset rela- in the definitions below. tion. We write ≥b for ≥ (b). The following example illustrates a single agent, who ∈ likes to cultivate crop, does not like to be sanctioned, and R n A who can also build a fence around a field. V P GD  PP Example 1 A = {a}, X = {crop, fence}, P = {s}, ≥ . a  AD P ...... G ) ? Pq ...... D = {d , d }, ≥ = {d } ≥ {d }. There is a single agent, ...... a 1 2 a 2 1 ...... P X B M ...... agent a, who can build a fence and grow crop. Moreover, it ...... MD . H can be sanctioned. It has two desires, one to cultivate crop HH D Hj ? (d ), another one not to be sanctioned (d ). The second de- I Lit Rul 1 2 C sire is more important than the first one. A multiagent system contains, besides an agent set, an organizational structure based on roles and hierarchical con- Figure 1: Conceptual model of a normative system. tainment relations. Moreover, beliefs, desires and goals are abstract concepts which are described by rules (R) built The definition of the agents (A) is inspired by the rule from literals (L). A technical reason to distinguish men- based BOID architecture (Broersen et al. 2002). Beliefs tal attitudes from rules is to facilitate the description of the (B), desires (D) and goals (G) are represented by different priority ordering. To keep the framework simple and to fo- sets representing the epistemic and motivational states of the cus on the subject of this paper, we do not introduce nested agent. We assume that the base language contains boolean mental attitudes, such as beliefs or desires of an agent about variables and logical connectives. The variables (X) are beliefs or desires about another agent. The consequence of either decision variables of an agent, which represent the the absence of such agent profiles is that we can formalize agent’s actions and whose truth value is directly determined only a relatively simple kind of games, as is explained later by it, or parameters (P ), which describe both the state of in this paper. the world and institutional facts, and whose truth value can Definition 2 (Multiagent system) A multiagent system only be determined indirectly. Our terminology is borrowed is a tuple hA, R, ∈, X, B, D, G, AD, MD, ≥i, where from (Lang, van der Torre, & Weydert 2002). Desires (D ) b hA, X, B, D, G, AD, ≥i is an agent set, and: and goals (Gb) express the attitudes of the agent b towards a given state, depending on the context. Agents may share • the roles R are a finite set disjoint from A, X, B, D and decision variables or mental attitudes, though this complica- G. tion is not used in this paper. • the containment relation ∈: R → 2A×A is for each role Given the same set of mental attitudes, agents reason and an irreflexive transitive relation on the set of agents. act differently: when facing a conflict among their motiva- • the set of literals built from X, written as Lit(X), is X ∪ tions, different agents prefer to fulfill different goals and de- {¬x | x ∈ X}, and the set of rules built from X, written sires. We express these agent characteristics by a priority Lit(X) relation (≥) on the mental attitudes which encode, as de- as Rul(X) = 2 × Lit(X), is the set of pairs of a set tailed in Broersen et al. (2002), how the agent resolves its of literals built from X and a literal built from X, written conflicts. The priority relation is defined on the powerset of as {l1, . . . , ln} → l. We also write l1 ∧ ... ∧ ln → l and the motivations such that a wide range of characteristics can when n = 0 we write > → l. Moreover, for x ∈ X we be described, including social agents that take the desires or write ∼x for ¬x and ∼(¬x) for x. goals of other agents into account. The priority relation con- • the mental description MD :(B ∪ M) → Rul(X) is a tains at least the subset-relation which expresses a kind of total function from the sets of beliefs, desires and goals to independence between the motivations. the set of rules built from X. For a set of mental attitudes S ⊆ B ∪ M, we write MD(S) = {MD(s) | s ∈ S}. Definition 1 (Agent set) An agent set is a tuple hA, X, B, D, G, AD, ≥i, where: Our running example illustrates the mental description; the roles and the related hierarchical structure of the agents • the agents A, variables X, agent beliefs B, desires D and are illustrated after the introduction of the normative system. goals G are five finite disjoint sets. We write M = D ∪ G for the motivations defined as the union of the desires and Example 2 (Continued) MD(d1) = > → crop, goals. MD(d2) = > → ¬s.

KR 2004 257 In the description of the normative system, we do not in- In this paper, input and output are respectively a set of troduce norms explicitly, but we represent several concepts literals and a literal. We use a simplified version of in- which are illustrated in the following sections. Institutional put/output logics, since it keeps the formal exposition sim- facts (I) represent legal abstract categories which depend on ple and it is sufficient for our purposes here. In Makinson the beliefs of the normative agent and have no direct coun- and van der Torre’s input/output logics, the input and output terpart in the world. F = P \ I are what Searle calls “brute can be arbitrary propositional formulas, not just sets of liter- facts”: physical facts produced by the actions of the agents. als and literal as we do here. Consequently, in input/output V (x, b) represents the decision of agent n that recognizes x logic there are additional rules for conjunction of outputs as a violation by agent b. The goal distribution GD(b) ⊆ Gn and for weakening outputs. represents the goals of agent n the agent b is responsible for. Definition 4 (Input/output logic) Let a rule set S be a set Definition 3 (Normative system) A normative multiagent of rules {p1 → q1, . . . , pn → qn}, read as ‘if input p1 system, written as NMAS, is a tuple then output q1’, etc., and consider the following proof rules strengthening of the input (SI), disjunction of the input (OR), hA, R, ∈, X, B, D, G, AD, MD, ≥, n, I, V, GDi cumulative transitivity (CT) and Identity (Id) defined as fol- lows: where the tuple hA, R, ∈, X, B, D, G, AD, MD, ≥i is a p → r p ∧ q → r, p ∧ ¬q → r multiagent system, and SI OR p ∧ q → r p → r • the normative agent n ∈ A is an agent. p → q, p ∧ q → r CT Id • the institutional facts I ⊆ P are a subset of the parame- p → r p → p F = P \ I ters, and we write for brute facts. The following output operators are defined as closure op- • the norm description V : Lit(Xa ∪ P ) × A → Xn ∪ P erators on the set S using the rules above. is a function from the literals and the agents to the de-

cision variables of the normative agent together with the out1: SI (simple-minded output) parameters. out2: SI+OR (basic output) Gn • the goal distribution GD : A → 2 is a function from out3: SI+CT (simple-minded reusable output) the agents to the powerset of the goals of the norma- out4: SI+OR+CT (basic reusable output) tive agent, such that if L → l ∈ MD(GD(b)), then l ∈ Lit(Xa ∪ P ). Moreover, the following four throughput operators are de- fined as closure operators on the set S. Our running example illustrates the role hierarchy and the + normative agent. Agent a is a member of the normative sys- • outi : outi+Id (throughput) tem, and the normative agent has the goal that crop is only We write out(S) for any of these output operations and cultivated on property. out+(S) for any of these throughput operations. We also write l ∈ out(S, L) iff L → l ∈ out(S), and l ∈ out+(S, L) Example 3 (Continued) A = {a, n}, R = {member}, iff L → l ∈ out+(S). ∈(member) = {ha, ni}. There is a new agent, agent n, and a role called member. Agent a is a member of normative The following definition of the so-called input/output and system n. output constraints checks whether the derived conditional goals are consistent with the input. Xn = {s}, P = {property}, Dn = Gn = {g1}, MD(g1) = {crop → property}, GD(a) = {g1}. Agent Definition 5 (Makinson & van der Torre 2001) Let S be a n can sanction agent a, because s is no longer a parameter. set of rules, and C a set of literals. S is consistent with It has the goal that crop is build on property only, and it has C, written as cons(S|C), iff there do not exist two con- distributed this goal to agent a. tradictory literals in C ∪ out(S, C). We write cons(S) for cons(S|∅). Before we can define the regulative and constitutive norms, we have to introduce a logic of rules. We use Due to space limitations we have to be brief on technical a simplified version of the input/output logics introduced details with respect to input/output logics, see (Makinson & in (Makinson & van der Torre 2000; 2001). A rule set is a van der Torre 2000; 2001) for the semantics of input/output set of ordered pairs p → q. For each such pair, the body p is logics, further details on its proof theory, alternative con- thought of as an input, representing some condition or situ- straints, and examples. ation, and the head q is thought of as an output, representing In the following sections, we use two input/output log- what the norm tells us to be desirable, obligatory or whatever ics. First, to define whether a desire or goal implies another in that situation. We use input/output logics since they do not one, we use an output operation written as out. Moreover, necessarily satisfy the identity rule. Makinson and van der to define the application of a set of belief rules to a set of + Torre write (p, q) to distinguish input/output rules from con- literals, we use a throughput operation, written as out . We ditionals defined in other logics, to emphasize the property do not specify which output and throughput operations are used, but in the examples we assume the use of out3 and that input/output logic does not necessarily obey the identity + rule. In this paper we do not follow this convention. Fol- out3 . Thus, in this paper we consider out(MD(M)) and + lowing Makinson and van der Torre, we call operations that out (MD(B)). To simplify the notation we write out(M) + satisfy the identity rule throughput operations. and out (B) instead.

258 KR 2004 Regulative norms In this way it is possible that regulative norms refer to in- Regulative norms are based on the notion of conditional stitutional abstractions of the reality rather than to physical obligation with an associated sanction. Obligations are de- facts only. Obligations are illustrated in our running exam- fined in terms of goals of the normative agent n, because ple. regulative norms refer to states of affairs which are currently Example 4 (Continued) false or that can eventually be false. The rules in the def- MD(g2) = {crop, ∼property} → V (∼property, a) inition of obligation are only motivations, and not beliefs, MD(g3) = > → ¬V (∼property, a) because a normative system may not recognize that a viola- MD(g4) = {crop,V (∼property, a)} → s tion counts as such, or that it does not sanction it. Both the MD(g5) = crop → ∼s recognition of the violation and the application of the sanc- {g1, g2, g4} = Gn, Gn ∪ {g3, g5} = Dn, {g1} = GD(a) tion are the result of autonomous decisions of the normative system that is modelled as an agent. NMAS |= Oan(property, s | crop), since: The definition of obligation contains several clauses. The 1. crop → property ∈ out(Dn) ∩ out(GD(a)) first and central clause of our definition defines obligations 2. {crop, ∼property} → V (∼ property, a) ∈ out(Dn) ∩ of agents as goals of the normative agent, following the out(G ) ‘your wish is my command’ metaphor. It says that the obli- n gation is implied by the desires of the normative agent n, 3. > → ¬V (∼property, a) ∈ out(Dn) implied by the goals of agent n, and it has been distributed 4. {crop,V (∼property, a)} → s ∈ out(Dn) ∩ out(Gn) by agent n to the agent. The latter two steps are represented 5. crop → ∼s ∈ out(Dn) by out(GD(a)). 6. crop → ∼s ∈ out(D ) The second and third clause can be read as “the absence a of p is considered as a violation”. The association of obli- One has to be careful when defining multiple obligations gations with violations is inspired by Anderson’s reduction with the same sanction. For example, when both for speed- of deontic logic to alethic logic (Anderson 1958). The third ing and for parking in a no parking street there is a penalty clause says that the agent desires that there are no violations, of 100 euros, then it is implicitly assumed that one can also which is stronger than that it does not desire violations, as be sanctioned 200 euros for violating both obligations at the would be expressed by > → V (∼x, a) 6∈ out(Dn). same time. We do not discuss this problem any further in The fourth and fifth clause relate violations to sanctions. this paper, since it has to do with the formalization of re- The fourth clause says that the normative system is moti- sources which is beyond the scope of this paper. We simply vated not to count behavior as a violation and apply sanc- assume that there is a separate sanction for each obligation. tions as long as their is no violation, because otherwise the Other regulative norms like prohibitions and permissions norm would have no effect. Finally, for the same reason the can be defined in an analogous way. Prohibitions are obliga- last clause says that the agent does not like the sanction. tions concerning negated variables. Definition 6 (Obligation) Let NMAS = Definition 7 (Prohibition) Agent a ∈ A is prohibited to see hA, R, ∈, X, B, D, G, AD, MD, ≥, n, I, V, GDi be to x ∈ Lit(Xa ∪ P ) with sanction s ∈ Lit(Xn ∪ P ) if Y ⊆ a normative multiagent system. Agent a ∈ A is Lit(Xa∪P ) in NMAS, written as NMAS |= Fan(x, s|Y ), obliged to see to x ∈ Lit(X ∪ P ) with sanction a if and only if NMAS |= Oan(∼x, s|Y ) s ∈ Lit(Xn ∪ P ) if Y ⊆ Lit(Xa ∪ P ) in NMAS, written as NMAS |= Oan(x, s|Y ), if and only if: Permissions are defined as exceptions to obligations. A permission to do x is an exception to a prohibition to do x 1. Y → x ∈ out(D ) ∩ out(GD(a)): if Y then agent n n if agent n has the goal that x does not count as a violation desires and has as a goal that x, and this goal has been under some condition. The permission overrides the prohi- distributed to agent a. bition if the goal that something does not count as a violation 2. Y ∪ {∼x} → V (∼x, a) ∈ out(Dn) ∩ out(Gn): if Y and (Y ∧ x → ¬V (x, a)) has higher priority in the ordering on ∼x, then agent n has the goal and the desire V (∼x, a): goal and desire rules ≥n with respect to the goal of a cor- to recognize it as a violation by agent a. responding prohibition that x is considered as a violation 3. > → ¬V (∼x, a) ∈ out(Dn): agent n desires that there (Y 0 ∧ x → V (x, a)): are no violations. Definition 8 (Permission) Agent a ∈ A is permitted by 4. Y ∪ {V (∼x, a)} → s ∈ out(Dn) ∩ out(Gn): if Y and agent n decides V (∼x, a), then agent n desires and has agent n to see to x ∈ Lit(Xa ∪ P ) under condition Y ⊆ as a goal that it sanctions agent a. Lit(Xa ∪ P ), written as NMAS |= Pan(x | Y ), iff 5. Y → ∼s ∈ out(Dn): if Y , then agent n desires not to • Y ∪ {x} → ¬V (x, a) ∈ out(Gn): if Y and x then agent sanction. This desire of the normative system expresses n wants that x is not considered a violation by agent a. that it only sanctions in case of violation. In this paper we do not consider the problem of how the 6. Y → ∼s ∈ out(Da): if Y , then agent a desires ∼s, which normative system is constructed by the sources of norms expresses that it does not like to be sanctioned. such as governments. See for example (Boella & van der Since conditions of obligations are sets of decision vari- Torre 2003f) for a discussion of the problem of the legal ables and parameters, institutional facts can be among them. sources of norms.

KR 2004 259 Constitutive norms Games Constitutive norms introduce new abstract classifications of The games we consider in this paper are based on recursive existing facts and entities, called institutional facts, or they modelling, in which an agent chooses an optimal decision by describe the legal consequences of actions on the normative assuming that other agents make optimal decisions too. For system. According to Searle, institutional facts like mar- example, in the running example agent a makes an optimal riage, money and private property emerge from an indepen- decision from its point of view, assuming that the normative dent ontology of “brute” physical facts through constitutive agent thereafter makes an optimal decision from its point rules of the form “such and such an X counts as Y in context of view. We call the order of the agents making decisions C” where X is any object satisfying certain conditions and Y the protocol of the game. When an agent imagines the de- is a label that qualifies X as being something of an entirely cision of another agent, it must have a profile of the other new sort. Examples of constitutive rules are “X counts as a agent’s mental state. More precisely, if we consider a re- presiding official in a wedding ceremony”, “this bit of paper cursive model with protocol of n agents b1 . . . bn, then each counts as a five euro bill” and “this piece of land counts as agent bi has to have a profile of each sequence of agents somebody’s private property”. bi+1 . . . bn. We formalize the counts-as conditional as a belief rule of the normative agent n. Since the condition x of the belief agent b1 deliberates about optimal decision rule is a variable it can be an action of an agent, a brute → considers optimal decision of agent b2 fact or an institutional fact. So, the counts as relation can be agent b2 deliberates about optimal decision iteratively applied. An additional condition of the counts- → considers optimal decision of agent b3 as conditional is that if it is triggered by an agent, then this agent b3 deliberates about optimal decision agent must participate in the normative system. → considers optimal decision of agent b4 ... Definition 9 (Counts-as relation) Let NMAS= agent b deliberates hA, R, ∈, X, B, D, G, AD, MD, ≥, n, I, V, GDi be a n normative multiagent system. A literal x ∈ Lit(X) Figure 2: Recursive modelling counts-as y ∈ Lit(I) in context C ⊆ Lit(X), NMAS |= counts-as(x, y|C), iff: However, in this paper we have not defined agent profiles. + 1. C ∪ {x} → y ∈ out (Bn): if agent n believes C and x Moreover, it is unrealistic to assume that agents have such then it believes y. detailed agent profiles. We therefore assume in our games 2. If x ∈ Lit(Xb), then there is r ∈ R such that hb, ni in that each agent has the same profile of the other agents. ∈ (r): if the condition is a decision of an agent, then it More precisely, we assume in our games that the mental must play a role in the normative system. state AD(bi) is the profile of agent bi according to the other The constitutive rules are illustrated by our example. agents. There are two main complications: Example 5 (Continued) • If an agent makes two decisions in the recursive mod- elling, then this assumption is unrealistic. For example, Bn = {e1} and MD(e1) = fence → property. Conse- quently we have NMAS |= counts-as(fence, property|>), when the normative agent creates a new norm, considers because we also have ∈ (member) = {ha, ni}. This for- the reaction of an agent, and thereafter may sanction the malizes a society which believes that a field fenced by an agent (Boella & van der Torre 2003f). In this paper we agent who is member of the normative system counts as the exclude this kind of games. fact that the field is a property of that agent. The presence of • If an agent can observe the effects of decisions of other the fence is a physical “brute” fact, while being a property agents, then the assumption is unrealistic. This problem is an institutional fact. A regulative norm which forbids tres- normally does not occur with institutional facts, since they passing refers to the abstract concept of property rather than cannot be observed, but it occurs for brute facts. In this to fenced fields: Obn(trespass, s | property). As the system paper we assume that agents do not observe the effects of evolves, new cases are added to the notion of property by decisions of other agents, the only effects of decisions are means of new constitutive rules, without changing the regu- derived by the agents’ belief rules. lative norms about property. E.g., if a field is inherited, then Moreover, to define games we have to consider how we it is property of the heir: inherit → property ∈ MD(Bn). define the effects of decisions (by applying belief rules), and From a knowledge representation point of view, constitu- how we evaluate the effects of the decisions. For the belief tive norms behave as data abstraction in programming lan- rules any kind of logic can be plugged into our framework, guages: types are gathered in new abstract data types; new to keep our system simple we use a an input/output logic that procedures are defined on the abstract data types to manip- does not deal with exceptions. Consequently, the rules are ulate them. So it is possible to change the implementation monotonic, there are no constraints, and there is no belief of the abstract data type without modifying the programs revision. See (Makinson & van der Torre 2001) for a discus- using those procedures. In our case, it is possible to change sion on these issues in the present setting, and an approach the constitutive norms defining the institutional facts without to introduce them. modifying the regulative norms which refer to those institu- However, the absence of exceptions in our logic of be- tional facts. lief rules introduces the problem that it may be the case that

260 KR 2004 an agent makes a decision, but then agents recursively mod- crop, since it fulfills all the agent’s desires and goals, and elled believe that the earlier decision is not possible, and thereafter the optimal decision of the normative agent is not they therefore cannot define a response to the first decision. to sanction. In this paper we do not further consider this problem, but we Example 7 (Continued) The decision profile simply exclude such games. There are several ways in which h{crop, fence}, {s}i is optimal, but the decision pro- it can formally be forced that such situations do not occur, file h{crop}, {s}i is not. for example by assuming that if an agent recursively models another agent, then the belief rules of the former agent are Modifying the normative system a superset of the belief rules of the latter: since the former agent knows the latter agent’s beliefs, it believes them too. Searle’s analysis of constitutive rules has focused mainly on Clearly this property only holds for a particular kind of be- the attribution of a new functional status to entities, as, for liefs (of the type usually identified with knowledge), but this examples, weddings, money, property. Searle’s idea is that is exactly the case of constitutive rules we are discussing. constitutive rules “create the possibility or define that activ- Constitutive rules do not concern reality, but they are estab- ity”. However, we believe that the role of constitutive rules lished by the normative system, so they cannot be wrong. is not limited to the creation of an activity and the construc- A decision profile for a decision problem is a sequence tion of new abstract categories. Constitutive norms specify of decisions, one for each agent. We thus do not consider both the behavior of a system and the evolution of the sys- simultaneous decisions. Agents evaluate states of affairs ac- tem: the normative system n itself specifies by means of its cording to which motivational attitudes remain unfulfilled: belief rules how its beliefs, desires and goals can be changed, their body is part of the expected effects of the decision, but who can change them, and the limits of the possible changes their head is not. depending on the role played by an agent. In this section we only define actions that create new beliefs, desires and goals; Definition 10 Let NMAS be a normative multiagent system. actions that modify or delete mental attitudes can be defined analogously. Technically, we have defined the beliefs, de- • A protocol is a sequence of distinct agents hb1, . . . bni.A decision problem hnmas, protocoli consists of a norma- sires and goals as part of the structure of the normative mul- tive multiagent system and a protocol. tiagent system. To change these mental attitudes, we have to • A decision profile for a protocol is a sequence introduce a more general structure that also contains actions hδ , . . . , δ i such that cons(B | δ) for i = 1 . . . n that change the normative structure. b1 bn bi Despite their name, create actions are modelled as insti- and δ = ∪i=1...nδbi . We also write ∆ for the set of all decisions profiles. tutional facts, since they belong to social reality. As insti- tutional facts are parameters, they cannot be directly con- • Agent b prefers a state of affairs S1 ⊆ Lit(X) to an- trolled by any agent. But the normative system itself, by other one S2 ⊆ Lit(X) iff U(S2, b) >b U(S1, b), where means of constitutive rules, assigns the power of executing U(S, b) = create actions to agents playing roles in it: agents execute ac- tions which count as the execution of creation actions, and, {m ∈ Mb | MD(m) = L → l, L ⊆ S and l 6∈ S} thus, they change the normative system following the agent’s The protocols are illustrated by our running example. decision. Example 6 (Continued) Assume the protocol ha, ni, in Definition 12 An extended normative multiagent system which first agent a takes a decision, and thereafter agent ENMAS is a tuple n reacts on it. The decision profile h{crop}, {s}i repre- hA, R, ∈, X, B, D, G, AD, MD, ≥, n, I, V, GD, Ci sents that first agent a cultivates crop, and thereafter agent n sanctions agent a. that consists of a normative multiagent system The games the agents can play in this extended game the- hA, R, ∈, X, B, D, G, AD, MD, ≥, n, I, V, GDi ory are based on a recursive definition. Due to the fact that together with a set of actions that can change the normative the protocol is finite, the definition is well founded. system:

Definition 11 A decision profile δ1 dominates deci- • the belief create actions CBn : Rul(X) → I, is a map- sion profile δ2 for agent bi if they have the same ping from belief rules to institutional fact, where CBn (r) set of decisions δb1 . . . δbi−1, and for every decision stands for the creation of m ∈ Bn, together with the up- 0 0 profile agent δ1 and δ2 that coincide with δ1 and date of MD such that r = MD(m). δ2 for b1 to bi and that are optimal for agent • the desire create actions CD : Rul(X) → I, where + 0 + 0 n bi+1 . . . bn, bi prefers out (Bbi , δ1) to out (Bbi , δ2), i.e., CD (r) stands for the creation of m ∈ Dn, together with + 0 + 0 n U(out (Bbi , δ2), bi) >bi U(out (Bbi , δ1), bi). the update of MD such that r = MD(m). A decision profile is optimal for agent b if it is not dom- i • the goal create actions CG : Rul(X) → I, where inated by another decision profile, and it is optimal for all n CGN (r) stands for the creation of m ∈ Gn, together with agents bj with j > i. A decision profile is optimal if it is the update of MD such that r = MD(m). optimal for agent b1. • the goal create actions CGD : Rul(X) × A → I, where

The games are introduced in our running example. The CGN (r, b) stands for the update of GD such that m ∈ optimal decision for agent a is to build a fence and cultivate GD(b) and r = MD(m).

KR 2004 261 Games in ENMAS are defined analogously to games in counts as being the owner counts-as(fence, property | >), NMAS, with the only exception that all references to the as discussed in Example 5. agents’ mental states are made to the normative multiagent In the normative system ENMAS1 considered in this sec- system that results after the normative multiagent system of tion, having a property creates an obligation that if the owner the decision problem has been updated with the decisions. a of the field grows crop, the agent must be authorized 0 (autho) by the land registry Oan(autho, s | crop). For Definition 13 Let ENMAS be an extended normative multi- example, the registry has to keep track of the crops for agent system. A protocol is defined in the same way as for taxation purposes. To be registered it is sufficient to ap- NMAS. Moreover: ply (apply), since the application counts as being autho- • A decision profile for a protocol is a sequence rized: counts-as(apply, autho | >). This means that Bn = hδ , . . . , δ i such that cons(B0 | δ) for i = 1 . . . n and {e , e } and MD(e ) = fence → property, MD(e ) = b1 bn bi 1 2 1 2 δ = ∪ δ , where B0 contains the beliefs from B apply → autho, and ∈ (member) = {ha, ni}, see Exam- i=1...n bi bi bi together with the beliefs that occur in the create actions ple 3. + The normative agent n specifies how the normative sys- in out (Bbi , δ). tem can be changed. In the example, it specifies how a new • Agent b prefers a state of affairs S1 ⊆ Lit(X) to an- 0 obligation Oan(autho, s | crop) is created by means of cre- other one S2 ⊆ Lit(X) iff U(S2, b) >b U(S1, b), where U(S, b) = ate actions. Since this obligations is defined in terms of the goals and desires of the normative agent, the create actions 0 {m ∈ Mb | MD(m) = L → l, L ⊆ S and l 6∈ S} must add these goals to the normative system. Note that the desire of agent a not to be sanctioned (crop → ¬s0) must 0 where Mb contains the motivations from Mb together be already true in ENMAS1 for the obligation to be defined with the motivations that occur in the create actions in correctly. Moreover, to create the obligation ENMAS1 must + out (Bb, δ). specify the following create actions:

The dominance relation and optimal decisions are defined in c1 = CGn (crop → autho) the same way as for NMAS. c2 = CDn (crop → autho) c = C (crop ∧ ¬autho → V (¬autho, a)) Since regulative norms are defined in terms of goals of 3 Gn c = C (crop ∧ ¬autho → V (¬autho, a)) the normative agent and constitutive norms in terms of its 4 Dn c = C (crop → ¬V (¬autho, a)) beliefs, by means of creation actions it is possible to create 5 Dn c = C (crop ∧ V (¬autho, a) → s0) new regulative and constitutive rules. The following exam- 6 Gn c = C (crop ∧ V (¬autho, a) → s0) ple illustrates the modification of the normative system by 7 Dn c = C (crop → ¬s0) creating a constitutive norm. 8 Dn c9 = CGD(crop → autho, a) Example 8 An example of a constitutive norm for cre- Since these create actions are parameters in I, they are not ating new constitutive rules in a normative agent is directly controlled by the normative agent: it cannot perform counts-as(y, CBn (x → p) | >) where y ∈ Xb is an ac- them by itself. Rather the normative agent specifies who is tion of agent b ∈ A, x ∈ Xc is an action of agent c ∈ A and able to do these changes by counts-as rules: it specifies who p ∈ I is an institutional fact. In the normative system agent is able to execute those create actions by means of its deci- b must play, e.g., the role of legislator, who has the power to sion variables. In our example, we have that property counts create norms; agent c is in a role that has the power to make as the creation of the obligation: counts-as(property, ci | >) the institutional fact p true by doing x. where 1 ≤ i ≤ 9. Let us assume that these constitu- In the following section, we illustrate the modification of tive norms are based on ei ∈ Bn for 1 ≤ i ≤ 9 with the normative system by the creation of a regulative norm MD(ei) = property → ci. in our running example. As discussed in (Boella & van der We consider now for the decision problem Torre 2003e) creating regulative norms is more complicated, hENMAS1, ha, nii the decision profile hENMAS1, hδa, δnii because it does not make sense to create all the items of Def- The possible decisions of agent a are doing nothing, inition 6. Some of the conditions of obligation are precon- to fence the field, to cultivate crop, to apply for an au- ditions for norm creation, and others are postconditions of thorization and all the possible combinations of these such an action. actions. The possible decisions of agent n are doing nothing, considering some violation, sanctioning and all the possible combinations of these actions. Agent a has Norm creation in the running example to take a decision to fulfill its desire of cultivating crop In this section we conclude our example showing the other (MD(d1) = > → crop and d1 = MD(Da)). At the same role played by constitutive rules: not only as abstractions for time, it does not desire to be sanctioned with s and s0 if an institutional classification of reality, but also as specifica- it grows crop (MD(d2) = > → ¬s and d2 ∈ Da, and 0 tions of the possible changes to the normative agent. Thus MD(d3) = > → ¬s and d3 ∈ Da) by the normative far, we have considered a game played by an agent a who, agent n, and the latter desires are stronger than the former in order to cultivate some crop (crop), fences (fence) a field (≥a⊇ {{d2} ≥ {d1}, {d3} ≥ {d1}}). to show that it is its property (property), as requested by the To achieve its desire to grow crop, it has also to take into obligation Oan(property, s | crop) in Example 4. Fencing account the consequences of this action. First, it knows that

262 KR 2004 the normative agent will consider crop without property a vi- a few beliefs, or a few goals and many beliefs. Interest- olation (crop∧¬property → V (¬property, a) ∈ out(Gn), ingly, a similar trade-off can be found in knowledge-based from Example 4). For this reason it has also to fence the systems. Traditional planning systems are based on a sin- field. Second, it knows that the existence of a property gle goal, but modern agent systems typically contain many makes the normative agent a create a new obligation: that goals and a goal selection mechanism. Other inspirations for it gets an authorization to grow crop. Hence, agent a has not this trade-off can be found in legal theory. Traditionally, law only to consider the effects of its behavior, but also to con- scholars like Hart (Hart 1961) distinguish between primary sider that the second agent in the protocol, when it will act, laws, whose purpose is to direct the behavior of citizens, could be in a different normative system, due to the effects and secondary laws, which, among other functions, serve to of agent a’s actions. the maintenance and dynamic management of the normative The optimal decision of agent a has to take into account system. These rules form a “subsystem of rules for change” which is the reaction of agent n. Hence, agent a has to (Biagioli 1997): rules which have juridical effects and which consider not the state immediately following its decision δa, are instrumental to the primary system, in that they regulate rather the final state. So, even if the decision {crop} satisfies the regulation (e.g., art. 2 of Italian Civil Code: “the creation all its desire in state > → crop, it is not the optimal decision, of laws [...] is regulated by constitutional laws” Cost. 70). since in the subsequent state the effect includes the sanc- This subsystem, according to Hart, does not include only the tions s and s0 as a result of the normative agent’s decision rules of change which specify how new laws are introduced that the obligation Oan(autho, s | crop) has been violated. or old ones removed, but it also includes rules about “pow- The optimal decision is, instead, {fence, crop, apply}: agent ers for the private citizen”. These rules are at the basis of a knows that fencing counts as property and applying for civil code and allow testaments and contracts; for Hart they authorization counts as being authorized for agent n, hence, allow the exercise of limited legislative powers by the citi- both the existing obligation and newly created obligation of zens. These rules do not create or remove general laws but ENMAS2 is not violated. they introduce and remove individual obligations and per- enmas1 = missions: e.g., in the Italian Civil Code art. 1173 (sources hA, R ∈, X, B, D, G, AD, MD, ≥, n, I, V, GD, Ci of obligations) specifies that obligations are created by con- A = {a, n},R = {member},Xa = {fence, crop, apply}, tracts (a contract being an agreement among two or more Ba = Bn,Ga = ∅,Da = {d1, d2, d3} parties to regulate a juridical relationship about valuables by X = {V (¬property, a), s, V (¬autho, a), s0} art. 1321). n The normative agent metaphor allows also using our Bn = {e1, e2, e1, e1,..., e9},Gn = {g1, g2, g4}, D = {g , . . . , g }, GD(a) = {g } framework to support legislative drafting. One of the issues n 1 5 1 in writing laws is that also regulative norms can be expressed I = {property, autho, c1, . . . , c9} by means of assertions, i.e. in the same way as constitutive δa = {fence, crop, apply} norms, rather than by means of deontic statements concern- + F1 = out (Ba, δa = {fence, crop, apply}) = ing what is obligatory or permitted: e.g. “murder is punished {fence, crop, apply, property, autho, c1, . . . , c9} with ten years of jail.” This sentence does not describe a con- F1 ∩ I = {property, autho, c1, . . . , c9} stitutive norm. It is a description of the fact that murder is prohibited and murderers are sanctioned enmas2 = hA, R, ∈,X,B,D0,G0, AD, MD, ≥, n, I, V, GD0,Ci The normative system as agent metaphor provides us with 0 a criterion for distinguishing the two types of norms. If an Bn = {e1, e2, e1, e1,..., e9}, 0 0 assertion refers to actions performed by the normative agent, Gn = {g1, g2, g4, g6, g7, g9},Dn = {g1, . . . , g10}, GD0(a) = {g , g } the norm is a regulative one: it is a description of the be- 1 6 havior of the normative agent, and, since agent behavior is MD(g6) = crop → autho described in terms goals, what the sentence really describes MD(g7) = crop ∧ ¬autho → V (¬autho, a) is the goal of the agent: in case of murder the normative MD(g8) = > → ¬V (¬autho, a) 0 agent has the goal of sanctioning the murderer with a ten MD(g9) = crop ∧ V (¬autho, a) → s 0 year term. This is the goal contained in the fourth clause of MD(g10) = crop → ¬s the definition of obligation. δn = ∅ In contrast, assertions describing facts are constitutive + 0 F2 = out (Bn, δa = {fence, crop, apply} ∪ δn) = norms. For example the sentence “a contract is an agree- {fence, crop, apply, property, autho, c1, . . . , c9} ment among two or more parties” (art. 1321 of Italian Civil Code) is not a regulative norm in that it does not describe U(F1, a) = ∅ an action of the normative agent: it describes the belief of U(F2, n) = ∅ the normative agent that an agreement (a brute fact among agents) is considered as an institutional fact which does not Regulative or constitutive norms? exist without the normative system: a contract. One relevant problem in encoding norms is whether to use Analogously for constitutive norms concerning norm many regulative norms and a few constitutive norms, or a changing. For example “obligations derive by contract” few regulative norms and many constitutive norms. In our (ibid. art. 1372): the sentence does not refer to any action framework, the question is whether to use many goals and of the normative system. Rather it specifies that the goals

KR 2004 263 of the normative agents which define an obligation are mod- the normative system allows agents which have a role in the ified by a contract. I.e., the sentence describes the beliefs normative system to change its beliefs and motivations. of the normative agent about the consequences of a contract What distinguishes our approach from other models of made among agents. counts-as relations is that we can connect goals, and obliga- tions defined as goals, to institutional facts inside the overall Related work frame of the attribution of the status of agent to the norma- This work is part of a wider research on normative reason- tive system: institutional facts are beliefs of the normative ing of autonomous agents based on the attribution of mental agent as any other belief. Here, we take full advantage of attitudes to the normative system (Boella & van der Torre the metaphor, as also Tuomela (1995) argues: 2003a). In (Boella & van der Torre 2004) we consider the The notions of goal, belief, and action are linked in the social delegation of goals to the normative system, in (Boella case of a group to approximately the same degree as in & van der Torre 2003f) we introduce permissions, in (Boella the individual case. In the latter case their interconnec- & van der Torre 2003d) the definition of the role of a de- tion is well established; given that the person-analogy fender which fulfills the task of identifying violations and applies to groups [. . . ], these notions apply to groups as sanctioning them on behalf of the normative system. In well. (Boella & van der Torre 2003c) we apply the framework to the regulation of virtual communities of agents based on The ontology presented by (Gangemi, Sagri, & Tiscornia the grid infrastructure. Finally, in (Boella & van der Torre 2003) adopts a perspective similar to our view of constitu- 2003b) we explore how to formalize the trias politica using tive rules as beliefs: institutional facts are considered human facts depending on consciousness (and not on will) and are the standard BDICTL logic (Rao & Georgeff 1998) for agent verification. legally constituted by (satisfying) constitutive rules. Other related work is (Lopez y Lopez, Luck, & d’Inverno 2002). They propose a model of obligation compliance of Conclusions agents that emphasizes their autonomy. They classify dif- In this paper we introduce a formal framework for norma- ferent motivations for which agents decide to stick to obli- tive multiagent systems including regulative and constitu- gations or to violate them. In (Lopez y Lopez, Luck, & tive norms. We model constitutive and regulative norms as d’Inverno 2001), the same authors stress the importance of conditional rules representing, respectively, the beliefs and having a model of the other agents in order to reason about goals of the normative system, i.e., by attributing mental at- the dependance relations with them. But, while, as in our titudes to normative systems. work, in (Lopez y Lopez, Luck, & d’Inverno 2002) sanctions We show how to reason about modifications of the nor- are associated with the definition of an obligation, there is no mative system in this framework. Constitutive norms play agent in charge of sanctioning violations since “the applica- not only the role of creating new abstract categories which tion of punishments and rewards is taken for granted”. So compose the institutional reality, but they also specify how they are not able to model unpunished violations when the the normative system evolves over time by introducing or addressee of the obligation exploits the recursive modelling removing norms and obligations. Roles are used to specify of the normative system’s decisions. the powers of agents to create institutional facts or to modify Also (Dignum et al. 2000) are interested in integrating the norms and obligations of the normative system. norms and obligations in a BDI approach to multiagent sys- We also show how to play games in this normative multia- tems. We focus on obligations since in their model obliga- gent system, including violations of obligations. The games tions and not norms are intended as associated with an ex- are based on recursive modelling, concerning, e.g., the de- plicit agent which is responsible for enforcing the penalty. cision to fulfill or violate a norm, and the decision of which In doing so they adopted van der Torre and Tan’s approach norms to create in order to achieve the desired social order. (1999) on preference based dyadic deontic logic built on An issue for further research is a formal analysis of the Kripke models. On the contrary our approach is entirely balance between regulative and constitutive norms. We based on input/output logic. would like to derive guidelines in which circumstances a As in (Castelfranchi et al. 2000) we consider norms as normative system should create mainly regulative norms, “mental objects entering the mental processing” which in- and in which circumstances it should create mainly consti- teract with beliefs, goals and decisions. Moreover, they also tutive norms. Moreover, we would like to formally char- claim that norms cannot be hard constraints, but they just acterize the set of normative systems that can be generated can influence an agent to a certain behavior if they directly by a normative system, given the rules it contains to modify or indirectly satisfy some of his goals. itself. In Artificial Intelligence, the modelling of counts-as rela- Another issue for future research is an analysis of the tions has been introduced by Jones and Sergot (1996). We properties of the counts-as relation based on the properties depart from their model in that constitutive norms are not of the belief rules. For example, an ordering on beliefs can modelled as operative constraints of an institution but as be- be used to achieve non-monotonicity of beliefs and thus of liefs of the normative agent. We distinguish also from the counts-as, as required by (Gelati et al. 2002). It is possible subsequent work of Sadighi Firozabadi and Sergot (1999) that under some circumstances a certain state of affairs does who propose a model for power and permission in security not count as something else. For example, it is not possible policies; in our work, powers are defined in terms of how to fence a public garden to make it ones property.

264 KR 2004 Moreover, we like to study the application of our frame- Gangemi, A.; Sagri, M.; and Tiscornia, D. 2003. Metadata work in legal reasoning. Constitutive rules are at the basis for content description in legal information. In Procs. of of legal institutions: “systems of [regulative and constitu- LegOnt Workshop on Legal Ontologies. tive] rules that provide frameworks for social action within Gelati, J.; Governatori, G.; Rotolo, N.; and Sartor, G. 2002. larger rule-governed settings” (Ruiter 1997). Declarative power, representation, and mandate. A formal analysis. In Procs. of JURIX 02. IOS press. References Hart, H. L. A. 1961. The Concept of Law. Oxford: Claren- Anderson, A. R. 1958. The logic of norms. Logic et anal- don Press. yse 2. Jones, A., and Carmo, J. 2001. Deontic logic and contrary- Biagioli, C. 1997. Towards A legal rules functional micro- to-duties. In Gabbay, D., ed., Handbook of Philosophical ontology. In Procs. of 1st LegOnt Workshop on Legal On- Logic. Kluwer. 203–279. tologies. Jones, A., and Sergot, M. 1996. A formal characterisation Boella, G., and Lesmo, L. 2002. A game theoretic ap- of institutionalised power. Journal of IGPL 3:427–443. proach to norms. Cognitive Science Quarterly 2(3-4):492– Lang, J.; van der Torre, L.; and Weydert, E. 2002. Utilitar- 512. ian desires. Autonomous Agents and Multi-agent Systems Boella, G., and van der Torre, L. 2003a. Attributing mental 329–363. attitudes to normative systems. In Procs. of AAMAS’03, Lopez y Lopez, F.; Luck, M.; and d’Inverno, M. 2001. 942–943. ACM Press. A framework for norm-based inter-agent dependence. In Boella, G., and van der Torre, L. 2003b. Game specifica- Procs. of 13rd Mexican International Conference on Com- tion in the trias politica. In Procs. of BNAIC’03. puter Science, 31–40. Boella, G., and van der Torre, L. 2003c. Local policies for Lopez y Lopez, F.; Luck, M.; and d’Inverno, M. 2002. the control of virtual communities. In Procs. of IEEE/WIC Contraining autonomy through norms. In Procs. of AA- Web Intelligence Conference, 161–167. IEEE Press. MAS’02, 674–681. ACM press. Boella, G., and van der Torre, L. 2003d. Norm gov- Makinson, D., and van der Torre, L. 2000. Input-output erned multiagent systems: The delegation of control to au- logics. Journal of Philosophical Logic 29:383–408. tonomous agents. In Procs. of IEEE/WIC Intelligent Agent Makinson, D., and van der Torre, L. 2001. Constraints Technology Conference, 329– 335. IEEE Press. for input-output logics. Journal of Philosophical Logic Boella, G., and van der Torre, L. 2003e. Permissions and 30(2):155–185. obligations in hierarchical normative systems. In Procs. of Neal, S.; Cole, J.; Linington, P. F.; Milosevic, Z.; Gibson, ICAIL’03, 81–82. ACM Press. S.; and Kulkarni, S. 2003. Identifying requirements for Boella, G., and van der Torre, L. 2003f. Rational norm cre- business contract language: a monitoring perspective. In ation: Attributing mental attitudes to normative systems, Procs. of EDOC’03, 50–62. IEEE press. part 2. In Proceedings of ICAIL’03, 109–118. ACM Press. Rao, A. S., and Georgeff, M. P. 1998. Decision proce- Boella, G., and van der Torre, L. 2004. ∆: The social dures for BDI logics. Journal of Logic and Computation delegation cycle. In Procs. of DEON’04 Workshop. 8(3):293–343. Breuker, J.; Valente, A.; and Winkels, R. 1997. Legal on- Ruiter, D. 1997. A basic classification of legal institutions. tologes: A functional view. In Procs. of 1st LegOnt Work- Ratio Juris 10(4):357–371. shop on Legal Ontologies, 23–36. Sadighi Firozabadi, B., and Sergot, M. 1999. Power Broersen, J.; Dastani, M.; Hulstijn, J.; and van der Torre, L. and permission in security systems. In Security Protocols. 2002. Goal generation in the BOID architecture. Cognitive Springer Verlag. 48–53. Science Quarterly 2(3-4):428–447. Searle, J. 1969. Speech Acts: an Essay in the Philosophy Castelfranchi, C.; Dignum, F.; Jonker, C. M.; and Treur, of Language. Cambridge, England: Cambridge University J. 2000. Deliberate normative agents: Principles and ar- Press. chitecture. In Intelligent Agents VI (ATAL-99). Springer- Searle, J. 1995. The Construction of Social Reality. New Verlag. 364–378. York: The Free Press. Castelfranchi, C. 2000. Engineering social order. In Procs. Tuomela, R. 1995. The Importance of Us: A Philosoph- of ESAW’00, 1–18. Springer Verlag. ical Study of Basic Social Notions. Standford: Stanford Dellarocas, C. 2001. Negotiated shared context and so- University Press. cial control in open multi-agent systems. In Conte, R., and van der Torre, L., and Tan, Y. 1999. Contrary-to-duty rea- Dellarocas, C., eds., Social Order in MAS. Kluwer. soning with preference-based dyadic obligations. Annals Dennett, D. C. 1987. The Intentional Stance. Cambridge, of Mathematics and Artificial Intelligence 27:49–78. MA: The MIT Press. Dignum, F.; Morley, D.; Sonenberg, E. A.; and Cavedon, L. 2000. Towards socially sophisticated BDI agents. In Procs. of ICMAS’00, 111–118. IEEE press.

KR 2004 265