Reasoning About Others: Representing and Processing Infinite Belief Hierarchies
Sviatoslav Brainov and Tuomas Sandholm Department of Computer Science Washington University St. Louis, MO 63130 {brainov, sandholm}@cs.wustl.edu
Abstract to represent them. The second issue that deserves In this paper we focus on the problem of how infinite consideration is the feasibility of decision making belief hierarchies can be represented and reasoned based on infinitely nested beliefs. with in a computationally tractable way. When Finite hierarchies of beliefs have been studied by modeling nested beliefs one usually deals with two Gmytrasiewicz, Durfee and Vidal [11,12,13, 20]. The types of infinity: infinity of beliefs on every level of main advantage of their recursive modeling method is reflection and infinity of levels. In this work we that a solution can always be derived. The recursive assume that beliefs are finite at every level, while the modeling method is based on the assumption that number of levels may still be infinite. We propose a once an agent has run out of information his belief method for reducing the infinite regress of beliefs to a hierarchy can be cut at the point where there is no finite structure. We identify the class of infinite belief sufficient information. At the point of cutting, trees that allow finite representation. We propose a absence of information is represented by a uniform method for deciding on an action based on this distribution over the space of all possible states of the presentation. We apply the method to the analysis of world. The absence of information with which to auctions. We prove that if the agents’ prior beliefs model other agents implies that belief hierarchies are are not common knowledge, the revenue equivalence potentially finite. In our work we take a different theorem ceases to hold. That is, different auctions approach. We suppose that every belief hierarchy is yield different expected revenue. Our method can be potentially infinite and we consider the problem of used to design better auction protocols, given the how such a hierarchy can be represented using a finite participants’ belief structures. structure and manipulated in a computationally tractable way. We show that some infinite belief trees 1. Introduction allow finite representation in the form of pointed accessible graphs. Reasoning about others and interactive knowledge The most typical and the most studied example of have been the subject of continuous interest in infinite belief hierarchies is common knowledge multiagent systems [11,12,13,20], artificial intelligence [6,7,8] and game theory [1,3,15]. In [4,7,10]. Common knowledge means that everyone multiagent interaction, where an agent’s action knows that everyone knows that… Common interferes with other agents’ actions, hierarchies of knowledge, however, is a very special case of a belief beliefs arise in an essential way. Usually an agent’s hierarchy, namely, a hierarchy that consists of infinite optimal decision depends on what he believes the repetition of some event. Such a hierarchy can be other agents will do, which in turn depends on what reduced to a finite representation if we treat all the he believes the other agents believe about him, and so repetitions in the hierarchy as identical. In this paper on. An infinite regress of this kind gives rise to we extend this idea to infinite belief hierarchies with several important issues. The first issue concerns the more complex structure. methods for representing infinitely nested beliefs. In The research presented in this paper is closely order to maintain and update such beliefs, agents related to the work in game theory devoted to games need some finite and computationally tractable way with incomplete information [9]. In our work, the epistemic state of each agent is modeled as an infinite hierarchy of beliefs. Harsanyi suggested that each 1 shows a belief tree that represents first-order beliefs hierarchy of beliefs could be summarized by the of agent i. notion of agent’s type [9]. Later Mertens and Zamir Second order beliefs of agent i include his beliefs proved that the space of all possible types is closed in about the true state of the environment and his beliefs the sense that it is large enough to include even about agent j’s beliefs about the true state of the higher-order beliefs about itself [15]. Brandenburger environment. A second-order belief tree for agent i is has shown that if agents’ beliefs are coherent the represented in Figure 2. space of all possible types is closed [4]. i’s level of reflection The paper is organized as follows. In Section 2 we propose the notion of balanced strategy labeling. In p1 p Section 3 we show how regular belief trees can be n represented with finite trees. In section 4 we propose j’s level of reflection t1 ………………………….…. tn a graph representation for infinite belief trees. We σ1 σn apply our approach to the analysis of auctions in q1 q2 qn r1 r2 rn Section 5. Finally, the paper concludes by summarizing the results and providing directions for t1 t2 ……. tn t1 t2 ……. tn future research. 2. Infinite belief hierarchies Figure 2. A second-order belief tree σ Let 1 denote the first-order beliefs of agent j that In every multiagent interaction, an agent faces two the true state of the environment is t1 with probability types of uncertainty: basic and belief uncertainty. q1 and t2 with probability q2, and so on. Similarly, let Basic uncertainty relates to the elements of the σ n denote agent j’s beliefs that the true state of the physical environment, which are uncertain to agents. environment is t with probability r and t with We model basic uncertainty by a finite set T, 1 1 2 probability r2, and so on. According to Figure 2, agent T={t1,t2,…,tn}, including all uncertain elements of the i assigns probability p1 to the event that the true state physical environment. We represent agents’ basic is t and the first order beliefs of agent j are σ . At the beliefs as a subjective probability distributions on T. 1 1 same time, agent i believes that with probability pn While basic uncertainty deals with the elements of σ the true state is tn and agent j’s beliefs are n. the physical environment, belief uncertainty relates to Therefore, each level of a belief tree is associated other agents’ beliefs. That is, belief uncertainty with one of the agents. Levels alternate: the first level includes an agent’s beliefs about other agents’ corresponds to agent i, the second to agent j, and so beliefs, his beliefs about other agents’ beliefs about an. This alternation of levels produces alternation of other agents’ beliefs, and so on. beliefs: agent i believes that agent j believes that agent i believes, and so on. Agent i In this paper we assume that all belief trees are locally finite. That is, the branching factor of every
p1 p2 pn node is finite. At first sight this may seem to be a substantial constraint. Since the cardinality of the space of all first-order beliefs is a continuum, one t1 t2 ….………. tn might expect that trees are not an adequate representation of an agent’s beliefs. In interactive Figure 1. A first-order belief tree epistemology and game theory [1,3,15] recursive Suppose that the agents under consideration are beliefs are usually represented as nested probability agent i and agent j. First order beliefs of agent i can distributions. Such representations, although be represented by a discrete probability distribution theoretically elegant, are not often tractable from a σ σ , =(p1,p2,…,pn). That is, agent i believes that the computational point of view. In this paper we focus true state of the environment is t1 with probability p1, on the problem of how beliefs can be represented in a t2 with probability p2, and so on. First order beliefs of computationally tractable way. In recursive modeling, agent i can be represented by a belief tree. The nodes one usually deals with two types of infinity: infinity of the tree are labeled with the elements of T and arcs of beliefs on every level of reflection and infinity of are labeled with probabilities. For every node, the levels. In this work we assume that beliefs are finite sum of the probabilities on outgoing arcs is 1. Figure at every level, while the number of levels can be infinite. We propose a method for reducing the the following notation: Γ ag infinite regress of beliefs to a finite structure. The i (v) returns the name of the agent whose ∈ Γ idea behind our approach is that some infinite belief beliefs are represented at node v, v N( i). Γ ag Γ → hierarchies display repetitive patterns and That is, i : N( i) {i,j}. For example, Γ ag regularities. This makes it possible to “merge” some i (v) returns i for every node that is Γ parts of an infinite hierarchy and to “cut” other parts. located at an odd-numbered level of i. Previous research of Gmytrasiewicz and Durfee Γ succ i (v) denotes the set of the successors of a node [11,20] assumes that both the number of levels of ∈ Γ v, v N( i). beliefs and the number of beliefs at each level are Γ prob i (v) denotes the probability distribution finite. ∈ Γ assigned to node v, v N( i). According to the game-theoretic tradition [1], the Γ With each node, v, of the belief tree, i, we assign belief structure of an agent can be represented by a Γ ag a strategy that tells what agent i (v) will do at that hierarchy of beliefs. node. For a node corresponding to agent i’s beliefs, Definition 1. A belief hierarchy, B ∞, of agent i is an the assigned strategy belongs to agent i’s strategy set i S . Similarly, for a node corresponding to agent j’s infinite sequence of finite belief trees. That is, i ∞ 1 2 3 k beliefs the strategy belongs to S . Formally, we denote B =(b ,b ,b ,…), where b represents agent i’s k- j i i i i i a strategy labeling by φ:N(Γ )→S ∪S . Every strategy order beliefs. i i j Γ labeling of i represents agent i’s beliefs about the A belief hierarchy is a complete enumeration of all strategies of both agents. It is clear that agent i’s orders of belief. Since a rational agent can hold strategy depends on his believes about agent j’s beliefs of an arbitrary order, every belief hierarchy is strategy. Agent j’s strategy depends on agent j’s inherently infinite. beliefs about agent i’s strategy, and so on. That is, a Since every k-order belief tree implicitly strategy labeling is a solution to the problem of determines k-1-order beliefs, it is natural to assume Γ strategy choice for agent i given his beliefs i. that all trees in a belief hierarchy are consistent. By For every infinite belief tree, Γ , there exists an k-1 i this we mean that bi represents the k-1-order beliefs infinite number of strategy labelings. However, only a k k-1 determined by bi . In other words, bi is a subtree of few of them (if any) meet the Bayesian rationality k bi . Belief consistency says that the different levels of requirement, i.e., that a strategy at each node, v, has beliefs do not contradict one another. to be a best response to the other agents’ strategies at The principle of consistency allows us to reduce ∞ the successors of v. That is, if at level m of his the infinite sequence of belief trees, Bi , to a single ∞ reflection, agent i believes that at level m+1 agent j is infinite tree. Since in Bi every tree carries all the going to use some (possibly mixed) strategy, then the information presented in the preceding trees, the strategy of agent i at level m should be a best sequence of belief trees is monotonically increasing response to agent j’s strategy given the probability Γ ∞ and we can take the limit. Let (Bi ) denote the limit distribution at level m+1. The following definition ∞ tree that corresponds to the sequence Bi . For the sake introduces the class of balanced strategy labelings. A Γ ∞ Γ of simplicity we will denote (Bi ) by i whenever strategy labeling, φ, is balanced if the strategy this is not a source of confusion. associated with each node is a best response to the Γ Infinity of i creates several problems. First, it is strategies associated with the successor nodes, given problematic how an agent can build, update and the probabilities assigned to the successors. manipulate such a structure. Second, how can an Formally, agent predict the behavior of other agents if his own φ beliefs run to infinity? Definition 2. A strategy labeling is balanced iff for ∈ Γ φ In this paper we will show that some infinite belief every node v, v N( i), (v) is a best response to the φ Γ succ Γ prob trees can be represented by finite belief trees or by mixture of strategies [ ( i (v)), i (v)]. finite graphs. By this, we provide a uniform way to Another way to look at a balanced strategy deal with uncertainty that does not depend on the φ Γ labeling of i is to see it as the limit of balanced depth of agents’ beliefs. strategy labelings for finite trees of the infinite Γ ∞ ∞ 1 2 3 Let Sk be the strategy set of agent k, k=i,j. i hierarchy Bi , Bi =(bi ,bi ,bi ,…). That is, denotes a belief tree (infinite or finite) of agent i. We φ k φ Γ Γ Γ Γ lim i = (1) represent i as a pair (N( i),A( i)), where N( i) is the Γ k→∞ set of nodes and A( i) is the set of arcs. We also use φ k k ≤ α β Here i represents a balanced strategy labeling of bi . level of reflection M, M N, such that = . According to Definition 2, if the limit (1) exists, it Proof. Let α ,α ,…,α be the distinct subtrees of Γ . Γ 1 2 n i should be a balanced strategy labeling for i. This ρ α ρ α Then N=max ( k), k=1,..,n, where ( k) is the level constructive interpretation of balanced strategy α o of reflection of k. labelings gives us an idea of how to compute them. Γ If i is a regular tree, then every balanced strategy First, it is necessary to compute a balanced strategy labeling φ of Γ will have at most N different labeling for the first-order belief tree b 1 (in fact, i i strategies, where N is the number from Proposition 1. computation can start from any tree b k). This can be i In order to find φ we need to consider only the first done, for example, by applying backward induction. Γ * N+1 levels of reflection. Let i denote the tree that is The strategies computed at the first stage are further obtained from Γ by removing all subtrees which level refined by computing a balanced strategy labeling for i of reflection is greater than N+1. Since φ is a the second-order belief tree b 2. After that, the i balanced strategy labeling, the strategy s , s =φ(v), computation proceeds with the third-order belief tree v v assigned to a node v, should be a best response to the b 3, and so on. This process can stop at any time or i strategies assigned to the successors of v. That is, when some predefined solution precision is achieved. prob s =f(Γ (v), s ,s ,…,s ), There is no guarantee that this process of iterative v i v1 v2 vk where s ,s ,…,s are the strategies assigned to the refinement is convergent. If it is convergent, v1 v2 vk successors v ,v ,…,v and f() is a best response however, the limit is a balanced strategy labeling. It 1 2 k function. is worth noting that any stage of the computation can If we write this equation for every node of Γ *, we yield several intermediate solutions. Therefore, i will obtain a system of N simultaneous equations: uniqueness of the final balanced strategy labeling is Γ prob not guaranteed. If, however, two balanced strategy s1=f( i (v1), s11,s12,…,s1k), Γ prob labelings exist, they will be payoff equivalent. That s2=f( i (v2), s21,s22,…,s2k), is, they give agent i the same expected payoff. … (2) s =f(Γ prob(v ), s ,s ,…,s ), 3. Representation of infinite belief trees N i N N1 N2 Nk where s ∈{s ,s ,…,s }, n=1,..,N, m=1,..,n . Let with finite trees nm 1 2 N k SN={s1,s2,…,sN}. Some strategies from SN appear only In this section we show how regular infinite belief on the left side of equations (2) and some strategies trees can be represented in a finite way. Such appear on both sides. Let SL denotes the set of all representation has several important advantages. strategies from SN that appear on the right side of at First, agents can cope with infinite belief hierarchies least one equation. Without loss of generality we may by reducing them to finite graphs. Second, agents can assume that SL={s1,s2,..,sL}. Solving (2) consists of apply to infinite beliefs the same techniques they use two stages. First, we should find all strategies in SL to handle finite beliefs. and after that we should find SN\SL. Since every strategy in S \S is a function of strategies from S , it Definition 3. An infinite tree is regular if and only if N L L is straightforward to calculate S \S having S . In the number of its distinct subtrees is finite [5]. N L L order to find SL we have to solve the following system It is evident that a regular belief tree is a repetition of simultaneous equations: of a finite number of subtrees. This means that by Γ prob s1=f( i (v1), s11,s12,…,s1k), extending a regular tree to infinity we do not add new prob s =f(Γ (v ), s ,s ,…,s ), strategically relevant information. The following 2 i 2 21 22 2k … (3) proposition states that for every regular belief tree s =f(Γ prob(v ), s ,s ,…,s ). there exists some finite level of reflection that is L i L L1 L2 Lk sufficient for finding a balanced strategy labeling. In This means that the vector of strategies T={t1,t2}. Assume that valuations are independent and According to Proposition 2, the notion of an π accessible pointed graph is a generalization of both that there exists some objective distribution =(p,1-p) finite and infinite belief trees. This, however, does from which valuations are drawn. That is, with probability p each bidder’s valuation is t , and with not imply that every accessible pointed graph 1 probability 1-p it is t . Since π is not common represents some beliefs. Nevertheless, this is the case: 2 knowledge, each bidder can hold some private beliefs Proposition 3. For every accessible pointed graph G about π. there exists a belief tree that is contractible to G. τ Suppose now that at some moment 0, before the π Proof. Every pointed graph, G, can be unfolded into a beginning of the auction, the actual distribution has π belief tree, Γ, whose root is the point of the graph. been (1/2,1/2) and has been common knowledge. Just before the action, at time τ , τ <τ , some event The nodes of Γ are the finite paths that start from the 1 0 1 π point of G. o occurs that changes distribution from (1/2,1/2) to (p,1-p), where p≠1/2. This event is not mutually 5. An example: auction analysis observable and the fact that π has changed is not common knowledge. In this situation each agent With the growing impact of electronic commerce, believes that the actual distribution is (p,1-p) and that auctions are going to play an increasingly important the other agent still believes that there is common role in multiagent systems and distributed artificial knowledge about (1/2,1/2). This is a realistic intelligence [17,18,14]. In this section we assumption, since in many electronic commerce demonstrate how the belief graph reasoning, applications bidders do not have sufficient presented here, can be applied to the analysis of information about their rivals. auctions. The belief structure of each bidder can be Most theoretical results in auction theory draw represented by the infinite belief tree shown in Figure crucially on the revenue equivalence theorem [19]. 5. In order to find a balanced strategy labeling for According to the theorem, the first-price sealed bid, that tree we need to analyze the first four levels of the second-price sealed bid, English and Dutch auctions tree (N=3). That is, we should look for 31 strategies are all optimal selling mechanisms, provided that corresponding to the first 31 nodes of the tree. Instead they are supplemented by an optimally set reserve of doing so, we look for the accessible pointed graph H that corresponds to the tree. The graph H is be a best response to the other. Since there does not represented in Figure 6. It shows that there are only 5 exist an equilibrium in pure strategies, we look for an non-identical nodes: the root, t1- and t2- nodes for the equilibrium where each bidder with valuation t1 bids first bidder and t1- and t2- nodes for the second bidder. t1 (t1 Because F(t1)=0, it follows that c= ½ (t2-t1). Thus, the Figure 5. The belief tree of bidder i. continuous distribution function F(x) is implicitly defined by (t2-x)( ½+ ½)F(x)= ½ (t2-t1) (4) Agent i Substituting a2 for x in Equation (4) and taking into p 1-p account that F(a2)=1, we obtain a2= ½t1+ ½t2. Agent j t1 t2 What remains to be done is to find a bidding 1/2 1/2 strategy b* corresponding to the root of the tree. It is clear that b* must be a best response to the strategy mixture [p,t ;1-p,b**], where b** is the strategy 1/2 1/2 1/2 1/2 1 defined by Equation (4). We can solve Equation (4) for F(x), thereby obtaining Agent i 1/2 1/2 t1 t 2 F(x)=(x-t1)/(t2-x). The expected utility of submitting bid x, given that Figure 6. The belief graph of bidder i the rival adheres to the strategy mixture [p,t1;1-p,b**] The solution for the first-price sealed bid auction is: ≤ ≤ without common knowledge about prior beliefs is (t2-x)(p+(1-p)(x-t1)/(t2-x)) if t1 x ½(t1+t2) provided by the following proposition. and t -x if ½(t +t ) is optimal. The expected utility is ½(t2-t1); Proof. Since the situation is symmetric, we are (iii) p>½: the optimal bid is ½(t1+t2). The looking for a symmetric solution. Therefore every r expected utility is ½(t2-t1). balanced strategy labeling consist of 3 strategies: one Proposition 5. When there does not exist common for the point of the graph H, one for t -nodes and one 1 knowledge about private beliefs, the revenue for t -nodes. The strategies assigned to t and t -nodes 2 1 2 equivalence theorem ceases to hold. The bidder’s should be in equilibrium. That is, each of them should expected utility is different in the first price sealed bid auction and Vickrey auction. [5] Courcelle B. Fundamental Properties of Infinite Trees. Theoretical Computer Science, 25: 95-169, Proof. Consider the first-price sealed bid auction and 1983. the second-price sealed bid auction. It follows from [6] Fagin R., Halpern J., Moses Y., Vardi M. Reasoning Proposition 4 that the expected utility for the bidder About Knowledge. MIT Press, 1995. with valuation t2 is ½(t2-t1) in the first-price sealed bid auction without common knowledge about prior [7] Fagin R., Geanakoplos J., Halpern J., Vardi M. The beliefs. On the other hand, for the second-price Hierarchical Approach to Modeling Knowledge and auction the dominant strategy for every bidder is to Common Knowledge. Int. Journal of Game Theory. bid his own valuation. Therefore, in the second-price [8] Halpern J., Moses Y. Knowledge and Common auction the expected utility for the bidder with Knowledge in Distributed Environment. Journal of ACM, 37(3):549-587, 1990. valuation t2 is (t2-t1)p, where p is the subjective probability that the other bidder’s valuation is t1. [9] Harsanyi J. Games with Incomplete Information Thus, the two auctions yield different expected Played by Bayesian Players. Part I. Management utility. r Science, 14(3):159-182, 1967. 6. Conclusions [10] Geanakoplos J. Common Knowledge. In K. J. Arrow and M.D. Intriligator, (eds.). Handbook of Game In this paper we identified a class of infinite belief Theory with Economic Applications. Elsevier, 1994. trees that allow finite representation. Such [11] Gmytrasiewicz P., Durfee E. A Rigorous, representation has several important advantages. Operational Formalization of Recursive Modeling. First, agents can cope with infinite belief hierarchies In Proceedings of ICMAS’95, pp. 125-132, 1995. by reducing them to finite trees or finite graphs. [12] Gmytrasiewicz P., Durfee E. Decision-Theoretic Second, agents can apply to infinite beliefs the same Recursive Modeling and the Coordinated Attack techniques they use to handle finite beliefs. Problem. In Proceedings of the First International As an example of our approach we showed that Conference on Artificial Intelligence Planning without common knowledge about prior beliefs, the Systems, pp. 88-95, 1992. revenue equivalence theorem of auctions ceases to [13] Gmytrasiewicz P., Durfee E., Wehe D. A Decision- hold. Since different auctions yield different Theoretic Approach to Coordinating Multiagent revenues, auction designers should be careful when Interactions. In Proceedings of IJCAI’91, pp. 62-68, choosing auction rules. Our solution concept for 1991. infinite belief trees provides an analytic tool for [14] Lehmann D., O’Callaghan L., Shoham Y. Truth comparing different auction forms and more general Revelation in Rapid, Approximately Efficient interaction mechanisms Combinatorial Auctions. In Proceedings of the ACM Most mechanism design today is based on Nash Conference on Electronic Commerce, 1999. equilibrium which assumes common knowledge of [15] Mertens J-F., Zamir S. Formulation of Bayesian priors. At the same time it is known that common Analysis for Games with Incomplete Information. knowledge is not obtainable with any amount of International Journal of Game Theory, 14:1-29, communication. Therefore, our method that does not 1985. rely on common knowledge holds promise as a future [16] Rheinboldt W. Methods for Solving Systems of analysis and design tool. Nonlinear Equations. SIAM, Philadelphia, 1998. References [17] Sandholm T. Limitations of the Vickrey Auction in Computational Multiagent Systems. Proceedings of [1] Armbruster W., Boge W. Bayesian Game Theory. In the ICMAS’96, pp. 229-306, 1996. O. Moeschlin and D. Pallaschke (eds.). Game Theory [18] Sandholm T. An Algorithm for Optimal Winner and Related Topics. North-Holland, 1979. Determination in Combinatorial Auctions. In [2] Aczel P. Non well-founded sets, CSLI, 1988. Proceedings of IJCAI’99, pp. 542-547, 1999. [3] Bacharach M., Gerard-Varet L., Mongin P. and Shin [19] Vickrey W. Counterspeculation, Auctions and H. Epistemec Logic and the Theory of Games and Competitive Sealed Tenders. Journal of Decisions. Kluwer, 1997. Finance,16:8-37, 1961. [4] Brandenburger A., Dekel E. Hierarchies of Beliefs [20] Vidal J., Durfee E. The Impact of Nested Agent and Common Knowledge. Journal of Economic Models in an Information Economy. Proceedings of Theory, 59:189-198, 1993. the ICMAS’96, pp. 377-384, 1996.