Bayesian Rationality and Social Norms Herbert Gintis

Bayesian Rationality and Social Norms

Herbert Gintis∗

November 13, 2007

1 Introduction

There is a glaring contrast between economics, which models social interaction as a Nash equilibrium of a game played by rational decision-makers (Mas-Colell, Whinston and Green 1995), and sociology, which models social interaction as the role-playing of individuals guided by social norms (Durkheim 1933[1902], Parsons 1967).1 The incentive compatibility requirements of Nash equilibrium are virtually ignored by sociologists, who argue that humans are prosocial by nature, and generally carry out the duties and obligations associated with the social roles they assume rather than maximizing personal gain. The commonality of beliefs stressed by sociologists as key to social efficiency have traditionally been ignored by economists, who typically celebrate the irreducible heterogeneity of rational agents (Becker and Stigler 1977). The remarkable body of empirical evidence supporting the sociological model has made little impression on economists, who favor deriving social norms from the interaction of rational actors. The equally remarkable body of evidence supporting the rational actor model is rejected by sociologists on grounds that the standard sociological model incorporates all the strengths of the rational actor model, and in addition can deal with the socially determined nature of beliefs (Boudon 2003). The discrepancy between these two firmly held positions has been a major factor sustaining theoretical disarray in the behavioral sciences (Gintis 2007). Recent findings suggest, however, that economic and sociological theory can be rendered consistent and mutually supportive. First, epistemic game theory has ∗ Santa Fe Institute and Central European University. I am grateful for the comments of Masahiko Aoki, RobertAumann, Adam Brandenburger, Gerry Mackie,Yusuke Narita, Robert Sugden, Giacomo Sillari, Luca Tumolini, and Peter Vanderschraff. I would like to thank the John D. and Catherine T. MacArthur Foundation for financial support. 1A Bayesian rational agent maximizes expected utility, given a subjective prior over the various possible outcomes (Savage 1954, Anscombe and Aumann 1963, Kreps 1988).

1 demonstrated the centrality of common priors and common knowledge in ensuring that rational agents play a Nash equilibrium (Aumann and Brandenburger 1995, Polak 1999). Second, behavioral game theory has demonstrated that normative behavior need not, and usually does not, violate Bayesian rationality (Fehr and Gächter 2002, Andreoni and Miller 2002, Gintis, Bowles, Boyd and Fehr 2005, Gneezy 2005). Third, Aumann (1987) has shown that the natural equilibrium condition for game theory is the correlated equilibrium, and in this paper I will show that the most promising candidate for a correlating device—which epistemic game theory does not provide, even in principle—is the social norm. Finally, epistemic game theory has shown how a commonality of beliefs can be maintained by a set of rational actors facing asymmetric information. The need for such commonality was stressed by Durkheim (1933[1902]), who observed that the division of labor in modern society fosters a heterogeneous citizenry and an ethic of individual autonomy, yet social efﬁciency requires organic solidarity based on collective representations that harmonize the mental constructs of individuals.2 Never answered in the sociological literature is how a common culture can be maintained by rational actors facing asymmetric information. Aumann (1976) answered this question by showing that if rational agents have a common prior and if their posteriors following an event are common knowledge, then their posteriors agree as well.3 This analysis shows that collective representations can be maintained by rational actors, so the ‘suspension of disbelief’ prima facie required by Durkheim’s organic solidarity is in fact unnecessary. If Bayesian rational agents have common prior probabilities or expectations, the receipt of asymmetric information conserves the commonality of posteriors, provided these are common knowledge. However, game theory fails to supply conditions under which posteriors will be common knowledge, and supplies no reasons for the existence of common priors. If game theory could explain collective representations (common priors and common knowledge), even in principle, as an outcome of the interaction of rational agents, it would be plausible to view game theory as the unique foundation for all of the behavioral sciences (economics, sociology, anthropology, social psychology, and political science). While the aim of recent game-theoretic research has been exactly this, I will show that this task cannot be achieved, even in principle. The foundations of behavioral science must therefore include analytical constructs in addition to game theory, and the minimum set of human characteristics needed to

2To the extent that there are ethnic, regional, cultural, and social differences within a society, collective representations may differ across groups. Moreover, organic solidarity is obviously an equilibrium conditions that fails in periods of social change. 3Geanakoplos (1992) provides an overview of the literature inspired by Aumann’s original Agree- ment Theorem.

2 model strategic interaction must include more than Bayesian rationality. Game theory would provide a mechanism for the formation of common priors if the Harsanyi doctrine (Harsanyi 1967–68) were correct. The Harsanyi doctrine holds that rational individuals can have divergent beliefs only if they have different information. Asymmetric information, according to Harsanyi, can be modeled by assuming common prior beliefs and formally incorporating the mechanisms of informational asymmetry in the model itself. This argument may be plausible under favorable observational conditions when the events in question are natural occurrences for which causal covering laws and probabilistic frequencies exist. However, it is not plausible when the events involve the subjective priors of other agents (Morris 1995, Gul 1998). Aumann’s (1998) classic response to Gul’s (1998) critique of the common prior assumption is that the common prior assumption, “embodies a reasonable and use- ful approach to interactive decision problems, though by no means the only such approach.” (p. 929) Aumann’s position, then, is that the concordance of beliefs across individuals (common priors) is sufficiently widespread in social groups that it may be taken as a starting-point for analytical purposes, although it does not derive from principles of Bayesian rationality. While this position is pragmatically defensible as a provisional step in the development of social theory, it is clearly not the final word on the topic. Indeed, game theory is rejected by many behavioral scientists who, like Durkheim and Parsons, consider the commonality of beliefs a highly problematic yet key aspect of human sociality, and study the mechanisms that foster such commonality. We say an event E is common knowledge for agents i = 1,...,n if the each agent i knows E, each i knows that each agent j knows E, each agent k knows that each j knows that each i knows E, and so on. So, for instance, we say that common priors are common knowledge if each agent knows that each agents has the same priors, each agent knows that each agent knows that each agent has the same priors, and so on. The importance of common knowledge for interactive epistemology was first stressed by Lewis (1969) and Aumann (1976), its critical role among the conditions for correlated and Nash equilibrium being established in Aumann (1987) and Aumann and Brandenburger (1995). Aumann’s (1976) epistemological analysis of common knowledge derives collective representations from axioms of epistemic logic, in the form of a theorem asserting that if E is a public event, meaning that it is self-evident for each of a set of agents, then at every state ω ∈ E, E is necessarily common knowledge (precise definitions are given later). However, as we shall see, if the knowledge system is constructed without specific inter-agent epistemological assumptions, the only public event that can be shown to follow from the tautologies of the system and assumptions concerning the distribution of first-order knowledge, is the trivial event

3 , the whole event space. We conclude that social norms and conventions, which do assert the commonality of mental representations, are emergent properties of human social systems that must be posited along side of Bayesian rationality to explain human strategic interaction. The capacity to share beliefs is also an evolved characteristic of the human brain (Premack and Woodruff 1978, Heyes 1998, Tomasello, Carpen- ter, Call, Behne and Moll 2005). This capacity doubtless arose during the gene- culture coevolutionary process that gave rise to our species (Cavalli-Sforza and Feldman 1981, Boyd and Richerson 1985, Durham 1991). Indeed, human culture is only the latest in a series of emergent transitions characteristic of the biological evolution of complexity from the most primitive microbes to the present (Maynard Smith and Szathmary 1997, Morowitz 2002). Affirming the emergent character of social norms means they have the status of fundamental principles that cannot be derived analytically from other fundamental principles. Affording this status to social norms requires that we rethink certain basic theoretical presuppositions, because it entails denying the principle of methodological individualism, which holds that all social phenomena can be explained purely in terms of the characteristics of the agents involved. Methodological individualism is a deeply held, but unsupported, precept of game theory. Afford- ing explanatory power to social norms, however, will allow us to solve heretofore insoluble problems of epistemic game theory.4 The sociological and economic arguments, properly cast, are thus in fact com- plementary, not alternative, explanations. Sociological models explain why there are common priors, and common knowledge: only arguments taken from psychology and sociology can make scientifically grounded assertions as to when the conditions for the transformation of mutual knowledge into common knowledge might obtain. Economics provides the basic model of rational choice, shows how informational heterogeneity can be consonant with common posteriors, and iden- tifies the correlated equilibrium as the natural equilibrium condition for rational actors with common priors. Sociology provides the analysis of correlating devices, in the form of social norms and conventions, and psychology elucidates the brain mechanisms that permit and foster the commonality of beliefs. The framework for integrating social norms and Bayesian rationality proposed in this paper recognizes three inferential stages. First, there are natural occurrences

4Game theorists often take methodological individualism as synonymous with the scientiﬁc method, on the grounds that scientiﬁc explanation consists precisely in modeling the whole in terms of the interaction of its parts. However, in a complex system, the modeler cannot abandon higher-level constructs. For instance, it is rarely fruitful to analyze the behavior computer software in terms of the interactions of individual transistors, or to model the immune system of an organism in terms of the quantum states of its constituent chemical molecules.

4 N, such as “the ball is yellow,” or “it is raining in Paris” that are immediately perceived by individuals as first-order sensory experience. Under some conditions these natural occurrences are public signals that are mutually accessible to members of a group, meaning that if one member knows N, then he knows that each other member knows N. For instance, if i and j are both looking at the same yellow ball, the ball’s color may be mutually accessible: i knows that j knows that the ball is yellow. Second, there are higher-order socially defined events which we call games G, which specify the type of strategic interaction appropriate to the social situation at hand. Games are not mutually accessible, but social conventions may specify that a mutually accessible event F indicates G. We call F a frame, we write G = φ(F), and we think of the relation “F indicates G to agent i” as asserting that when i knows F , he proceeds through a series of mental steps involving the consideration of known social regularities, such as norms and conventions, at the conclusion of which i knows G (Lewis 1969, Cubitt and Sugden 2003). Assuming F is a public indicator of G, and that the individuals involved are symmetric reasoners (precise definitions are left for later), then G will be common knowledge. Third, given epistemic game G, certain social processes that transform private into public information may justify the assumption of common prior for G, which in turn determines a correlated equilibrium of G. The correlating device is a social norm or convention N = ψ(G), which specifies exactly how the game will be played by rational agents. Of course, in the real world, at any stage there may be irregularities that produce non-equilibrium outcomes.

2 Informational Asymmetry and Common Knowledge

Aumann (1976) proved that if Bayesian rational agents have a common prior concerning the expected value of a variable or the probability of an event, the receipt of asymmetric information conserves the commonality of posteriors, provided these are common knowledge. This theorem is usually treated as a curiosum, because there is considerable “agreement to disagree” on asset markets. In fact, it is a very general argument explaining why common knowledge can be updated via asymmetric information and yet remain common knowledge among rational social agents. We present an elegant generalization of Aumann’s argument. Consider a ﬁnite state space with typical element ω ∈ . Subsets of are called events. Each agent i has a subjective prior pi(·) over . We represent asymmetric information by endowing each agent i with a partition Pi of such that in state ω ∈ P ∈ Pi, i knows only that the true state is in P . This implies that pi(·) must be constant on the elements P ∈ Pi. We call the elements P ∈ P the cells of P. We write the cell of i’s partition containing ω as Piω ∈ Pi, so the conditional probability of an event

5 E ⊆ for i in state ω is pi(E|Piω). We assume pi(Piω) > 0 for all ω ∈ . The common knowledge partition P∗ corresponding to individual partitions P1,...,Pn ω is the finest common coarsening of the individual partitions. That is, the cell P∗ of P∗ containing ω is the union of cells of Pi for each i = 1,...,n. We call this construction in modal logic the semantic model of interactive epistemology.5 P Let f :2 i →. We say a f satisfies the sure thing principle on if for all P,Q ⊆ with P ∩ Q =∅,iff(P) = f (Q) = a, then f(P ∪ Q) = a.For instance, if p is a probability distribution on and E is an event, then the posterior probability f(X)= p(E|X) satisfies the sure thing principle, as does the expected value f(X) = E[x|X] of a random variable x given X ⊆ . We then have the following Agreement Theorem (Collins 1997):

Theorem 1. Suppose for agent i = 1,...,n, fi satisﬁes the sure thing principle ω on , and suppose it is common knowledge at ω that fi = ai. Then fi(P∗ ) = ai.

ω Proof: Because P∗ is the disjoint union of i’s partition cells Piω , and fi = ai on ω each of these cells, by the sure thing principle, fi = ai on P∗ .

Corollary 1. Suppose agents i = 1,...,nshare a common prior on , indicating an event E has probability p(E). Suppose each agent i now receives private information that the actual state ω is in Piω. Then, if the posterior probabilities ai = p(E|Piω) are common knowledge, then a1 = ...= an.

ω Proof: Let fi = p(·|Piω). Applying Theorem 1, we have ai = p(E|P∗ ) for all i.

3 The Agreement Theorem and the Conditions for Nash Equi- librium

Agreement theorems are at the heart of game theory because the conditions under which rational agents play a Nash equilibrium are precisely those that create conditions for an Agreement Theorem (Aumann and Brandenburger 1995). We define an epistemic game G to be a normal form game with players i = 1,...,n and a finite pure strategy set Ai for each player i,soA = A1 × ...× An is the set of pure strategy profiles for G, with payoffs πi :A→R. In addition, G includes a set of possible states of the game, a knowledge structure (Pi, Ki, Pi) defined over for each player i, and a subjective prior pi(·; ω) over the actions A−i for the other players that is a function of the state ω. Because player i cannot distinguish among states in Piω, pi(·; ω) = pi(·; ω ) for all ω ∈ Piω. Finally, we specify that each state ω specifies the players’pure strategy choices a(ω) = (a1(ω),...,αn(ω)) ∈ A

5I have found no uniform terminology for this construction in the literature.

6 ω ω and their beliefs φ1 ,...,φn concerning the choices of the other players. We call ω φi i’s conjecture concerning the behavior of the other players at ω. Player i’s ω = ; conjecture is derived from i’s subjective prior by φi (a−i) pi(a−i Piω). Thus, at state ω ∈ , each player i takes the action ai(ω) ∈ Ai and has the probability ω ω distribution φi over A−i. Note that φi is constant on Piω, so we could write it as Pi ω φi , but we will generally use the more concise notation. A player i is Bayesian [ ω ] rational at ω precisely if ai(ω) maximizes E πi(ai,φi ) , which is deﬁned in the usual way as [ ω ]= ω E πi(ai,φi ) φi (a−i)πi(ai,a−i). (1) a−i ∈Ai

We say that “player i knows the other player’s actions a−i at state ω”ifPiω ⊆ { | = } ω = ∈ ω ω (a−i) a−i , which means φi (a−i(ω )) 0 for all ω / Piω. We assume { | ω = }⊆{ | = } that ω φi (a−i) 1 ω α−i(ω ) a−i ; i.e., if agent i knows something, ω = ω ∈ then what he knows is true. Note that φi φi for all ω Piω, because i cannot distinguish among states is Piω. Aumann and Brandenburger (1995) and Polak (1999) are responsible for the following theorem, in which knowledge is interpreted as belief with probability one, and where “mutually known” means “known by all players”: Theorem 2. Let G be an epistemic game with n>2 players, and let φ = φ1,...,φn be a set of conjectures. Suppose at ω ∈ it is mutually known that the game is G, mutually known that players are rational, and commonly known that φ is the set of conjectures for the game. Then for each j = 1,...,n, all i = j induce the same conjecture σj (ω) about j, (σ1(ω),...,σn(ω)) form a Nash equilibrium of G, and it is commonly known that each player is Bayesian rational. Because the assumptions that the players know the game and are rational are relatively weak, Theorem 2 indicates that common knowledge of conjectures is the key problematic condition we need to conclude that rational agents will implement a Nash equilibrium. The question, then, is under what conditions is common knowledge of conjectures likely to be instantiated in real-world strategic interactions?

4 When Mutual Knowledge becomes Common Knowledge

Let P be the knowledge partition of the state space for an agent. The cell of P containing ω is written Pω. We say the agent knows an event E at state ω if Pω ⊆ E. If the agent knows E for all ω ∈ E, we say E is self-evident.6 It is easy to see that a self-evident event is the union of cells of P. If two events are

6This usage conforms to Bacharach (1992). Binmore and Brandenburger (1990) call a self-evident event a “truism,” and refer to an event that is a truism for all agents as a “common truism.” Following

7 self-evident, so is their union. Hence for any event E, it we can deﬁne KE as the largest self-evident event contained in E. Clearly, KE ={ω|Pω ⊆ E}. An event E is thus self-evident if and only if KE = E. Suppose we have a set of n of agents, each of whom has a partition Pi of and a corresponding knowledge operator Ki, i = 1,...,n. We say an event E is a public event if E is self-evident for all i = 1,...,n. If two events are self-evident for an agent, so is their intersection. Hence, the intersection of public events is also a public event. Thus, for any ω ∈ , there is a minimal public event P∗ω containing ω. It is easy to check that the events P∗ω are the cells of the common knowledge partition P∗ corresponding to P1,...,Pn. We say an event E is common knowledge at ω if P∗ω ⊆ E. We deﬁne K∗E ={ω ∈ E|P∗ω ⊆ E},soK∗ is the knowledge operator corresponding to P∗. It might be thought that an event E being common knowledge implies nothing concerning what agents know about the knowledge of other agents. This, however, is not the case. Indeed, Aumann (1976) showed that an event E is common knowledge precisely when everyone knows E, everyone knows that everyone else knows E, and so on.7 This shows that the self-evidence of an event must incorporate in some deep sense the structure of mutual interrelatedness of beliefs. A weakness of the semantic model is its failure to elucidate this sense. The idea behind this result is illustrated in Figure 1, where E is a public event for two agents, Alice and Bob. Here, PAω ⊆ E, so Alice knows E at ω. For each ω ∈

⊆ ⊆∪ ⊆ PAω, PB ω E, because E is self-evident for Bob. But PAω ω ∈PAωPB ω E, so Bob knows that Alice knows E at ω. In Figure 1, two of Bob’s partition cells cover Alice’s partition cell at ω. A similar argument shows that Alice knows that Bob knows that Alice knows E at ω, and so on. The conclusion that when a public event occurs, each agent knows that the others know that the event occurred, is quite striking, for it asserts that self-evidence, which is ostensibly a characteristic of a single agent, when shared among agents, permits agents to know the content of the minds of other agents. Of course, the term “to know” implies that the object of knowledge is true, and therefore tautologically, if i knows E, then i knows that j = i does not know that ¬E; i.e., by deﬁnition, if an agent knows E, he knows that other agents know that E is “possible.” But, this tautological implication cannot be extended to a determination of what other agents

Milgrom (1981), we prefer the term “public event” to the term “common truism.” Aumann (1987) has no term for what we call “self-evident,” and uses the term “self-evident” for what we call “public.” Because there is no accepted usage, I will retain the self-evident/public terminology on grounds that these terms best describe the situation under investigation. 7In fact, Aumann (1987) deﬁnes E to be common knowledge if each agent knows E, each knows the others know E, and so on. He then proves that common knowledge is equivalent to KiE = E for each agent i; i.e., using our terminology, E is self-evident for each agent.

8 E

ω ω PAω P ω B PB ω

Figure 1: At ω, Bob knows that Alice knows that E. positively know.8 Certainly, the standard axioms of Bayesian rationality provide no means by which one rational actor can know the contents of another rational actor’s mind, beyond the minimal delimitations expressed in the previous paragraph. Of course, one could posit plausible conditions under which rational actors might know the content of one another’s minds, and I shall do that below. However, without adding to the standard axioms of Bayesian rationality (Savage 1954, Kreps 1988), this would appear to be impossible. In fact, the commonality of beliefs across agents is smuggled in under the tacit assumption that each agent “knows” the partition structure Pi of other agents. For instance, referring to Figure 1, unless Bob “knows” Ann’s cell PAω, how can he “know” that Ann knows E? Note that if Bob “knows” that E is self-evident to Ann, then he can deduce analytically the chain of assertions “I know that Ann knows that I know…”. But, how does Bob “know” that E is self-evident to Ann?9 Aumann (1976) defends the semantic model by asserting that no additional assumptions are involved: The implicit assumption that the information partitions…are them- selves common knowledge…constitutes no loss of generality. Included in the full description of a state of the world is the manner in which information is imparted to the two persons. (p. 1273) However, in the standard presentations of the epistemic model found in the literature, states of the world never include such full descriptions. Indeed, it is rare to include

8 As explained below, if kix means that i knows x, then ¬ki¬x expresses the idea that i takes x as possible. Actually, by the Axiom of Transparency (Equation 5), ¬ki¬x implies that i knows that x is possible. 9I have put ‘know’ in double quotes in the above argument to differentiate this general sense of the term from the deﬁnition within the semantic model, in which knowledge is explicated in terms of the knowledge partition.

9 any speciﬁcation of epistemic conditions beyond the ﬁrst order. Moreover, as we shall see, expanding the model to allow states to include such full descriptions does not lead to a model in which one can assert that individuals know the beliefs of others, unless substantive statements concerning second- and higher-order epistemic conditions are added to the usual axioms of epistemic modal logic. To give the reader some idea what a “full description” might look like, and what implications can be drawn from such a full description, we analyze a famous common knowledge model, which we call The Three Tactful Ladies, in Section 10. Aumann (1987) expands on his reasoning as follows:

Because the speciﬁcation of each ω includes a complete description of the state of the world, it includes also a list of those other states ω of the world that are, for Player 2, indistinguishable from ω.…Therefore the very description of the ω’s implies the structure of [the partition]…. The description of the ω’s involves no ‘real’knowledge; it is only a kind of code book or dictionary. (p. 9)

However, there is nothing in the definition of common knowledge that specifies what one agent knows about the partition of another agent. Aumann asserts that one can prove in complete generality that using the “tautology” of partition structures, mutual self-evidence implies each agent knows what other agents know. There is surely something amiss here. As we shall see, when the knowledge system “involves no real knowledge,” the only event that is common knowledge and can be shown to obtain based on the tautologies of the model and the distribution of first-order knowledge specified by the problem at hand is the trivial event .

5 A Syntactic Model of Distributed and Shared Knowledge

Aumann (1999) elaborates on his defense of the notion that the partition structure is purely a formalism, developing a syntactic model of common knowledge in which partitions of the state space are not employed, and shows that it is isomorphic to a semantic model that employs the partition machinery. This elegant model lays bare the epistemological presumptions of the standard semantic model of common knowledge and reveals the presuppositions that permit common knowledge to be inferred. In fact, we will see that Aumann’s construction shows the substantive rather than the tautological nature of the partition construction in the standard semantic model. Following Aumann (1999), suppose we have n individuals, and a set of letters from an alphabet X ={x,y,z,...}, symbols ∨, ¬, k1,...,kn and left, ’(’and right, ’)’ parentheses. Formulas are constructed recursively as follows:

10 a. Every letter is a formula. b. If f and g are formulas, so are (f ) ∨ (g), ¬(f ), and ki(f ) for each i. We abbreviate (¬f)∨ g as f ⇒ g, ¬(¬f ∨¬g) as f ∧ g, (f ⇒ g) ∧ (g ⇒ f)as f ⇔ g, and we drop parentheses where no ambiguity results, assuming the usual precedence ordering of the propositional calculus, and assigning the highest order to the knowledge symbols ki. The above conditions ensure that every tautology of the propositional calculus based on X is a formula (Hedman 2004). A list L is a set of formulas. A formula is a tautology if it is a tautology of the propositional calculus, or it has one of the following forms, where f and g are formulas:

kif ⇒ f (2)

kif ⇒ kikif (3)

kif ∧ ki(f ⇒ g) ⇒ kig (4)

¬kif ⇒ ki¬kif. (5)

f ∈ T ⇒ kif ∈ T (6)

Formula (5), called the axiom of transparency, is required to ensure that the semantic realization of the syntactic system has a partition structure. Note that (4), which says that the knowledge operator satisﬁes modus ponens, is equivalent to ki(f ⇒ g) ⇒ (kif ⇒ kig). To see this, suppose (4), which can be written as ¬kif ∨¬ki(f ⇒ g)∨kig, which can be written as ¬ki(f ⇒ g)∨(¬kif ∨kig), which is equivalent to ¬ki(f ⇒ g) ∨ (kif ⇒ kig), which can be written as ki(f ⇒ g) ⇒ (kif ⇒ kig). Moreover, each of these steps is reversible, proving the equivalence. Rule (6) says that an agent knows all tautologies of the propositional calculus. We call a system consisting of the alphabet X , the formulas and the tautologies a syntactic system S. The set of tautologies T is closed under modus ponens (i.e., f, (f ⇒ g) ∈ T implies g ∈ T ) and the knowledge operator (i.e., f ∈ T implies kif ∈ T ). Note that (4) and (6) have the following implications:

ki(f ∧ g) ⇒ kif ∧ kig (7)

kif ⇒ ki(f ∨ g) (8)

To prove (7), note that f ⇒ (g ⇒ f ∧ g) is a tautology, so by two applications of modus ponens,wehavekif ⇒ (kig ⇒ ki(f ∧ g)), which is equivalent to kif ∧ kig ⇒ ki(f ∧ g). Moreover, f ∧ g ⇒ f ,soki(f ∧ g) ⇒ kif , and similarly ki(f ∧ g) ⇒ kif ,soki(f ∧ g) ⇒ kif ∧ kig.

11 A state ω is list that is closed under modus ponens, and for every formula f , exactly one of f and ¬f is in ω. It is easy to see that if ω is a state, then T ⊂ ω.For otherwise ω would contain a false formula from the propositional calculus, which by modus ponens implies the ω contains all formulas, which is false by construction. Moreover, every state ω is a complete list of the formulas that are true in that state; i.e., we cannot add another formula to ω without violating the list property. As an exercise in using these concepts, we may explain footnote 8 more fully. We deﬁne pif , interpreted as “f is possible for i”bypif ⇔¬ki¬f . Then, by (5) and (6), pif ⇔ kipif ,soiff is possible for i, then i knows this. Moreover, taking the contrapositive of kj f ⇒ f ,wehave¬f ⇒¬kj f . Because this is a tautology, by (6), we have ki(¬f ⇒ ki¬kj f), so by (5), we have ki¬f ⇒ ki¬kj f . Because this is true for all formulas f , replace f by ¬f , getting kif ⇒ ki¬kj ¬f , which we can rewrite as kif ⇒ kipj f ; i.e., if i knows f , then i knows that f is possible for j. More complicated inferences concerning interpersonal knowledge can also be shown. For instance, following Hart, Heifetz and Samet (1996), we can show that pikj kif ⇒ kif (i.e., if it is possible for i that j knows that i knows f , then i knows f ). Taking the contrapositive of kj kif ⇒ kif , we get ¬kif ⇒¬kj kif . Because this is a tautology, and distributing the knowledge operator across the implication, we have ki¬kif ⇒ ki¬kj kif , which by the axiom of transparency, gives ¬kif ⇒ ki¬kj kif . Taking the contrapositive of this we get ¬ki¬kj kif ⇒ kif . Substituting pi for ¬κi¬, we get the desired expression. Note carefully, however, that the implication includes the possibility operator pi. There are no cross-agent knowledge implications involving the knowledge operators without terms of the form ¬ki¬. The following are some properties of the possibility operator p:

f ⇒ pf (9) kf ⇒ pf (10) p(f ∨ g) ⇒ pf ∨ pg (11) kf ∧ pg ⇒ p(f ∧ g) (12) k(f ∨ g) ⇒ pf ∨ kg (13)

6 Characterizing States in the Syntactic System S

Let us ﬁrst state and prove some elementary facts about formulas and states. For any formulas f, g and any state ω,wehave:

¬f ∈ ω iff f/∈ ω (14) f ∨ g ∈ ω iff f ∈ ω or g ∈ ω (15)

12 f ∧ g ∈ ω iff f ∈ ω and g ∈ ω (16) f ⇒ g ∈ ω iff f ∈ ω implies g ∈ ω (17) f ⇔ g ∈ ω iff f ∈ ω iff g ∈ ω. (18)

Proof: Statement (14) follows from the definition of a state. For (15), suppose first that f ∈ ω. Then, because f ⇒ f ∨ g is a tautology, we have f ⇒ f ∨ g ∈ ω, so by modus ponens, f ∨ g ∈ ω. Similar reasoning shows g ∈ ω. Now, suppose f ∨γ ∈ ω but f, g∈ / ω. Then, ¬f, ¬g ∈ ω, and because ¬f ⇒ (¬g ⇒¬(f ∨g)) is a tautology, and hence in ω,bymodus ponens,wehave¬g ⇒¬(f ∨ g) ∈ ω, so again by modus ponens,wehave¬(f ∨ g) ∈ ω, which is a contradiction. The remaining statements follow simply from the first two. We define the atomic formulas of S as those formed by repeated application of ¬ and ki, and kj to the letter x (i.e., the formula can be written without ∨). We say an atomic formula is reduced if it contains no substrings of the form ¬¬, kiki,or ki¬ki. Note that in the first case, both symbols, and in the second and third cases the first symbol, can be dropped, obtaining a logically equivalent formula, by (5) and (2). The result is that every atomic formula has an equivalent reduced form. If we write ± to mean ‘either ¬ or nothing’, then a reduced atomic formula is of the form f =±ki ± kj ...± kr ± x, where x ∈ X and no two successive k s has the same subscript. We will assume forthwith that all atomic formulas in a state ω are reduced. It is also easy to see that every reduced atomic formula can be written as f = yiyj ...yr ±x, where each y is either k or p, no two successive subscripts are equal, and x ∈ X . This is obvious if f is first order. Let f =¬f where f is order r,we can propagate the ¬ towards the end of the reduced atomic formula for f , crossing each p and turning it into k, and crossing each k and turning it into a p, until we reach ±x, which is then negated. For instance

¬p1p2k3k4p5¬x = k1¬p2k3k4p5¬x = k1k2¬k3k4p5¬x

= k1k2p3¬k4p5¬x = k1k2p3p4¬p5¬x

= k1k2p3p4k5x.

The only other possibility is f = kif where f is order r, in which case the assertion is obvious. Every formula f in a state ω is the disjunction of reduced atomic formulae fr ,at least one of which is in ω. To prove the latter assertion, suppose f = f1 ∨...∨fk ∈ ω, and f1,...,fk ∈/ ω. Then, ¬f1,...,¬fk ∈ ω,so¬f1 ∧ ...∧¬fk ∈ ω,so ¬(¬f1 ∧ ...¬fk)/∈ ω. However, ¬(¬f1 ∧ ...∧¬fk) ⇔ f1 ∨ ...∨ fk, which is a contradiction.

13 By (15), if f ∈ ω, then f ∨ g ∈ ω for any formula g. We say a formula f ∈ ω is clean if every term in its expression as a disjunction of atomic formulae is in ω. Clearly, every formula in ω is the disjunction of a clean formula and formulae that are not in ω. Moreover, every clean formula can be expressed as a disjunction (as well as a conjunction) of atomic formulae, each of which is in ω. This shows that every f ∈ ω is a disjunction, one of whose terms is a reduced atomic formula in ω. Another way of expressing this fact is that for every f ∈ ω, there is a reduced atomic formula fo ∈ ω such that fo ⇒ f is a tautology. For any formula f , either kif ∈ ω or ¬kif ∈ ω, and either kikj f ∈ ω or ¬kikj f ∈ ω for all agents i, j and formulas f , so each state of in the syntactic model includes all the information each player needs to ascertain what other players know in that state. Aumann interprets this situation in which j appears to know something about i’s mental state (“knowing that f ”) is purely formalistic. He correctly observes:

Does each individual “know” the operators ki of the others (in addition, of course, to his own)? The answer is “yes.” The operator ki operates on formulas; it takes each formula f to another formula. Which other formula? What is the result of operating on f with the operator ki? Well, it is simply the formula kif . “Knowing” the operator ki just means knowing this deﬁnition. Intuitively, for an individual j to “know” ki means that j knows what it means for i to know something. It does not imply that j knows any speciﬁc formula kj f . (p. 277).

7 The Semantic Interpretation of Syntactic System S

Let be the set of all states derived from S using the above construction. We deﬁne a semantic knowledge system S∗ on state space as follows. A set E ⊆ is called an event. Let κi(ω), ω ∈ be the set of all formulas in ω of the form kif for some formula f . The cells of the partition Pi for player i are deﬁned by

Piω ={ω |κi(ω) = κi(ω )}. In words, an agent can distinguish between two states if and only if he has some knowledge that is true in one state but false in the other. The knowledge operator KiE is deﬁned in the usual manner by KiE ={ω|Piω ⊆ E}. We call the resulting system the canonical semantic knowledge system S∗ corresponding to S. To construct a map from S to S∗, we deﬁne

Ef ={ω ∈ |f ∈ ω}; (19) i.e., the event Ef is the set of states in which f is true. The most important

14 implications of this mapping are = KiEf Eki f (20)

Ef ⊆ Eg if and only if f ⇒ g is a tautology, (21) and KiE ={ω| Ef ⊆ E}. (22)

ki f ∈ω

These properties imply that the syntactic operator ki does in fact correspond to the knowledge operator Ki, and logical implication in the syntactic system is the counterpart of set inclusion in the semantic system. Moreover, what an agent knows, given an event E, is whatever follows logically from the formulas that the agent knows, given E. The proofs of these assertions, which are straightforward, are given in Aumann (1999). The map f → Ef has several powerful features, but it is not an isomorphism because no inverse map is defined. Moreover, the range of the mapping contains a continuum of states rather than the finite or denumerable universe in the standard semantic knowledge model (Hart et al. 1996). Finally, as Aumann notes, there are ∗ events in S that are not of the form Ef for some formula f , because there are denumerable sets of states in S∗ that cannot be represented by finite conjunctions of formulas. In terms of S∗, the theorem that an event is common knowledge if and only if it is self-evident for all agents is transparent. An event E is self-evident for i when KiE = E.IfE is in the image of S under the above isomorphism, the E = Ef for = = ∈ some formula f , in which case KiEf Eki f Ef , so whenever ω Ef ,wehave ∈ = ω Eki f .Iff is a tautology of the propositional calculus, then Ef , which is trivially common knowledge. If f is epistemic, however, following the reasoning of the previous sections, we can consider f to be the disjunction of a finite number of reduced atomic formulae, the order of each being less than or equal to some finite integer k. Let fo be a reduced atomic formula of maximal order appearing the disjuctive form of f . Then, there is a state ω ∈ Ef including f and kif , and another ω ∈ Ef including f and ¬kif . Thus, E = Ef cannot be self-evident. We conclude that nothing in the image of S is common knowledge except itself. The Aumann common knowledge theorem is true, but trivial. It may be thought that this result is due to the poverty of the semantic system S∗. To see that this is not the case, we can interpret S itself semantically without regard for the operator f → Ef , and simply say that a formula f is in an event E if f ∈ ω for every ω ∈ E. For any event E, ω ∈ KiE means Piω ⊆ E for each agent i, which means that every state ω ∈ for which κi(ω) = κi(ω ) satisfies ω ∈ E. However, if f is a reduced atomic formula in ω of order r terminating in ki, then

15 there are states ω and ω that agree with ω up to atomic formula of order r, and there are atomic formulae f and f of order r + 1 such that f ∈ ω , f ∈ ω , and f terminates in kj while f terminates in ¬kj for any j = i. Thus, if E is public, both f and f are in E. In other words, the fact that f ∈ Pi places no knowledge restrictions on other agents. Thus, E cannot satisfy a common knowledge condition unless E = .

8 The Two Astronomers: A Syntactic Analysis

We elucidate this point with a simple example. Let G be the set stars visible from tall buildings in New York City on a certain date and time. Let xs be the natural occurrence that the light from star s ∈ G appearing in the NewYork sky at this date and time indicates that s is in supernova. Deﬁne a state ω by the condition that, for all s ∈ G, xs ∈ ω if and only if xs is true. Let be the set of such states. Suppose two astronomers observing the night sky from two distinct tall buildings in New York city witness the same supernova of star s ∈ G. According to normal usage, we would say that each astronomer knows xs, and that the event Es ={ω ∈ |xs ∈ ω} is self-evident to each astronomer, because the supernova in question is the only one appearing at that time, place and date. However, neither astronomer knows that the other knows xs, because the two astronomers have absolutely no awareness of the communality of mental states between them. Thus, being self-evident for each astronomer does not imply common knowledge. There is thus something clearly non-tautologous about the deﬁnition of the knowledge partitions of . The syntactic model helps clarify this. Clearly, the semantic model underlying the two astronomer story is not the canonical semantic model S∗ analyzed in the previous section, which does not assert that ω ∈ E ⇒ kikj x ∈ ω. That is, the syntactic model does not assert that either astronomer knows that the other knows xs. The reason for this is that, despite the fact that there is only one natural occurrence xs, and both agents know that xs occurred, this fact (the event E)is not self-evident for either astronomer. To see this, consider the state ω ∈ E in which all reduced atomic formulas of order greater than 1 begin with ¬. Then, κ1(ω) consists of formulas in ω of the form k1f ,soκ1(ω) ={k1xs}. However, E1 ≡{ω ∈ |k1xs ∈ ω}⊆\ E, because E1 has states containing ¬k2xs. This analysis suggests that the semantic model of common knowledge is not an instrument for asserting the commonality of knowledge across agents. The notion that a mutually self-evident event entails common knowledge is correct, but the notion of a self-evident event, independent from the knowledge structure of other agents, already presumes common knowledge. Indeed, if we assume common knowledge in the syntactic model S for the two astronomers situation, the

16 continuum of states collapses to two states, and the model becomes isomorphic to Sx, the ‘pre-syntactic’ epistemic model for the problem. Moreover, the syntactic model S is superior to the semantic model in that the notion that “i knows that j knows…” is perfectly clear, but the system suggests no conditions under which this might or not be the case. Moreover, the syntactic model is impractical, replacing a straightforward ﬁnite-state semantic model with a complex construction of modal logic involving a continuum of non-constructible mental events.

9 The Commonality of Mental Constructs

Because the Harsanyi doctrine and the semantic common knowledge model are the only theoretical tools for asserting commonality of mental constructs across agents, and because neither is defensible, it follows that epistemic game theory provides no means of asserting the commonality of mental constructs across agents.

10 The Three Tactful Ladies: a Syntactic Analysis

While walking in the garden, Alice, Bonnie and Carole encounter a violent thun- derstorm and duck hastily into a nearby salon for tea. Carole notices that Alice and Bonnie have dirty foreheads, although each is unaware of this fact. Carole is too tactful to mention this embarrassing situation, which would surely lead them to blush, but she observes that, like herself, each of the two ladies knows that someone has a dirty forehead but is also too tactful to mention this fact. The thought occurs to Carole that she also might have a dirty forehead, but there are not detection devices handy that might help resolve her uncertainty. At this point, a little boy walks by the three ladies’ table and exclaims “I see a dirty forehead!” After a few moments of awkward silence, Carole sees that Alice is not blushing, and she sees that Bonnie is observing Alice and also is not blushing. Carole then realizes that she has a dirty forehead, and blushes. The problem is interesting because each of the three ladies already knew what the little boy told them, and indeed each knew that the others knew. It appears mysterious, then, that Carole can deduce anything after the little boy’s exclamation that she could not have decide before. The standard recount of Carole’s reasoning is as follows. Suppose Carole’s forehead is not dirty. Then, Bonnie would see that Carole’s forehead is not dirty. She would reason that if her own forehead were not dirty, then Alice would see two clean foreheads, and would know from the little boy’s comment that her forehead was dirty, and she would blush. Because she did not blush, Bonnie would know that her own forehead was dirty, and she would blush. Because Carole sees that Bonnie

17 did not blush, she concludes that her own forehead is dirty, and she blushes. This famous problem nicely puts the common knowledge framework through its paces. But, there are many unstated epistemological assertions going far beyond the common knowledge of rationality involved in the conclusion that Carole knows the state of her forehead. Let us see exactly what they are. Let xi be the condition that i has a dirty forehead, and let ki be the knowledge operator for i, where i = A, B, C, standing for Alice, Bonnie, and Carole, respec- tively. When we write i, we mean any i = A, B, C, and when we write i, j,we mean any i, j = A, B, C with j = i. Let yi be the condition that i blushes. The six symbols xi and yi represent the letters of a syntactic structure S, with state space . Let E be the event prior to the little boy’s exclamation b = xA ∨ xB ∨ xC. The statement of the problem tells us that xi ∈ E, and kixj ∈ E; i.e., each lady sees the forehead of the other two ladies, but not her own. The problem also asserts that kixi ⇒ yi ∈ E (a lady who knows she has a dirty forehead will blush), and yi ⇒ kj yi ∈ E (yi is a public signal). It is easy to check that these conditions are compatible with ¬kixi ∈ E; i.e., no lady knows the state of her own forehead at event E. These conditions also imply that kib ∈ E (each lady knows the little boy’s statement is true). While the problem intends that kCxA ⇒ kCkB xA ∈ E; i.e., if Carole knows that Alice has a dirty forehead, she then knows that Bonnie knows this as well, this implication does not follow from any axiom of the syntactical system, so we must include it as a new principle. We say a natural occurrence z is mutually accessible to i, j if kiz implies kikj z. The mutual accessibility of the xi to the other ladies may appear to be a weak assumption, but in fact it is the ﬁrst time we have made a substantive assertion that one agent knows that another agent knows something.

Let E be the state of knowledge following the exclamation b = xA ∨ xB ∨ xC.

Assuming xl is mutually accessible to i, j, it follows from b ∈ E that kj kib ∈ E ; i.e.,each lady knows that the other ladies know b. To see this, note for instance that xA ⇒ kCxA, so by the mutually accessibility of xA to C, B,wehavekCkB xA, which implies kCkB b. To prove that the occurrence of b as a public signal will lead Carole to blush, we must also assume that the yi are mutually accessible to all the ladies. This is because, in the above analysis, Carole infers that Bonnie knows that Alice does not blush from the fact that Alice does not blush. We say that a natural occurrence z is mutually accessible to order r if the occurrence of kiz entails the occurrence of kikj1 ...kjr z. In our problem, we must assume yi is public of order two. The fact than a natural occurrence z is a mutually accessible public signal says something about how agents know that they share the external world around them and how they transform perceptions in a parallel manner into knowledges. The concept of mutual accessibility is thus closely associated with the fact the agents know

18 that they share certain species-level sensory and information processing systems, and certain types of mental events are reliably connected to sensory experiences across individuals. We define a natural occurrence z to be an indicator of a mental state if z is both public and mutually accessible to all agents. The fact that z is an indicator of an internal state may be rooted in elementary physiology and sense perception. For instance, there are reliable, universal external events indicating fear, pain, joy, sleepiness, anger, and other primary emotions (i.e., emotions we share with many other vertebrate species). However, the property of being an indicator may also be socially specific. Blushing is a human universal (Brown 1991), but the mental states that lead to blushing are highly socially specific. Ladies blush when they have dirty foreheads in some societies, but not in others. Moreover, little boys generally do not blush when their foreheads are dirty. Where does the little boy’s exclamation b enter the analysis? We must assume b is public to some appropriate order, or the problem cannot be solved. Moreover, kib is true in E because xi is a public signal. Assuming xi is mutually accessible kj kib is also true in E. So, if b gives new information, it must be mutually accessible of order at least three. We now have the following argument. The reasoning following the little boy’s statement can be summarized as follows. Step 1: Carole assumes ¬xC and infers kA¬xC and kB ¬xC; Step 2: Bonnie assume ¬xB and concludes, using the fact that yi is a public event for j = i, that kA¬xB . Step 3: Suppose ¬xB and ¬xC are mutually accessible. then by assumption kB ¬xB , so kB kA¬xB , and also by assumption kB ¬xC,sokB κA¬xC. Because kB kA(xA ∨ xB ∨ xC),wehave

kB (kA(xA ∨ xB ∨ xC) ∧ kA¬xB ∧¬xC) ⇒ kB kAxA ⇒ kB yA.

However, yA is false, so kB yA is false. Thus Bonnie’s assumption that ¬xB is wrong, so she logically concludes xB , which means kB xB , and hence yB . Step 4: Because p is third-order public and assuming xi are also third order mutually accessible, Carole knows all of the above reasoning, and hence she knows that ¬xC implies yB . Because yB is false, she concludes that xC,sokCxC, which implies yC.

11 A Semantic Approach to Mutual Accessibility

We will construct a ﬁnite semantic epistemic model SK in which states directly incorporate the distribution of knowledge. Without the need for a syntactical model, this semantic model shows that unless we add some production of the form kif ⇒ kikj g, where both sides of the implication are reduced atomic formulas, we cannot conclude that any agent knows that another agent knows something.

19 We assume agents i = 1,...,n and mutually exclusive natural occurrences x1,...,xr . A state ω ∈ K is an n + 1 vector the ﬁrst component of which is a natural occurrence, and the remaining components of which are the truth values T and F for i = 1,...,n. We write ω0 = xk if the ﬁrst component of ω is xk, and ωi = T or ωi = F according as the i + 1 component of ω is T or F . We interpret ω0 = xk and ωi = T (resp. ωi = F ) to mean that xk occurred and i knows this (resp. i does not know this). We form partitions of K for the agents by specifying = = ={ ∈ | = ∧ = } that Piω K if ωi F , and Piω ω K ω0 ω0 ωi T . That is, i cannot distinguish among states at which he has no knowledge, and if he does know what occurred, he does not know who else knows that this occurred. Suppose E = K is a public event in SK . Then, for any i suppose ω ∈ KiE. Then ωi = T . This shows that E must be the single state {xk,T,...,T} for some ∈ ={ ∈ | = ∧ = }⊆\ natural occurrence xk. Then, for ω E, Piω ω E ω0 xk ωi T E. This shows that KiE =∅. This proves that the only public event in SK is the whole state space. This implies, of course, that the only self-evident event for an agent is also the whole state space. Now, suppose xk is mutually accessible to all agents, by which we mean that for all ω ∈ , ω0 = xr implies for all i, j, kixr ∈ ω implies kikj xr ∈ ω, which implies kj xr ∈ ω,soωi = ωj . Thus, at any state ω ∈ SK with ω0 = xr , the event xr is common knowledge.

12 Mutually Accessible Events and Symmetric Reasoning

Given the syntactic system S with agents i = 1,...,n, we have defined a set N of natural occurrences (letters of the alphabet X )tobemutually accessible if, for any i, j and any x ∈ N, kix ⇒ kikj x, We also defined x ∈ N to be a public signal at event E if x ∈ ω ∈ E implies kix ∈ ω for i = 1,...,n. We say event E is common knowledge in S at state ω if ω includes all atomic formulas of the form ki1 ki2 ...kir E for all orders r>0. Finally, we say i and j are symmetric reasoners with respect to a mutually accessible event A if for any event E, kiA ⇒ kiE implies kiA ⇒ kikj E. Before deploying these concepts in proving Theorem 3, it is well to pause to see what they mean. A mutually accessible event must be a natural occurrence pro- viding first-order sensory data to the agents involved. Thus, mental events are not mutually accessible, nor are inferences of mental events by virtue of overt behavior (e.g., “John believes he knows the answer” is not the type of event that could be mutually accessible, although “John looks ill” may be, and “John looks green” is even more likely to be mutually accessible). Moreover, the mutual accessibility of an event depends on the agents involved being aware that they all are receiving the same

20 sensory input, that all possess normal mental capacities, all are attentive to the event, etc. Mutually accessible natural occurrences are the minimum irreducible transmission mechanisms of mental constructs across minds.10 In humans, only sight and sound give rise to reliably mutually accessible events. It is unclear whether mutually accessible events occur in other species, although there is (disputed) evidence that chimpanzees share the capacity to recognize mutually accessible events (Premack and Woodruff 1978, Heyes 1998, Tomasello 1999, Tomasello et al. 2005). The concept of symmetric reasoning allows agent i to infer from the fact that he shares mutually accessible events with agent j, that both i and j will engage in a parallel sequence of mental activities in transforming the data that they share into further knowledges, so they will share certain mental constructs derived from the parallel processing of the same information. For instance, if an event E is mutually accessible, so i knows that j knows E and conversely, i might reason that j knows that i knows E, and the therefore i might know that j knows that they share the knowledge that i knows that j knows E. Of course, the symmetric reasoning assumption is very strong and will often fail to apply, because it may be the case that several people witness the same natural occurrences and draw extremely heterogeneous conclusions therefrom (the ‘Rashamon Effect’).11

Theorem 3. Consider the syntactic system S on letters X with agents i = 1,...,n. Let N ⊆ X be a set of natural occurrences that are public signals and mutually accessible at ω. If the agents are symmetric reasonsers with respect to each x ∈ N, then N is common knowledge at ω.

Proof: Choose an x ∈ N. Then, if x ∈ ω, ω includes the ﬁrst-order atomic formulas kix because x is mutually known. Because x is mutually accessible, ω contains the second order atomic formulas {kikj x} for all i, j. The proof follows by induction. ∈ Suppose ω contains all r-order atomic formulas ki1 ki2 ...kir x. For all i, kix ω ∈ implies kiki2 ...kir x ω, so because agents are symmetric reasoners with respect ∈ ∈ + to x, for any j, kix ω implies kikj ki2 ...kir x ω. Hence, all (r 1)-order atomic formulas of the form kikj ...klx ∈ ω.

10The relationship between natural occurrences and the ‘sense data’ of the logical positivists (Wittgenstein 1999[1921]) is close. However, I do not grant natural occurrences the epistemological primacy accorded them in logical positivist thought. Such occurrences are simply the ﬁrst links in a chain of constructs that jointly explain common knowledge. 11The term “symmetric reasoning” is deﬁned in Vanderschraaf and Sillari (2007), who attribute the term to personal communication with Chris Miller and Jarah Evslin. The concept is attributed to Lewis (1969).

21 13 Public Indicators and Social Frames

Let G be the event that the current social situation is a game G. G is not a natural occurrence and hence cannot be mutually accessible to the players of G.How- ever, mutual knowledge that G is being played is a condition for Nash equilibrium according to Aumann and Brandenburger’s (1995) Theorem B.12 How does G become mutually known? There may be a mutually accessible event F that reliably indicates that G is the case, in the sense that for any individual i, KiF ⊆ KiG (Lewis 1969, Cubitt and Sugden 2003, Vanderschraaf and Sillari 2007). We think of G as representing the game that is socially appropriate when the “frame” F occurs. For instance, if I wave my hand at a passing taxi in a large city, both I and the driver of the taxi will consider this an event of the form “hailing a taxi.” The underlying mutually accessible natural occurrences F constituting a frame for this game include the color of the automobile (yellow), the writing on the side of the automobile (“Joe’s Taxi”), and my frantic waving of a hand while looking at the automobile. When the driver stops to pick me up, I am expected to enter the taxi, give the driver an address, and pay the fare at the end of the trip. Any other behavior would be considered bizarre and perhaps suspicious. For instance if, instead of giving the driver an address, I invited the taxi driver to have a beer, or asked him to lend me money, or sought advice concerning a marital problem, the driver would consider the situation to be egregiously out of order. In many social encounters, there are mutually accessible cues F that serve as a frame indicating that a speciﬁc game G is being played, or is to be played. These frames are learned by individuals through a social acculturation processes. When one encounters a novel community, one undergoes a process of learning the particular mutually accessible indicators of social frames in that community. Stories of misunderstanding such indicators, and hence misconstruing the nature of a social frame is the common subject of amusing anecdotes and tales.13 We may summarize these concepts by deﬁning a frame F ⊆ as a public indicator of G for n individuals if F indicates G for all agents, and F is mutually accessible for all pairs of agents. We then have

Theorem 4. Suppose F is a public indicator of G and F is mutually accessible to all agents i = 1,...n. Then G is mutually known for all ω ∈ F .

12Knowledge of G is implicit in Theorem 2, because the state space is a public event, and hence mutually known. 13I am reminded of such an event that I experienced in an unfamiliar city, Shanghai. At rush hour, I went through our usual motions to hail a taxi, with no success—several available taxis simply passed on by. A stranger motioned to us to stand at a certain spot along the street and hail from there. Although this spot looked no different to me than any other spot on the street, a taxi pulled over almost immediately.

22 14 Public Indicators and Common Priors

As we have seen, there are plausible scientific grounds for assuming common priors concerning natural occurrences in some situations where the agents involved have had enough experience to make accurate judgments concerning the frequencies of occurrence. But, there is no plausible argument for common priors concerning events, such as mental constructs, beliefs, expectations, and the like, that are not natural occurrences. However, the Agreement Theorem (Theorem 1) asserts that once common priors concerning probabilities or expectations are achieved, the arrival of asymmetric information may transform the priors of all agents, but will not undermine these priors, provided the posteriors are common knowledge. Suppose we have a sizeable group of agents who have achieved common priors with respect to a certain state space. From time to time members of the group leave (e.g., through emigration or death) and new members arrive. We can assume that new members immediately assume the priors of the group, either because it is rational to do so (Conlisk 1988), or because humans are predisposed to conformist imitation (Henrich and Boyd 1998). Asymmetry of information may nevertheless undermine common priors, unless the resulting posteriors are common knowledge. How do posteriors become common knowledge? If social institutions provide mutually accessible signals, Theorem 3 shows how the posteriors can be common knowledge. There are several types of social institutions that serve as indicators of posterior probabilities. The oldest is doubtless that of gossip. Assuming that reputational knowledge is widespread and the penalties for spreading falsehoods sufficiently severe and predictable, gossip can effectively turn private information into public information (Varian 1990, Kandori 1992, Kollock 1999). In more complex societies, courts of law can pass judgment on an event to which very few parties are privy, the result being an indicator of a change in the reputation (as well, of course, as the freedom and wealth) of individuals under legal review. Finally, individuals can be publicly recognized has having acquired certain skills, knowledge and capacities, and thereby can be given certain public indicators of such attainments, including badges, special headgear, uniforms and costumes, or certificates of accomplishment. Social institutions produce what Durkheim called organic solidarity, which sociologists consider a key force in maintaining social cohesion and the efficiency of social exchange. Standard economic theory, including game theory, is completely incapable of analyzing the sources of common priors over events that are not natural occurrences. However, interactive epistemology provides the analytical tools for rigorously analyzing the implications of common priors (they are preconditions for the attainment of Nash and correlated equilibria by rational actors), and for formulating the conditions under which common priors can remain common in the

23 face of asymmetric information (the object of knowledge involved must satisfy the Sure Thing Principle, and the posteriors must be common knowledge).

15 Social Norms as Correlating Devices

A player i in an epistemic game G is rational at ω ∈ if his pure strategy ai(ω) ∈ Ai [ ] ω maximizes his payoff E πi(ai,a−i) , where a−i is distributed as φi , i’s conjecture as to the strategy profile of the other players. A correlated strategy of a game G with players i = 1,...,nand pure strategy sets Ai is a function γ : → A = A1 × ... × An, where is the finite set of possible states of G. Suppose the players have a common prior p ∈ ().A correlated strategy γ is a random variable on whose values are strategy profiles. We can identify a correlated strategy with a probability density p˜ on A, where p(a)˜ = p({ω|γ(ω)= a}). We call p˜ the distribution of the correlated strategy, and any probability distribution on A that is the distribution of some correlated strategy is called a correlated distribution. A mixed strategy for G is a correlated distribution corresponding to a correlated strategy γ whose marginal probabilities γi are statistically independent. Specifi- th cally, suppose γi :→Ai is the projection of γ onto its i component, and let p˜i be the probability density on Ai induced by γi. Then, γ induces a Nash equilibrium when p˜ =˜p1 × ...×˜pn. 1 k = Suppose γ ,...,γ are correlated strategies, and let α (α1,...,αk) be a ≥ = = i lottery (i.e., αi 0 and i αi 1). Then γ i αiγ is also a correlated strategy defined on {1,...,k}×. We call such an γ a convex sum of γ 1,...,γk. In particular, any convex sum of Nash strategies is a correlated strategy. If γ is a correlated strategy, then πi(γ ) is a real-valued random variable with an expected value Ei[πi(x)]. We say that a correlated strategy γ is a correlated equilibrium if for each player i, and any i-measurable function νi : → Ai,we have Ei[πi(γ )]≥Ei[πi(γ−i,νi)]. A correlated equilibrium induces a correlated equilibrium probability distribution on A, whose weight for any strategy profile a ∈ A is the probability that a will be played. Note that a correlated equilibrium of G is simply a Nash equilibrium of the game generated from G by adding Nature, whose move at the beginning of the game is to observe the state of the world ω, and to indicate a move γi(ω) for each player i such that no player has an incentive to do other than comply with Nature’s recommendation, provided that the other players comply as well. Moreover, while the players’ strategies are correlated, they are by construction independent, conditional on Nature’s signal.

24 The geometric structure of correlated equilibrium distributions of a finite game is actually much simpler than that of Nash equilibrium distributions. A convex polytope in Rn is the convex hull of a finite set of points, or equivalently, the set of points x ∈ Rn that satisfy a set of linear inequalities of the form Ax ≤ b, where A is an m×n matrix, m being the number of inequalities. The set of Nash equilibrium distributions of a game is the union of convex polytopes, while the set of correlated equilibrium distributions is a single convex polytope. While this is true for games with any finite number of players, we will illustrate this for two player games. Let A = A1 × A2, where Ai is the pure strategy set of player i = 1, 2. We can write a correlated strategy as σ = αij (a1i,a2j ), ij

th th where a1i is player 1’s i pure strategy and a2j is player 2’s j pure strategy. Let k th πij be the payoff to player k when player 1 plays his i pure strategy and player 2 plays his jth pure strategy. Then, a correlated strategy with probability weights {αij } is a correlated equilibrium if and only if

n1 1 − 1 ≥ = (πjk πqk)αjk 0 j,q 1,...n1 (23) k=1 n2 1 − 1 ≥ = (πjk πjr)αjk 0 k,r 1,...n2 (24) j=1 where ni is the number of pure strategies in Ai. Note that these conditions are indeed linear inequalities. The proof is straightforward, because the sums represent the different in payoff from using the strategy with weights {αij }, compared with using any other correlated strategy. Consider an n-player epistemic game G with players i = 1,...nand finite pure strategy sets Ai, and let A = A1 × ...× An. Let be the finite set of possible states of the game, and suppose i’s knowledge partition of ω is Pi. Finally, suppose there is a common prior p(·) on such that each player’s subjective prior pi(·; ω) satisfies pi(·; ω) = p(·|ai(ω)), where ai(ω) is i’s move at ω. A player i is rational at ω if ai(ω) maximizes E{πi(a−i,ai)|Pi}. We have (Aumann 1987)

Theorem 5. If the players have a common prior p(·), then the probability distribution of {a(ω|ω ∈ )} is a correlated equilibrium distribution.

25 To prove this theorem, we let be the probability space for a random variable γ :→A with probability distribution p(·), where we set γ(ω) = a(ω). Then, for any player i and any function νi :→Ai that is Pi-measurable (i.e., that is constant on cells of the partition Pi), because i is Bayesian rational implies

E[πi(a(ω))|Piω) ≥ E(πi(a−i(ω), νi(ω))|Piω].

Multiply both sides of this inequality by p(Piω) and adding over the disjoint cells Piω in Pi gives, for any such νi,

E[πi(a(ω))) ≥ E(πi(a−i(ω), νi(ω))].

This proves that (, γ (ω)) is a correlated equilibrium. Note that the converse clearly holds as well. Why should rational agents actually play a correlated equilibrium? Note that if there are multiple Nash equilibria, there are generally even more correlated equilibria, because the set of correlated equilibria includes the convex hull of the set of Nash equilibria. It appears that we have merely aggravated the problem of equilibrium selection by introducing correlated equilibria. However, the introduction of the correlated equilibrium concept replaces an impossible task—that of choosing among a plethora of Nash equilibria—with a realistic one—that of suggesting why players might have a common prior and iden- tifying the appropriate correlating device. When either of these two tasks cannot be accomplished, there is simply no reason to believe agents will play a Nash equilibrium, except in very simple games of complete information with a unique pure strategy equilibrium (e.g., the prisoner’s dilemma). Player priors in epistemic games express the probability of the various states ω ∈ of the game. Because a state of the game can be extremely complex, including not only the behavior of players, but also their conjectures, their beliefs concerning the beliefs of other players, their beliefs concerning the beliefs of the beliefs of other players, and so on. There is no plausible argument based on principles of Bayesian rationality that human beings should have common priors over events that are unobservable in principle, and with only the most meager reﬂection in actual strategic behavior (Gul 1998). In the presence of historical social regularities, however, common priors over behaviors make sense, for reasons given in the previous section. We thus assume that we are dealing with an epistemic game in which tradition, culture, common experience, social institutions, and other historical regularities are such that players share a common prior. ω If γ(ω)is a correlating device, and if player i’s conjecture φi (a−i) the conditional distribution of γ ,givenγi = ai, then Theorem 2 asserts that if these conjectures are common knowledge, they induce a Nash equilibrium (σ1(ω),...,σn(ω))

26 of G+, the game derived from G by adding the correlating device as a player that signals γ(ω)at the start of the game. Moreover, in the case of a correlated equilibrium, the correlating device specifies a pure strategy for each player, so Theorem 2 in this case can be strengthened to affirm a Nash equilibrium in strategies as well as in conjectures. This suggests that the solution to the equilibrium refinement problem in the case of epistemic games lies in finding a correlating device that is common knowledge. Social norms are a candidate for such a device. We say a game G that is played in state G is norm-governed if there is a social norm γ = N (G) that specifies socially appropriate behavior γ(ω)⊆ A, γ(ω) =∅, for each state ω ∈ , where A is the strategy profile for an epistemic game. Note that we allow appropriate behavior to be correlated. Suppose G is an n player game. How does common knowledge of a social situation G affect the play of the game? The answer is that each player i must associate a particular social norm N (G) with G that determines appropriate behavior in the game, i must be confident that other players also associate N (G) with G, i must expect that others will choose to behave appropriately according to N (G), and behaving appropriately must be a best response for i, given all of the above. Suppose G indicates N (G) for i = 1,...,n because the players belong to a society in which, when a game G is played and event G occurs, then appropriate behavior is given by N (G). Suppose the players are symmetric reasoners with respect to G in the sense that if G is common knowledge for i = 1,...n, then all i associate N (G) with G, and i knows that j = 1,...nassociates N (G) with G. Then, similar reasoning to Theorem 3 shows that N (G) is common knowledge. We say an individual is prosocial if he always chooses socially appropriate behavior when it is costless to do so. We then have

Theorem 6. Given epistemic game G with prosocial players i = 1,...,n, suppose G is common knowledge and G indicates social norm N (G) for all players, who are symmetric reasoners with respect to G. Then, if appropriate behavior according to N (G) is a correlated equilibrium for G, the players will choose the corresponding correlated strategies.

16 Example

27 s1 s2 Consider the coordination game G with normal form matrix shown to the right. There are two pure strategy equilibria: s1 2,2 0,0 (2,2) and (1,1). There is also a mixed-strategy equilibrium s2 0,0 1,1 with payoffs (1/3,1/3), in which players choose s1 with probability 1/3. There is no principle of Bayesian rationality that would lead the players to coordinate on the higher-payoff equilibrium. Moreover, suppose they discussed the game before playing it, and they assured one another that they would each play s1. There is still no principle of Bayesian rationality that would lead the players to choose the (2,2) equilibrium, because there is no principle of Bayesian rationality that favors truth-telling over lying, or that offers a reason why there might be any value in keeping one’s promises in one-shot games. There are two obvious social norms in this case: “when participating in a pure coordination game, choose the strategy that gives players the maximum (respec- tively, minimum) common payoff.” The following is a plausible social norm that leads to a convex combination of the two coordinated payoffs. There are two neigh- boring tribes whose members produce and trade apples and nuts. Members of one tribe wear long gowns, while members of the other tribe wear short gowns. Individ- uals indicate a willingness to trade by visually presenting their wares, the quality of which is either 1 or 2, known prior to exchange to the seller but not the buyer. After exchanging goods, both parties must be satisﬁed or the goods are restored to their original owner and no trade is consummated. The social norm N that governs exchange is “never try to cheat a member of your own tribe, and always try to cheat a member of the other tribe.” When two individuals meet, the visibility of their wares, F , represents a mutually accessible natural occurrence that frames the game G, which is thus mutually known. Both traders know that F indicates the social norm N , conforming to which is a best response for both parties. If both have long robes or short robes, N indicates that they each give full value 2, while if their robe styles differ, each attempts to cheat the other by giving partial value 1. Because either trade is better than no trade, a trade is consummated in either case. The expected payoff to each player is 2p + (1 − p), where p is the probability of meeting a trader from one’s own tribe.

17 Conclusion

Throughout much of the previous century, economic theory placed little value on the commonality of beliefs and social norms that are the stock in trade of sociological theory. The Walrasian general equilibrium model requires few restrictions on the heterogeneity of preferences, and admitted no role for individual beliefs and social norms. The increased importance of game theory in the last quarter of the

28 century presented serious conceptual issues concerning the relationship between rationality and equilibrium criteria. Interactive epistemology in general, and epistemic game theory in particular, have shown that the commonality of beliefs, in the form of common priors, common knowledge, and correlating devices, is central in explaining how rational actors successfully coordinate their activities. Yet, economists have avoided grappling with the question of where common priors, common knowledge, and correlating devices come from. Harsanyi’s doctrine, asserting that all differences in subjective priors of rational individuals is due to asymmetric information has a degree of plausibility in dealing with we have called (following Aumann 1999) “natural occurrences,” but not otherwise. Seman- tic knowledge models (Aumann 1987) assert that mutual self-evidence logically implies common knowledge, but as we have seen, they achieve this by making im- plausible tacit assumptions concerning the structure of knowledge. UsingAumann’s (1987) syntactic knowledge model, we have shown that explicit epistemological assumptions concerning the sharing of knowledge among individuals are required to prove common knowledge. There is no way to deduce commonality of mental states across individuals based on the rational actor model. This conclusion has been avoided because it conﬂicts with a principle rarely explicitly stated, but deeply embedded in the economist’s view of society. This is the view that the terms used to explain social phenomena must refer solely to properties of individuals, and is known as methodological individualism. I have argued that further progress in epistemic game theory requires that we recognize that this principle has no empirical support.14 In particular, social norms cannot be explained in terms of the characteristics of individuals, rational or otherwise. This does not mean that our understanding of social norms is not deepened and illuminated by their relationship to rational choice. Quite the contrary. I have shown in this paper that this concept is intimately related to several of the most powerful principles of rational choice theory. However, social norms cannot be explained in terms of these principles alone. The depiction of social norms in this paper is radically simpliﬁed, for the pur- pose of highlighting one aspect of social norms: their articulation with Bayesian rationality. This perspective is shared with such works as Binmore (2005) and Greif (1994). Social norms, however, also link the sphere of economic transactions with the larger sphere of social transactions within which they are embedded, thus helping attenuate the negative externalities often associated with market transactions (Ostrom 1990, Granovetter 1985, Aoki 2007). Social norms also affect the

14Methodological individualism is rarely defended in terms of explanatory power, but rather as a welcome alternative to methodological holism, which rejects micro-level analysis altogether. The idea that social norms are emergent properties of complex systems is not a form of holism, and of course does not reject micro-level analysis.

29 balance between self- and other-regarding behaviors (Bicchieri 2005). Ultimately these larger issues, along with a theory of how social norms evolve (Bowles 2004) must be integrated into an overall picture of the interaction of rational choice and social norms (Elster 1989).

References

Andreoni, James and John H. Miller, “Giving According to GARP: An Experi- mental Test of the Consistency of Preferences for Altruism,” Econometrica 70,2 (2002):737–753. Anscombe, F. and Robert J. Aumann, “A Deﬁnition of Subjective Probability,” Annals of Mathematical Statistics 34 (1963):199–205. Aoki, Masahiko, “Linking Economic and Social-Exchange Games: From the Com- munity Norm to CSR,” 2007. Stanford University and VCASI. Aumann, Robert J., “Agreeing to Disagree,” The Annals of Statistics 4,6 (1976):1236–1239. , “Correlated Equilibrium and an Expression of Bayesian Rationality,” Econo- metrica 55 (1987):1–18. , “Common Priors: A Reply to Gul,” Econometrica 66,4 (1998):929–938. , “Interactive Epistemology I: Knowledge,” International Journal of Game Theory 28 (1999):264–300. andAdam Brandenburger, “Epistemic Conditions for Nash Equilibrium,” Econo- metrica 65,5 (September 1995):1161–80. Bacharach, Michael, “The Acquisition of Common Knowledge,” in Cristina Bic- chieri and Maria Luisa Dalla Chiara (eds.) Knowldge, Belief, and Strategic In- teraction (Cambridge: Cambridge University Press, 1992) pp. 285–316. Becker, Gary S. and George J. Stigler, “De Gustibus Non Est Disputandum,” Amer- ican Economic Review 67,2 (March 1977):76–90. Bicchieri, Cristina, The The Grammar of Society: The Nature and Dynamics of Social Norms (Cambridge: Cambridge University Press, 2005). Binmore, Ken and Adam Brandenburger, “Common Knowledge,” in Ken Binmore (ed.) Essays on the Foundations of Game Theory (Oxford: Basil Blackwell, 1990) pp. 105–150. Binmore, Kenneth G., Natural Justice (Oxford: Oxford University Press, 2005). Boudon, Raymond, “Beyond Rational Choice Theory,” Annual Review of Sociol- ogy 29 (2003):1–21. Bowles, Samuel, Microeconomics: Behavior, Institutions, and Evolution (Prince- ton: Princeton University Press, 2004).

30 Boyd, Robert and Peter J. Richerson, Culture and the Evolutionary Process (Chicago: University of Chicago Press, 1985). Brown, Donald E., Human Universals (New York: McGraw-Hill, 1991). Cavalli-Sforza, Luigi L. and Marcus W. Feldman, Cultural Transmission and Evo- lution (Princeton, NJ: Princeton University Press, 1981). Collins, John, “How We Can Agree to Disagree,” 1997. Department of Philosophy, Columbia University. Conlisk, John, “Optimization Cost,” Journal of Economic Behavior and Organiza- tion 9 (1988):213–228. Cubitt, Robin P. and Robert Sugden, “Common Knowledge, Salience and Conven- tion: A Reconstruction of David Lewis’ Game Theory,” Economics and Philos- ophy 19 (2003):175–210. Durham, William H., Coevolution: Genes, Culture, and Human Diversity (Stanford: Stanford University Press, 1991). Durkheim, Emile, The Division of Labor in Society (New York: The Free Press, 1933[1902]). Elster, Jon, “Social Norms and Economic Theory,” Journal of Economic Perspec- tives 3,4 (1989):99–117. Fehr, Ernst and Simon Gächter, “Altruistic Punishment in Humans,” Nature 415 (10 January 2002):137–140. Geanakoplos, John, “Common Knowledge,” Journal of Economic Perspectives 6,4 (Fall 1992):53–82. Gintis, Herbert, “A Framework for the Uniﬁcation of the Behavioral Sciences,” Behavioral and Brain Sciences 30,1 (2007):1–61. , Samuel Bowles, Robert Boyd, and Ernst Fehr, Moral Sentiments and Material Interests: On the Foundations of Cooperation in Economic Life (Cambridge: The MIT Press, 2005). Gneezy, Uri, “Deception: The Role of Consequences,” American Economic Review 95,1 (March 2005):384–394. Granovetter, Mark, “Economic Action and Social Structure: The Problem of Em- beddedness,” American Journal of Sociology 91,3 (November 1985):481–510. Greif, Avner, “Cultural Beliefs and the Organization of Society: An Historical and Theoretical Reﬂection on Collectivist and Individualist Societies,” Journal of Political Economy 102,5 (October 1994):912–950. Gul, Faruk, “A Comment on Aumann’s Bayesian View,” Econometrica 66,4 (1998):923–928. Harsanyi, John C., “Games with Incomplete Information Played by ‘Bayesian’ Players,” Management Science 14 (1967–68):159–182,320–324,486–502.

31 Hart, Sergiu, Aviad Heifetz, and Dov Samet, ““Knowing Whether,” “Knowing That,” and The Cardinality of State Spaces,” Journal of Economic Theory 70 (1996):249–256. Hedman, Shawn, A First Course in Logic: An Introduction to Model Theory, Proof Theory, Computability, and Complexity (Oxford: Oxford University Press, 2004). Henrich, Joseph and Robert Boyd, “The Evolution of Conformist Transmission and the Emergence of Between-Group Differences,” Evolution and Human Behavior 19 (1998):215–242. Heyes, C. M., “Theory of Mind in Nonhuman Primates,” Behavioral and Brain Sciences 21,1 (1998):101–034. Kandori, Michihiro, “Social Norms and Community Enforcement,” Review of Eco- nomic Studies 59,1 (January 1992):63–80. Kollock, Peter, “The Production of Trust in Online Markets,” in E. J. Lawler, M. Macy, S. Thyne, and H. A. Walker (eds.) Advances in Group Processes (Vol. 16) (Greenwich, CT: Jai Press, 1999) pp. –. Kreps, David M., Notes on the Theory of Choice (London: Westview, 1988). Lewis, David, Conventions: A Philosophical Study (Cambridge, MA: Harvard University Press, 1969). Mas-Colell, Andreu, Michael D. Whinston, and Jerry R. Green, Microeconomic Theory (New York: Oxford University Press, 1995). Maynard Smith, John and Eors Szathmary, The Major Transitions in Evolution (Oxford: Oxford University Press, 1997). Milgrom, Paul, “An Axiomatic Characterization of Common Knowledge,” Econo- metrica 49 (1981):219–222. Morowitz, Harold, The Emergence of Everything: How the WorldBecame Complex (Oxford: Oxford University Press, 2002). Morris, Stephen, “The Common Prior Assumption in Economic Theory,” Eco- nomics and Philosophy 11 (1995):227–253. Ostrom, Elinor, Governing the Commons: The Evolution of Institutions for Col- lective Action (Cambridge, UK: Cambridge University Press, 1990). Parsons, Talcott, Sociological Theory and Modern Society (New York: Free Press, 1967). Polak, Ben, “Epistemic Conditions for Nash Equilibrium, and Common Knowledge of Rationality,” Econometrica 67,3 (May 1999):673–676. Premack, D. G. and G. Woodruff, “Does the chimpanzee have a Theory of Mind?,” Behavioral and Brain Sciences 1 (1978):515–526. Savage, Leonard J., The Foundations of Statistics (New York: Wiley, 1954).

32 Tomasello, Michael, The Cultural Origins of Human Cognition (Cambridge, MA: Harvard University Press, 1999). , Malinda Carpenter, Josep Call, Tanya Behne, and Henrike Moll, “Understanding and Sharing Intentions: The Origins of Cultural Cognition,” Behavioral and Brain Sciences 28,5 (2005):675–691. Vanderschraaf, Peter and Giacomo Sillari, “Common Knowledge,” in Edward N. Zalta (ed.) The Stanford Encyclopedia of Philosophy 2007). Varian, Hal R., “Monitoring Agents with Other Agents,” Journal of Institutional and Theoretical Economics 46,1 (1990):153–8174. Wittgenstein, Ludwig, Tractatus Logico-Philosophicus (New York: Dover, 1999[1921]).

c\Papers\Modal Logic and Social Norms\Bayesian Rationality and Social Knowledge November 13, 2007