Maintaining Trust When Agents Can Engage in Self-Deception

Maintaining trust when agents can engage in self-deception

Andres´ Babinoa,b,1,2, Hernan´ A. Maksec,d,1, Rafael DiTellae,f,g,1, and Mariano Sigmanh,1

aDepartamento de F´ısica J.J. Giambiagi, Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires, Buenos Aires 1428, Argentina; bInstituto de F´ısica de Buenos Aires, Consejo Nacional de Investigaciones Cient´ıﬁcas y Tecnicas´ (CONICET), Buenos Aires 1428, Argentina; cLevich Institute, City College of New York, New York, NY 10031; dPhysics Department, City College of New York, New York, NY 10031; ePolitical Economy Group, National Bureau of Economic Research, Cambridge, MA 02138; fSocial Interactions, Identity and Well-being Program, Canadian Institute for Advanced Research, Toronto, ON M5G 1M1, Canada; gGovernment and the International Economy Unit, Harvard Business School, Boston, MA 02163; and hLaboratorio de Neurociencia, CONICET, Universidad Torcuato Di Tella, C1428BIJ Buenos Aires, Argentina

Edited by Albert-Laszl´ o´ Barabasi,´ Northeastern University, Boston, MA, and accepted by Editorial Board Member David A. Weitz July 17, 2018 (received for review February 28, 2018)

The coexistence of cooperation and selfish instincts is a remark- of Projection by which our actions affect how we think of oth- able characteristic of humans. Psychological research has unveiled ers (11, 12) is at the same time intuitive and paradoxical. From the cognitive mechanisms behind self-deception. Two important a rational perspective, beliefs about others should be based on findings are that a higher ambiguity about others’ social pref- what they have done, not on what we have done to them. How- erences leads to a higher likelihood of acting selfishly and that ever, it has been observed that subjects in economic games not agents acting selfishly will increase their belief that others are only take into account the previous actions of other players, but also selfish. In this work, we posit a mathematical model of also their past actions (13, 14). Additionally, people’s beliefs also these mechanisms and explain their impact on the undermin- depend on their own previous actions (10). ing of a global cooperative society. We simulate the behavior of Here, we use the colloquial term Paranoia to refer to P2 (the agents playing a prisoner’s dilemma game in a random network idea that if there is ambiguity about how another person may of contacts. We endow each agent with these two self-deception act, an agent will sample the distribution biased for the worse mechanisms which bias her toward thinking that the other agent outcomes). Closely related to P2 is the mechanism of “catego- will defect. We study behavior when a fraction of agents with the rization” and “malleability” (15). For example, stealing a pen is “always defect” strategy is introduced in the network. Depending more malleable than stealing the money needed to buy the pen. on the magnitude of the biases the players could start a cascade of Similarly, the distribution of beliefs on the moral judgment of the defection or isolate the defectors. We find that there are thresh- malleable case (stealing the pen) is ambiguous, and hence people olds above which the system approaches a state of complete may use this ambiguity in their favor to act more selfishly. distrust. The aim of this work is twofold: first, to provide a mathematical description of these self-deception mechanisms (Para- behavioral economics | cognitive neuroscience | corruption | noia and Projection); and second, based on this mathematical cooperation | self-deception description, to investigate the impact that they may have on the evolution of trust among the agents of a society. ndividuals often deviate from the behavior that maximizes The Model Itheir material reward (1, 2). For example, in the ultimatum We study a set of 105 interacting agents that play a modified Pris- game, people prefer to reject profitable offers that they consider oner’s Dilemma (SI Appendix, section 1.1) game against each unfair (3). This behavior, and other phenomena such as fairness other in a static random network. The main difference from or cooperation (2, 4), can be accounted for within a rational other similar approaches investigating networks and evolution of model that includes broader objectives or “social preferences” (altruism, fairness concerns, etc.) as part of the function which agents seek to optimize. Significance Naturally, agents seek to reduce the problems that arise when material rewards collide with social preferences. For example, “He who wants to kill his dog accuses it of having rabies,” believing that others are altruistic may make it more difficult for the French proverb says. The fact that we alter our beliefs an agent to act selfishly which, in turn, may reduce its monetary about others to act selfishly and, at the same time, keep a payoff. A way of solving this tension is to develop a self-serving positive self-view has been widely studied by behavioral sci- bias: that is, to believe that others are not altruistic to “justify” ences. Here, we propose a mathematical description of two of a selfish act. Cognitive dissonance theory (5, 6) aims to explain these mechanisms of altering beliefs and study a simulation of the emergence of belief with self-serving biases. The idea is a society of agents provided with these biases. We find that that dissonance (contradiction) between cognitions is psychologi- there are sets of parameters that make societies propagate cally uncomfortable, and so it triggers mechanisms of dissonance defection actions and others that protect them from spreading reduction—and one way of doing so is by altering beliefs (7, 8). malicious behavior. Self-deception mechanisms have been broadly studied in eco- Author contributions: A.B., H.A.M., R.D., and M.S. designed research; A.B. performed nomics (2, 9). Recently, using an experimental design called research; A.B. analyzed data; and A.B., H.A.M., R.D., and M.S. wrote the paper. “The Corruption Game,” we demonstrated two of these prin- The authors declare no conflict of interest. ciples (10): This article is a PNAS Direct Submission. A.-L.B. is a guest editor invited by the Editorial Principle 1 (P1) Selfish action alters beliefs about others’ social- Board. preferences. Published under the PNAS license. P2 Ambiguity regarding others’ social preferences 1 A.B., H.A.M., R.D., and M.S. contributed equally to this work.y increases the likelihood of acting selfishly. 2 To whom correspondence should be addressed. Email: [email protected] This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10. We use the term Projection to refer to P1, which is a trait that 1073/pnas.1803438115/-/DCSupplemental. describes how people blame others for their actions. The notion Published online August 13, 2018.

8728–8733 | PNAS | August 28, 2018 | vol. 115 | no. 35 www.pnas.org/cgi/doi/10.1073/pnas.1803438115 Downloaded by guest on September 28, 2021 Downloaded by guest on September 28, 2021 aioe al. et Babino eto 1.2 section Appendix, agent the not, agent. other if the cooperate; of estimates observations agent will An and defect. to player choose will other the trust will ie neato—reuvlnl,tepoaiiyo coopera- defection, of of probability probability the tion equivalently, agent’s interaction—or an given of estimation the these inference how Bayesian explain a we into subsection, incorporated process. be next can the biases In cognitive network. the (Paranoia in introduction incorporat- the how in described investigate and biases to cognitive is the goal ing primary initial the Our on depending conditions. other each with cooperating or defecting chooses reward. agent expected her the each maximizes stage, for that decision option reward the the expected in the Finally, is action. process possible inference the of output h xetdrwr fec cin ntedcso tg,teaetchooses agent the stage, decision options. the her in among action, each possible by of each reward biased expected of be used the reward could is expected reward agents) expected the This other estimate action. to the process of inferential actions the of cooperation in decisions and the defection from the of also of but result number agents, a as other agent of own actions the the with fed is stage 1. Fig. belief their parameter: of this value of mean estimation the an use as Agents variable. this of bution 1.3 section Appendix, (SI process mod- beta-binomial is Bayesian a stage a as This behavior. generate—using eled agents’ to other informa- priors about belief observed with model—a it uses other combines about process and information inference tion of The Observa- accumulation actions. of decision. agent’s process and the inference, is observation, tion 1): (Fig. stages strategies. mathematical of a propagation the generate on to impact necessary biases was cognitive of rule (similar model process This inference 20). here, and ref. rule that to updating is Bayesian (16–19) a reputation) used we or corruption (or cooperation ntebt-ioilmdl the model, beta-binomial the In h seta tpi h neeta rcs normdlis model our in process inferential the in step essential The up end will network the in agents the settings, these Under o lrt,w iietesrtg fteaet nothree into agents the of strategy the divide we clarity, For fetteeouino oprto rdefection or cooperation of evolution the affect Projection) p c ktho h eiinmkn rcs fec gn.Teobserving The agent. each of process decision-making the of Sketch 1 = − θ fteetmto of estimation the If . sue odsrb h ro eifdistri- belief prior the describe to used is ) Paranoia Projection and θ is h bevdeiec (the evidence observed The bias. ssaleog,teagent the enough, small is Projection θ Beta ial,bsdon based Finally, Paranoia. ae nhrprevious her on based itiuin(SI distribution n td their study and θ .The ). na in , where nodro antd fraitcvle of values realistic of magnitude of order an of can value we experimental this, From the (see belief. of the estimate in an 0.2 have of variation average an duces beliefs). to the this call We correspondingly. cooperates, symmetric or defects agent asymmetric this the call We defects. agent the when First, others that likely of more value cooperation. is the the on it agent case, variation this that the the In think its when too. to changes used cooperate it others be cooperates, about could agent belief estimated mechanism an its when same on cooperates—namely, This effect on. same the so have If would defects. agent Beta the agent an on of other impact tions the same parame- if the the than by has scaled agent is another defects agent the ter time each the changed of defections) observed the defects, cooperation. or defection of probability the estimate resulting the this beta on her impact With model modifies beliefs. she to of defects, observations, agent distribution an and this time constructs each priors characteristic, observer from ideal an only Although distribution actions. their for and ers actions), observed distribution. beta of the fragments of outcomes) were actions Paranoia own if (as Broadly, process. ence with. interacts belief specific she a agent has other likely agent each is each for model, agent distribution this other In the defect. that to believes choose to agent the that indicating 1, As cooperations. and tions fo eif oatos n ythe by and actions) estimation Bayesian to explic- a beliefs model using (from Our causation two-way belief. this alter ways: describes actions both itly and in corre- actions, is only alters causation not this Belief is Surprisingly, there causation. experimentally, but is shown model lation our been with has belief difference as the main our The with that, in too. associated selfish as are are acts way others selfish same that the the utility is, in That general (10). correlate more model action others’ a and and into beliefs fairness incorporated function, where models are in preferences fact, social In optimal. not is tion for value wrong the of behave. His all will defect. how with to interact regarding A will likely beliefs he Similarly, more his agents defect. other are changed to has agents likely defection other more own all is of that B effect believe the that will to believe due will A B, agent A with if the tion, interaction example, only given For not a interaction. in agents, specific defects other the all in about involved belief one the describe that ters nrf 0 esoe htteefc fadfcigato pro- action defecting a of effect the that showed we 10, ref. In the cases, both In ti motn ohglgtta,ee huhti isyed a yields bias this though even that, highlight to important is It eepoetodfeetvrain fthe of variations different two explore We agent the whenever changing, by done is this Specifically, Projection Paranoia Projection Projection eto 1.4 section Appendix, SI itiuinta eeto yteohrpae,and player, other the by defection a than distribution a and eut nsmln nvny(ouigo h worse the on (focusing unevenly sampling in results Projection. and b satatta ecie o epebaeoth- blame people how describes that trait a is a fet oh the both, affects nsc a htif that way a such in r,rsetvl,tenmeso bevddefec- observed of numbers the respectively, are, θ ˆ Projection Projection aaee wihmaue h ubrof number the measures (which parameter htde o ml htti ido computa- of kind this that imply not does that , PNAS θ ˆ b = Projection aaee fthe of parameter Beta Z | Projection 0 uut2,2018 28, August 1 a aeadfeetefc nteinfer- the on effect different a have θ fet nythe only affects nrae,rsetto respect increases, Beta Beta Projection itiuin hc h hnue to uses then she which distribution, o eal) hsrsl evsas serves result This details). for a iscagsthe changes bias n the and θ hne h eadistribution beta the changes itiuin o much How distribution. (a Projection Projection Projection , b eod when Second, Projection. Beta ) h cin fteagent the of actions the | d b o.115 vol. Projection aaeesatrthe after parameters , θ , a Projection itiuinatra after distribution Beta Projection is(rmaction (from bias 1 = aaee fthe if parameter 0 = Projection b | Beta , 0defec- 10 .1, θ ˆ eetn on defecting o 35 no. distribution approaches determines hnwe when parame- Projec- f0.54 of | bias. a 8729 is

PSYCHOLOGICAL AND COGNITIVE SCIENCES inquire about the impact of this parameter on the propagation of cooperation and defection. The other bias that we explore, Paranoia , acts on how θˆ is estimated from a distribution. If Paranoia = 0, the estimation of θ is the mean value of the variable, as it would be in an optimal inference process. When Paranoia assumes a nonzero value (here, we only investigate positive values), the estimation is given by this implicit equation.

Z θˆm Betaθ(a, b) dθ = Paranoia. [1] θˆ

ˆ Fig. 2. The active fraction changes as the Projection bias increases when This equation means that the agent shifts its estimate from θ only one ALLD agent is present in the network. The point marked with a to θˆm . The value of Paranoia measures the total area under dashed line is the theoretical value at which the biggest cluster of vulnerable the probability distribution between the optimal and biased esti- nodes percolates the network. As predicted, this value also indicates the mates θˆ and θˆm . This equation also implies that the impact transition from zero to positive values of Sa. of Paranoia on the estimation of θˆm depends on the shape of the probability distribution. If there is substantial evidence of nerability of the agents only if the value of the Paranoia is something, that will be reflected in a narrow and peaked dis- greater than zero. Then, a similar behavior is observed (SI tribution shape, and the effect of Paranoia on θˆm will be weak. Appendix, section 1.7). Instead, if the distribution is wide, meaning that there is insuf- In the case of the asymmetric bias, this value, Projection = 1.5, ficient evidence, this same amount of probability will cause a can be derived analytically following the method of Watts (22), higher distortion in the estimated belief. This mechanism, then, and the result is in agreement with the numerical simulation (SI reproduces the empirical fact that higher ambiguity increases the Appendix, section 1.6). likelihood of being selfish (P2). Note also that Paranoia is measured in units of the probability distribution (and hence has one Percolation and Phase Transitions. Now, we generalize this analysis as absolute maximum). to a broader situation, where not only one agent, but a frac- This bias is closely related to the models of reciprocal altru- tion, f, of ALLD agents are present in the network. We examine ism (10, 21) where belief may be altered too. In these models, the robustness of the system by analyzing its evolution as two changing the belief has a cost that increases as the differ- parameters are changed: the fraction, f , and the value of the Pro- ence between the unbiased and the biased belief increases. jection parameter. We calculate the robustness of the network The Paranoia bias could be thought of as step function where measuring Sa . Here, because there is more than one ALLD, there is no cost in changing a belief up to some point where we also incorporate a more refined measure of phase transition the cost, suddenly, approaches infinity. The explicit descrip- referred to as the size of the giant active component, Sgc . Sgc tion of the belief as the mean of a distribution allows us to measures the size of the largest cluster of agents that are defect- model P2. ing to each other. If there is only one agent, Sa and Sgc are equivalent. Results Fig 3 shows Sgc in the (Projection, f ) map for the asymmetric We study how the network evolves when it is contaminated with a version of the bias. It can be seen that the value of Sgc increases fraction of agents with the strategy ALLD (always defect). These as the value of f or Projection increases. To better understand agents do not learn or change their behavior in any way; they the transitions in this map, Fig. 4 shows the Sgc as a function of f stubbornly defect independently of the history of actions. for three values of Projection for the asymmetric bias. All simulations begin with a network in which agents trust When Projection = 0, the ALLD agents do not change the each other. That is, they believe that the expected reward of behavior of the regular agents of the network toward other cooperating is higher than the expected reward for defecting agents. Then, the change in Sgc is due only to the fact that (Fig. 5). Then, we replace a fraction of the regular agents of the more ALLD agents are defecting in the network. Marked with network by ALLD agents. These replacements are distributed an arrow in Fig. 4, for this case, a continuous transition can be at random in the sites of the network. Specifically, the ques- seen at f = fc1. If f < fc1, the value of Sgc is zero and if f > fc1 tion we ask here is how the network parameters convey more the value of Sgc take positive values. In this situation, the pro- resistance or vulnerability to this “infection” process. To do cess can be mapped into a standard percolation (SI Appendix, so, we let the network evolve under the influence of ALLD section 1.8). The analytical result yields the value fc1 = 0.05, agents and study whether the defection policy extends over which is in agreement with the numerical simulations, as can be the network. seen in Fig. 4. If the value of Projection is greater than zero, the actions of Cascades. First, we study how the system evolves when one the ALLD agents change the belief of the regular agents in the ALLD agent is introduced in the network. Under this condi- network. In Fig. 4, it can be seen that when Projection increases, tion, we measure the fraction of agents that are defecting to the value of Sgc increases and the value of the critical point fc1 each other, which is called the active fraction, Sa . Fig. 2 shows moves toward zero. how the Sa changes as Projection changes when we use the Interestingly, if the value of Projection ≥ 0.8, another kind of asymmetric version of the bias. There is a transition in the transition appears in the system. This second transition at f = fc2 value Projection = 1.5 (dashed line in Fig. 2). At this point, one is not continuous. ALLD agent is capable of changing the behavior of a finite If the symmetric version of the Projection bias is used, the fraction of all the other agents that start cooperating and end results remain equivalent only if the Paranoia bias is set to a up defecting with each other. Or, in cascades jargon, it pro- value higher than 0 (SI Appendix, section 1.7). Next, we study duces a “global cascade” of defection. If the symmetric version more generally for all models, how Projection and Paranoia can of the bias is used, the value of Projection changes the vul- interact to affect the robustness of the network.

8730 | www.pnas.org/cgi/doi/10.1073/pnas.1803438115 Babino et al. Downloaded by guest on September 28, 2021 Downloaded by guest on September 28, 2021 ifrn auso gns eoyin memory 1.10 and agents’ of values different eas xlr h eut sn ifrn nta ausof values initial different using results the explore also we the with of the on constraint tion a imposes the approach that this values the in integers, change are a to bution not a and to estimation system due the initial is of simulations dynamics the the in among change observed difference any that of value same and initial ors same the maintaining nltcapoc st netgt h yaiso e fnet- of set a of dynamics which the in investigate works to is approach or analytic defection toward shifted estimation defect has to agents) likely other her because are of observation the actions others network on the that only the of (based believe of distribution may internal condition her agent because initial An the well. but as dynamics, the affects ls,clrdgeni i.6 eeldadsotnostransi- discontinuous a revealed 6, both, Fig. in in tion activation, green tran- overall phase the colored hidden in a class, visible of evidence not is is This which 6C. sition, Fig. in seen be can as Projection S When regions: of different value two shows map heat The vary. rtrw niae htwe the when that indicates row) first 6A, (Fig. stability of the of version 1.7). symmetric section previous tran- the Appendix, and the (SI use states bias we in of if type As found same value. is the sitions initial with map specific and similar this robust a are simulations, on results depend these not then and do 1.9), section present Appendix, of (SI and also value bistable, are initial classes (ii) the These cooperation, when refer respectively. high We defection, (i) 6A). high as Fig. (iii) classes in three regions these (red net- to defection the shift total to defection toward enough high is work agent a ALLD in single a ends Introducing network state. the instead, for parameters, defection other to abruptly transitions and cooperative for Fig. pro- smooth in a blue of showing gression cooperative, in highly marked were the parameters, networks of the some 6B, version For symmetric classes. three the into for 6A, Fig. Projection in depicted As eters. of values four with 3. Fig. aioe al. et Babino gc pcfi nlsso hc rosyedt ifrn regimes different to yield priors which of analysis specific A of values four The Paranoia hne icniuul taciia value critical a at discontinuously changes S θ gc ˆ m hc hw hr,btcniuu,tasto at transition continuous, but sharp, a shows which , etmpo ieo h in opnn when component giant the of size of map Heat S nhne.Ta swyw netgt nyfu values four only investigate we why is That unchanged. . Paranoia gc Projection and is h vlto falntok ol egrouped be could networks all of evolution the bias, nrae otnosywith continuously increases θ S ˆ Paranoia m n o oefiegandsto ausa edo we as values of set fine-grained more a not and a S a θ safnto of function a as ˆ Paranoia shge hnwa h vdnesoshr Our her. shows evidence the what than higher is Paranoia Paranoia m and ssoni i.5 nti a,w a esure be can we way, this In 5. Fig. in shown as , θ aaeesi uhawyta hyyedthe yield they that way a such in parameters ˆ m Projection isadtefato fAL gns But, agents. ALLD of fraction the and bias S ic h aaeeso the of parameters the Since . Paranoia gc isprmtri rae hnzr n her and zero than greater is parameter bias For . Interaction. iscntk oke h nta estima- initial the keep to take can bias n h eno the of mean the and θ ˆ m f sblwtetrsodcooperation threshold the below is il arxo 6st fparam- of sets 16 of matrix a yield θ f ˆ < eitdi i.5i combination in 5 Fig. in depicted m neetnl,ti snttecase the not is this Interestingly, . f n,hne ecmiepri- combine we hence, and, , c 2 f h ewr ean mostly remains network the , if ; The Projection f etos1.9 sections Appendix, SI c 2 Paranoia of f . Beta Projection ≥ ,tevleof value the 0.8, Projection isntonly not bias S r covaried, are Beta a f second A . Projection Projection > < f f θ ˆ c ,the 0.8, m distri- 2 = and For . and f c 1 f , i.4. Fig. ahdln niae h icniuu rniinat transition discontinuous the indicates line dashed point the rcinAL gns h anrsl sta,i the the if zero, that, to is set result is main bias The agents. ALLD fraction model bootstrap k-core to jection a related used closely al. are et which (23). Ramos results percolation their 24, explain ref. extreme transitions to of same In spread of These the (24). in them. kinds observed opinions between been two have connection transitions a same of pro- kinds is the two there the of that between suggests appearance differences the process are percolation there cesses, a bootstrap though resem- in the Even in jumps transitions (23). present other of transitions each types the to two ble defecting These are manner. that noncontinuous of agents value to of the zero fraction If transition. being continuous from a Projection goes zero, than agents greater defecting being of of at value transition component one the giant only If is there introduced. enough, is agents simulations. ALLD our in ver- found asymmetric value the the with of coincides case and the lytically In the defect. of to sion start value agents the the If of other. each with of cooperating of value keep the agents in the threshold which a is there network, the agents. ALLD propagation of set the a affect by parameters started these defection how of inquire to this biases, use cognitive specific two how simulation, tion model agent-based to an able in were rule we updating Bayesian a using By high a in is Discussion finally, and, system bistable state. of a defection value as the behaves as network and, depending cooperation, behaviors, of high different value three the of the on values displays higher network For the class. network high-defection the cooperation), the jection full in in is begins that never network a of with values ble greater since maximum of strict values maximum for even iiyfrvle of values of for in values bility is low network for the 10, cooperation ref. ALLD high from of experimentally fraction estimated the a than of of the values inception paranoia, moderate the that For of under agents. means value robust This is the class. society of high-cooperation the the regardless to 0, belong to networks set is parameter diinly esuyhwteefc of effect the how study we Additionally, of fraction a when transitions of kinds two find we Then, in introduced is agent ALLD one only if that is result first Our Projection and xmlsof Examples neati h pedn ftedfcin rdcdby produced defections the of spreading the in interact 07) hc r lgtyaoeeprmna estimates, experimental above slightly are which (0.75), mattedcso-aigpoes hn we Then, process. decision-making the impact Paranoia, f c 1 is Projection twhich at shge hntetrsod hnapstv fraction positive a then threshold, the than higher is .,w n nte rniinat transition another find we >0.8, fi szr,te h ewr sin is network the then zero, is it If Paranoia. f PNAS vs. Paranoia S S gc is h hehl ol eddcdana- deduced be could threshold the bias, gc Paranoia | hne rmzr opstv ausadthe and values positive to zero from changes o he ausof values three for uut2,2018 28, August f03.Frti ee of level this For 0.36. of Paranoia Projection isde o aetenetwork the make not does bias Paranoia f ti svr ls othe to close very is (this Paranoia | c h rosshow arrows The Projection. Paranoia 1 02)wihaelower are which (0.25) o.115 vol. hr h ieo the of size the where f n hfst bista- to shifts and Paranoia c 2 . Projection Projection r incompati- are f | nrae,the increases, c 2 o 35 no. hr the where , Projection, Projection and Projec- slow is | pto up Pro- Pro- 8731

PSYCHOLOGICAL AND COGNITIVE SCIENCES A B die-rolling experiment (29) showed that there is a correlation between individual traits and global corruption. The corruption game (10) was measured in the United States and Argentina, where there are very different indices of corruption and results were not very different. This suggests that Projection is not the most likely psychological bias to account for. This is consistent with our findings that a wide range of behaviors is only observed for experimentally observed values of Projection. Within this range, variability in Paranoia may explain bifurcations between societies that (with similar initial states) converge to defection or cooperation, or similarly in our interest to a high or low level of corruption. Another source of variability could be the effective impact of the ALLD agents in a society. Some govern- C D ments have more efficient institutions which control the acts of the ALLD agents more effectively. Small changes in this control could lead to a different effective fraction of ALLD agents which, in a society that is in a bistable state, will lead to convergence to cooperation or defection. Our work builds on, and links, two different fields of behavioral sciences. On the one hand, a tradition that has studied agents with simple, yet effective, strategies, and which of these strategies prevails in different contexts. Under this approach, using unbiased agents, the appearance and prevalence of corruption have been investigated (17, 18, 20). The conundrum of the emergence of cooperation has been addressed under this framework, too (30, 31). On the other hand, our work builds Fig. 5. Belief distribution for four parameter sets of Paranoia, a and b. In A, on a Bayesian approach to decision making that has sought to we plot a Betaθ (4, 8) with Paranoia = 0; in B, a Betaθ (3, 9) with Paranoia = inquire how priors and evidence are used to generate beliefs and 0.22; in C, a Betaθ (2, 10) with Paranoia = 0.36; and in D, a Betaθ (1, 11) with guide actions (32, 33). This work can be seen as a mixture of these Paranoia = 0.37. The green line shows the mean value of the probability of defection, θˆ, while the red line shows the manipulated mean value θˆm. two traditions, where we can then ask how low-level psycholog- The area under the distribution between θˆ and θˆm is, by definition, the ical constructs which affect the inferential process (micro) have value of the Paranoia parameter (Eq. 1). The mean of the distribution (or, an impact on the large-scale organization of societies (macro). equivalently, the values of a and b) and the Paranoia parameters have been To build this bridge between these traditions, we use network chosen in such a way that they compensate each other and lead to the same theory, and the process that emerged under this framework was manipulated mean θˆm. The blue vertical line shows the limit above which very similar to an already-known network process: bootstrap per- the agent believes that the maximum reward is achieved by defecting and colation. We did not impose this process, but it was the result below which the agent believes that the maximum reward is obtained by of the cognitive model of the agents. Under this framework, ˆm cooperating. If θ is higher than this value, then the agent decides to defect; we find that small variations on cognitive biases could have if, on the contrary, θˆm lays to the left of the blue vertical line the agent chooses to cooperate. Under these four conditions, the agents, initially, a significant impact on the average behavior of all of society. cooperate with each other. PDF, probability density function. According to our model, society-level phenomena, including the broken window theory (34) and the broad range in the degree

less robust to the infection of ALLD agents. Only if it is combined with the Projection bias does it weaken the network. A distinguishing characteristic of our model is that it is built AB upon a network, and the agents can choose a different action for different contacts. In their seminal work, Nowak and May (25) use a regular lattice where each agent plays only one action in each step against their contacts. This model is suitable to study the evolution of cooperation, but we believe that, to investigate cooperation at a cultural level, we have to add the possibility of acting differently with different contacts. We used a static random network, but this is just another parameter of the model C that can be modified. For example, a network whose degree distribution follows a power law, or a small-world network, could be used (26, 27). It is also possible to add dynamics to the network, allowing the interaction to change iteration after iteration. Addi- tionally, the ALLD agents could be placed, not randomly, but rather in high-centrality nodes or in given k-shell within the network. This variation could be used to investigate the effect that highly connected people’s behavior might have on the rest of the community. Fig. 6. (A) Classification of the network according to its stability for 16 The long-term motivation for this study is to understand why different parameters. (B and C) Sa and Sgc as a function of the fraction different societies may converge to different policies of coopera- of ALLD agents. The dashed line in B shows the expected Sa due only to the presence of the ALLD agents and assuming that they do not interact tion and defection. One particular case which we are interested with each other. It can be seen that the high-cooperation region does not in, and which was the motivation for the work on the corruption have a sharp transition, while the bistable ones do, and the high-defection game (10), was how this might impact in different degrees of cor- one has a large Sa and Sgc for any nonzero values of the fraction of ALLD ruption. Recently, Gächter and Schulz (28) using an anonymous agents.

8732 | www.pnas.org/cgi/doi/10.1073/pnas.1803438115 Babino et al. Downloaded by guest on September 28, 2021 Downloaded by guest on September 28, 2021 1 reyD ownti ,Pee 20)Chrn rirrns:Sal eadcurves demand Stable arbitrariness: Coherent (2003) D Prelec Avoiding G, upset: Loewenstein Conveniently D, Ariely (2015) M 11. Sigman A, Babino R, Perez-Truglia R, Tella Di 10. 3 rjcJ oc ,Aaj ,Cet A ace 21)Sca xeiet nthe in experiments Social (2010) A Sanchez JA, Cuesta L, Araujo C, Fosco J, Grujic reveal—preferences. just 13. create—not actions How (2008) MI Norton D, Ariely 12. 4 rcaLzr ,e l 21)Htrgnosntok ontpooecooperation promote not do networks Heterogeneous (2012) al. et C, Gracia-Lazaro 14. 7 akE azMriM(02 nteclua rnmsino corruption. of transmission cultural cooperation. the of On (2002) evolution M the Saez-Marti E, and Hauk deliberation, 17. self- Intuition, of (2016) theory DG A Rand people: honest A, of Bear dishonesty The 16. (2008) D Ariely O, Amir N, Mazar 15. u omns hsrsac a upre yCneoNcoa de Nacional Consejo by supported was research This comments. ful ACKNOWLEDGMENTS. cogni- as of such consequence mechanisms the Paranoia. reduction is dissonance countries, tive among corruption of aioe al. et Babino .RbnM(1995) M Rabin 9. (1962) L Festinger 5. .IuaK ta.(00 erlcreae fcgiiedsoac n choice-induced and dissonance cognitive of correlates Neural (2010) attitude al. predicts et activity K, Neural Izuma (2009) CS 8. Carter compliance. JW, Schooler forced MK, Krug of V, consequences Veen van Cognitive 7. (1959) JM Carlsmith L, Festinger 6. cooperation. and competition, fairness, of theory A (1999) KM Schmidt E, Fehr 4. .Khea ,KeshJ,Tae H(96 aresadteasmtosof assumptions the and G allocation. in Fairness 3. dissonance cognitive (1986) and Accountability shares: RH Fair (2000) J Thaler Konow 2. JL, Knetsch D, Kahneman 1. eateto cnmc,Ui fClfri,Berkeley California, of Univ Economics, of (Department onSci Cogn preferences. stable without altruism. others’ about beliefs distorting by altruism change. preference dissonance. cognitive in change Psychol Abnorm 2. Vol CA), Econ eocl:Hmn lyn pta rsnrsdilemma. prisoner’s spatial a playing Humans mesoscale: bargaining. Rev Econ economics. 107:311–335. dilemma. Prisoner’s a play humans when rcNt cdSiUSA Sci Acad Natl Proc maintenance. concept t ,ShitegrR cwreB(92 neprmna nlsso ultimatum of analysis experimental An (1982) B Schwarze R, Schmittberger W, uth ¨ 114:817–868. 12:13–16. 90:1072–1091. Bus J cnBhvOrgan Behav Econ J 58:203–210. 59:S285–S300. oa rfrne,MrlCntans n efSrigBiases Self-Serving and Constraints, Moral Preferences, Moral hoyo ontv Dissonance Cognitive of Theory A rcNt cdSiUSA Sci Acad Natl Proc akRes Mark J 113:936–941. etakNclsE te-oe o i help- his for Stier-Moses E. Nicolas thank We Econ J Q a Neurosci Nat 3:367–388. 45:633–644. 118:73–105. rcNt cdSiUSA Sci Acad Natl Proc 107:22014–22019. 12:1469–1474. mEo Rev Econ Am Safr nvPes Stanford, Press, Univ (Stanford ). LSONE PLoS 109:12922–12926. Projection 105:3416–3442. 5:e13749. cnTheor Econ J Trends and Am J Q J 4 ezrK(08 h pedn fdisorder. of spreading The flex- (2008) K and Keizer Structure 34. (2015) TL Griffiths JB, Tenenbaum SJ, Gershman JL, Austerweil cognition. everyday 33. in one- predictions the Optimal in (2006) fairness JB Tenenbaum of TL, Evolution Griffiths (2013) MA 32. Nowak and H, cooperation Ohtsuki CE, of Tarnita Emergence DG, (2004) Rand D Fudenberg 31. C, Taylor A, Sasaki MA, Nowak 30. F U, Fischbacher 29. degree arbitrary G with graphs 28. Random (2001) DJ Watts SH, Strogatz networks. ME, ’small-world’ Newman of chaos. dynamics 27. Collective spatial (1998) and SH games Strogatz DJ, Evolutionary Watts (1992) 26. RM May extreme? become opinion public MA, does How Nowak (2015) al. on 25. et percolation M, Bootstrap Ramos (2010) JFF 24. Mendes networks. AV, Goltsev random SN, on Dorogovtsev cascades GJ, global Baxter of 23. model simple A (2002) D experiments. Watts in spitefulness 22. and altruism local Modeling of (1998) consequences DK Some Levine regions: 21. and countries across of Corruption dynamics (2007) the RK and Sah transmission cultural 20. of economics The (2001) T Verdier persistence A, the Bisin to applications (with 19. reputations collective of theory A (1996) J Tirole 18. S nomto n nelgn ytm rn 1515022. Grant Systems Intelligent National and and Information NIH R01EB022720 Grant NSF by Bioengineering and sponsored Imaging is Biomedical of H.A.M. Institute Award. Understand- in Cognition-Scholar Initiative Human Science ing Century 21st Foundation McDonnell James Cient Cient Investigaciones preferences. quality). firm to and corruption of blt nBysa oeso cognition. of Psychology models Mathematical Bayesian in ibility Sci game. ultimatum anonymous shot populations. finite in stability evolutionary cheating. societies. across applications. 64:026118. their and distributions 393:440–442. 826–829. networks. complex USA Sci Acad 1:593–622. osmosis. ctrS cuzJ 21)Itischnsyadtepeaec frl violations rule of prevalence the and honesty Intrinsic (2016) JF Schulz S, achter ¨ fiayTecnol y ´ ıfica 17:767–773. cnDnControl Dyn Econ J u cnAssoc Econ Eur J cnTheor Econ J 99:5766–5771. Nature lm-es 21)Le ndsus-neprmna td on study experimental disguise-an in Lies (2013) F ollmi-Heusi ¨ gc rn IT21 N 2013 PICT Grant ogica ´ hsRvESa olnSf atrPhys Matter Soft Nonlin Stat E Rev Phys PNAS fia T y ´ ıficas 531:496–499. Ofr nvPes e ok,p 187–208. pp York), New Press, Univ (Oxford 97:298–319. 11:525–547. 31:2573–2598. | uut2,2018 28, August cia n od aal Investigaci la para Fondo and ecnicas ´ e cnStud Econ Rev rcNt cdSiUSA Sci Acad Natl Proc hsRvESa olnSf atrPhys Matter Soft Nonlin Stat E Rev Phys xodHnbo fCmuainland Computational of Handbook Oxford Science Nature ◦ 63 ..i pnoe ythe by sponsored is M.S. 1653. 63:1–22. 322:1681–1685. 428:646–650. | o.115 vol. 110:2581–2586. 82:1–9. | c Rep Sci o 35 no. e cnDyn Econ Rev Nature 5:10032. rcNatl Proc | Psychol Nature 8733 359: on ´

PSYCHOLOGICAL AND COGNITIVE SCIENCES