Multiverse-wide Cooperation via Correlated Decision Making Caspar Oesterheld Abstract Some decision theorists argue that when playing a prisoner’s dilemma-type game against a sufficiently similar opponent, we should cooperate to make it more likely that our opponent also cooperates. This idea, which Hofstadter calls superrationality, has strong implications when combined with the insight from modern physics that we probably live in a large universe or multiverse of some sort. If we care about what happens in civilizations located elsewhere in the multiverse, we can superrationally cooperate with some of their inhabitants. That is, if we take their values into account, this makes it more likely that they do the same for us. In this paper, I attempt to assess the practical implications of this idea. I argue that to reap the full gains from trade, everyone should maximize the same impartially weighted sum of the utility functions of all collaborators. I also argue that we can obtain at least weak evidence about the content of these utility functions. In practice, the application of superrationality implies that we should promote causal cooperation, moral pluralism, moral reflection, and ensure that our descendants, who will be smarter and thus better at finding out how to benefit other superrationalists in the universe, engage in superrational cooperation. Contents 1 Introduction – the basic idea 3 1.1 An overview of the paper . 5 2 Superrationality 7 2.1 Lack of knowledge is evidential power, part I: the other agents . 9 2.2 A short survey of decision theories and their relation to superrationality . 11 2.3 CDT would self-modify to behave like a non-causal decision theory in some Newcomb-like problems . 12 2.4 Lack of knowledge is evidential power, part II: taking a step back . 14 2.5 Reasons and correlations . 15 2.5.1 Your back is not mine . 16 2.5.2 Does accepting superrationality commit us to irrational behavior in medical Newcomb problems? . 17 2.6 Are the correlations strong enough? . 20 2.6.1 Correlation only with close copies? . 20 2.6.2 Negative correlations? . 21 2.7 The relative importance of superrational cooperation: an example calculation 22 2.8 Compromise strategy . 23 1 2.8.1 Sharing gains from compromise in the face of asymmetries....... 23 2.8.2 The compromise problem......................... 24 2.8.3 Cooperation with and without coordination............... 26 2.8.4 Harsanyi’s aggregation theorem...................... 27 2.8.5 How to assign the weights......................... 28 2.8.6 Updateless weights............................. 33 2.8.7 Limitations................................. 34 2.8.8 Heuristics.................................. 34 2.8.9 Notes on superrational coordination................... 35 2.9 No reciprocity needed: whom to treat beneficially................ 40 2.9.1 Schemes of causal cooperation...................... 40 2.9.2 Circular cooperative structures and indirect causal reciprocity..... 41 2.9.3 Hierarchies and acyclic graphs...................... 42 2.9.4 Only helping superrational cooperators helps you superrationally... 46 2.10 Cheating, signaling, and half-heartedness.................... 48 3 Values 49 3.1 Orthogonality of instrumental rationality and values.............. 49 3.2 Necessary preconditions.............................. 50 3.2.1 Consequentialism.............................. 51 3.2.2 Caring about the multiverse........................ 52 3.2.3 Knowable values.............................. 53 3.2.4 The ability to help others......................... 56 3.2.5 Zero-sum and “below-zero-sum” tradeoffs on resources......... 58 3.3 What values?.................................... 59 3.3.1 Idealization................................. 60 3.3.2 Values and distance............................ 62 3.3.3 Different kinds of preferences....................... 64 3.4 The values of our superrational collaborators in the multiverse........ 65 3.4.1 On the far values of humans and human superrational cooperators.. 66 3.4.2 Biological evolution............................ 71 3.4.3 Cultural evolution............................. 72 3.4.4 Other considerations............................ 73 4 Interventions 74 4.1 Cooperation in the face of uncertainty about values............... 74 4.1.1 Universalism................................ 75 4.2 Moral advocacy................................... 76 4.2.1 Universalist values............................. 76 4.2.2 Concern for benevolence.......................... 78 4.2.3 Consequentialism.............................. 78 4.2.4 Pluralism.................................. 78 4.2.5 Promoting moral reflection........................ 79 4.2.6 Multiverse-wide preference utilitarianism................ 80 4.2.7 No multiverse-wide tug-of-war over values................ 81 4.3 Promoting causal cooperation........................... 81 4.4 Increasing capabilities............................... 83 4.5 Meta-activities................................... 84 2 4.5.1 Research................................... 84 4.5.2 Promoting multiverse-wide superrationality............... 85 4.6 Artificial intelligence................................ 85 4.6.1 AI safety not based on superrationality-related considerations..... 85 4.6.2 Multiverse-wide superrationality-inspired value-loading......... 86 4.6.3 Making an AI come up with superrational cooperation on its own.. 87 5 Acknowledgements 88 6 Appendix 88 6.1 Related work.................................... 89 6.1.1 Gary Drescher on superrationality.................... 89 6.1.2 Acausal trade................................ 89 6.1.3 Various mentions of multiverse-wide superrationality.......... 90 6.2 Many agents.................................... 90 6.3 Testability of superrationality........................... 91 6.4 Do people reason superrationally?........................ 92 6.5 The evolution of superrationality......................... 94 6.6 Superrational cooperation on Earth........................ 95 6.6.1 Fewer agents................................ 95 6.6.2 Argument from evolution: Superrationality did not evolve (strongly). 95 6.6.3 Causal cooperation seems more important................ 95 6.6.4 Hard-wired alternatives.......................... 96 6.7 Superrationality and morality........................... 96 6.7.1 Real altruism................................ 96 6.7.2 How much altruistic behavior does superrationality entail?...... 98 6.8 Multiverse-wide superrationality for causal decision theorists......... 98 6.9 Simulations..................................... 99 6.9.1 If being in a simulation is bad, avoid creating one............ 99 6.9.2 Happy simulations............................. 99 6.10 Infinite ethics.................................... 100 6.11 Objection based on uncertainty about the values of superrationalists in the multiverse...................................... 100 1 Introduction – the basic idea This paper makes an extraordinary claim: that a few interesting but by themselves inconse- quential ideas from decision theory and physics together give rise to a crucial consideration with strong implications for how to do the most good. In this first section, I will outline the main idea and forward-reference sections with the full arguments and detailed elaborations. Afterward, I give an overview of the entire paper, section by section (section 1.1). Consider the following thought experiment, adapted from Hofstadter’s (1983) Dilemmas for Superrational Thinkers, Leading Up to a Luring Lottery: Donation game with superrationality. Hofstadter sends 20 participants the same letter, asking them to respond with a single letter ‘C’ (for cooperate) or ‘D’ (for defect) without communicating with the other participants. Hofstadter 3 explains that by sending in ‘C’, a participant can increase everyone else’s payoff by $2. By sending in ‘D’, participants can increase their own payoff by $5. The letter ends by informing the participants that they were chosen for the similarity and rationality of their decision mechanisms, particularly in weird scenarios like this one. It should be noted that every participant only cares about the balance of her own bank account, and not about Hofstadter’s or that of the other 19 participants. Upon receiving the letter, should you cooperate or defect? Assuming the participants’ thought processes are sufficiently similar to each other, I think we should cooperate because this makes it more likely that our 19 fellow participants also cooperate (see chapter2 and the references given therein). After all, Hofstadter stated fairly explicitly that the thought processes of the participants are strongly correlated. Thus, if we cooperate, we should expect significantly more of the other participants to cooperate as well than if we defect, which means that cooperating has higher expected utility. Alternatively, we may reason that by our choice we determine what the rational choice is for all participants. Hofstadter calls this idea of cooperation via correlated decision making superrationality. By itself, superrationality does not seem particularly action-guiding. Usually, we have other evidence about other agents’ behavior and thought processes such that the evidence we gain from our own decisions is less important (see section 6.6). To apply superrationality in practice, we combine it with another intellectually stimulating but by itself inconsequential hypothesis: we probably live in a vast universe or even multiverse, most of which we cannot observe or interact with (see appendix section 6.2). In