sign in sign up
<<
Home , Great Filter, Nick Bostrom, The Millennial Project

Risks of Astronomical Future Suffering

Brian Tomasik Center on Long-Term Risk [email protected]

Abstract

It’s far from clear that values will shape an -based space-colonization wave, but even if they do, it seems more likely that will increase total suffering rather than decrease it. That said, other people care a lot about humanity’s survival and spread into the cosmos, so I think suffering reducers should let others pursue their spacefaring dreams in exchange for stronger safety measures against future suffering. In general, I encourage people to focus on making an intergalactic future more humane if it happens rather than making sure there will be an intergalactic future.

Contents

1 Epigraphs 2

2 values may not control the future2

3 Some scenarios for future suffering2 3.1 Spread of wild ...... 2 3.2 Sentient simulations...... 2 3.3 Suffering subroutines...... 3 3.4 Black Swans...... 3

4 Even a human-controlled future is likely to increase suffering3 4.1 Spread of wild animals...... 3 4.2 Sentient simulations...... 4 4.3 Suffering subroutines...... 4 4.4 Black Swans...... 4

5 What about paperclippers?4

6 What if human colonization is more humane than ET colonization?5

7 Why we should remain cooperative6

8 Possible upsides to an intelligent future7 8.1 Black swans that don’t cut both ways...... 7 8.2 Valuing reflection...... 7

Acknowledgments 8 Risks of Astronomical Future Suffering – Center on Long-Term Risk

1 Epigraphs important to keep in mind when we imagine what future humans might be able to bring about with If we carry the green fire-brand from to their technology. star, and ignite around each a conflagration of Some people believe that sufficiently advanced su- vitality, we can trigger a Universal metamorpho- perintelligences will discover the moral truth and sis. [...] Because of us [...] Slag will become soil, hence necessarily do the right things. Thus, it’s grass will sprout, flowers will bloom, and forests 1 claimed, as long as humanity survives and grows will spring up in once sterile places. [...] If we more intelligent, the right things will eventually hap- deny our awesome challenge; turn our backs pen. There are two problems with this view. First, on the living , and forsake our cosmic Occam’s razor militates against the existence of a destiny, we will commit a crime of unutterable moral truth (whatever that’s supposed to mean). magnitude. Second, even if such moral truth existed, why should – Marshall T. Savage, The Millennial Project: a care about it? There are plenty Colonizing the in Eight Easy Steps, 1994 of brilliant people on Earth today who eat meat. They know perfectly well the suffering that it causes, Let’s pray that the human race never escapes but their motivational systems aren’t sufficiently en- from Earth to spread its iniquity elsewhere. gaged by the harm they’re doing to farm animals. – C.S. Lewis The same can be true for . In- deed, arbitrary intelligences in mind-space needn’t If you can’t beat ’em, join ’em. have even the slightest inklings of empathy for the – proverb suffering that sentients experience.

2 Humans values may not control the 3 Some scenarios for future suffering future Even if humans do preserve control over the future ’s "The Future of Human Evolution" of Earth-based life, there are still many ways in (Bostrom, 2004) describes a scenario in which hu- which space colonization would multiply suffering. man values of fun, leisure, and relationships may be Following are some of them. replaced by hyper-optimized agents that can better compete in the Darwinian race to control our future 3.1 Spread of wild animals light cone. The only way we could avert this com- petitive scenario, Bostrom suggests, would be via Humans may colonize other , spreading a "singleton" (Bostrom, 2006), a unified agent or suffering-filled life via . Some governing structure that could control evolution. Of humans may use their resources to seed life through- course, even a singleton may not carry on human out the galaxy, which some sadly consider a moral values. Many naive AI agents that humans might imperative. build may optimize an objective function that hu- mans find pointless. Or even if humans do maintain 3.2 Sentient simulations hands on the steering wheel, it’s far from guaranteed Given astronomical (Bostrom, 2003) computing that we can preserve our goals in a stable way across power, post-humans may run various kinds of sim- major self-modifications going forward. ulations. These sims may include many copies of These factors suggest that even conditional on wild-animal life, most of which dies painfully shortly human technological progress continuing, the proba- after being born. For example, a superintelligence bility that human values are realized in the future aiming to explore the distribution of extraterres- may not be very large. Carrying out human values trials of different sorts might run vast numbers of seems to require a singleton that’s not a blind opti- simulations (Thiel, Bergmann and Grey, 2003) of mizer, that can stably preserve values, and that is evolution on various kinds of planets. Moreover, shaped by designers who care about human values scientists might run even larger numbers of simula- rather than selfish gain or something else. This is tions of organisms-that-might-have-been, exploring 1Because nature contains such vast amounts of suffering, I would strongly dislike such a project. I include this quotation for rhetorical effect and to give a sense of how others see the situation.

2 Risks of Astronomical Future Suffering – Center on Long-Term Risk

the space of minds. They may simulate decillions of the sort that we care very much about. (How- reinforcement learners that are sufficiently self-aware ever, if they were very sophisticated, it would be as to feel what we consider conscious pain. relatively unlikely that we would not consider them conscious.) 3.3 Suffering subroutines • It’s unclear whether humanity would support such missions. Environmentalists would ask us It could be that certain algorithms (say, reinforce- to leave ET habitats alone. Others wouldn’t ment agents (Tomasik, 2014)) are very useful in want to spend the energy on rescue missions un- performing complex machine-learning computations less they planned to mine resources from those that need to be run at massive scale by advanced planets. AI. These subroutines might be sufficiently similar to the pain programs in our own brains that we con- Contrast this with the possibilities for spreading sider them to actually suffer. But profit and power wild-animal suffering: may take precedence over pity, so these subroutines • Humans may spread life to many planets (e.g., may be used widely throughout the AI’s Matrioshka via terraforming, other Earth-like planets brains. via directed ). The number of plan- ets that can support life may be appreciably 3.4 Black Swans bigger than the number that already have it. The range of scenarios that we can imagine is limited, (See the discussion of fl in the .) and many more possibilities may emerge that we Moreover, the percentage of planets that can be haven’t thought of or maybe can’t even comprehend. converted into computers that could simulate wild-animal suffering might be close to 100%. 4 Even a human-controlled future is • We already know that Earth-based life is sen- likely to increase suffering tient, unlike for ETs. • Spreading biological life is slow and difficult, If I had to make an estimate now, I would give but disbursing small life-producing capsules is ~70% probability that if humans choose to colonize easier than dispatching Hedonistic Imperative space, this will cause more suffering than it reduces probes or berserker probes. on intrinsic grounds (ignoring compromise consid- Fortunately, humans might not support spread of erations discussed later). Think about how space life that much, though some do. For terraforming, colonization could plausibly reduce suffering. For there are survival pressures to do it in the near term, most of those mechanisms, there seem to be counter- but probably is a bigger prob- mechanisms that will increase suffering at least as lem in the long term. Also, given that terraforming much. The following sections parallel those above. is estimated to require at least thousands of years, while human-level digital intelligence should take at 4.1 Spread of wild animals most a few hundred years to develop, terraforming David Pearce coined the phrase "cosmic rescue mis- may be a moot point from the perspective of catas- sions" (Pearce, n.d.) in referring to the possibility of trophic risks, since digital intelligence doesn’t need sending probes to other planets to alleviate the wild terraformed planets. extraterrestrial (ET) suffering they contain. This is While I noted that ETs are not guaranteed to a nice idea, but there are a few problems. be sentient, I do think it’s moderately likely that • We haven’t found any ETs yet, so it’s not obvi- consciousness is fairly convergent among intelligent ous there are vast numbers of them waiting to civilizations. This is based on (a) suggestions of be saved from Darwinian misery. convergent consciousness among animals on Earth • The specific kind of conscious suffering known and (b) the general principle that consciousness to Earth-bound animal life would not neces- seems to be useful for planning, manipulating im- sarily be found among the ETs. Most likely ages, self-modeling, etc. On the other hand, maybe ETs would be bacteria, plants, etc., and even if this reflects the paucity of my human imagination they’re intelligent, they might be intelligent in in conceiving of ways to be intelligent without con- the way robots are without having emotions of sciousness.

3 Risks of Astronomical Future Suffering – Center on Long-Term Risk

4.2 Sentient simulations 4.3 Suffering subroutines Suffering subroutines may be a convergent outcome It may be that biological suffering is a drop in the of any AI, whether human-inspired or not. They bucket compared with digital suffering. The bio- might also be run by aliens, and maybe humans sphere of a is less than Type I on the Kar- could ask aliens to design them in more humane dashev scale; it uses a tiny sliver of all the energy ways, but this seems speculative. of its star. Intelligent computations by a Type II civilization can be many orders of magnitude higher. 4.4 Black Swans So humans’ sims could be even more troubling than their spreading of wild animals. It seems plausible that suffering in the future will be dominated by something totally unexpected. This Of course, maybe there are ETs running sims of could be a new discovery in physics, neuroscience, nature for science or amusement, or of minds in gen- or even more generally. Some make the eral to study biology, psychology, and sociology. If argument that because we know so very little now, we encountered these ETs, maybe we could persuade it’s better for humans to stick around because of them to be more humane. the "option ": If they later realize it’s bad to spread, they can stop, but if they realize they should I think it’s likely that humans are more empathetic spread, they can proceed to reduce suffering in some than the average civilization because novel way that we haven’t anticipated. 1. we seem much more empathetic than the aver- Of course, the problem with the "option value" age animal on Earth, probably in part due to argument is that it assumes future humans do the parental impulses and in part due to trade, al- right things, when in fact, based on examples of though presumably some of these factors would speculations we can imagine now, it seems future necessarily be true of any technologically ad- humans would probably do the wrong things much vanced civilization of the time. For instance, faced with a new discovery 2. implies that we’ll agree with our of obscene amounts of computing power somewhere, own society’s morals more than those of a ran- most humans would use it to run oodles more minds, dom other society because these are the values some nontrivial fraction of which might suffer terri- that we were raised with and that our biology bly. In general, most sources of immense power are impels us toward. double-edged swords that can create more happiness and more suffering, and the typical human impulse Based on these considerations, it seems plausible to promote life/consciousness rather than to remove that there would be room for improvement through them suggests that negative and negative-leaning interaction with ETs. Indeed, we should in general utilitarians are on the losing side. expect it to be possible for any two civilizations or Still, waiting and learning more is plausibly factions to achieve gains from compromise if they Kaldor-Hicks efficient, and maybe there are ways it have diminishing with respect to can be made Pareto efficient by granting additional amount of control exerted. In addition, there may be concessions to suffering reducers as compensation. cheap Pareto improvements to be had purely from increased intelligence and better understanding of 5 What about paperclippers? important considerations. Above I was largely assuming a human-oriented civ- That said, there are some downside risks. Post- ilization with values that we recognize. But what humans themselves might create suffering simula- if, as seems mildly likely, Earth is taken over by a tions, and what’s worse, the sims that post-humans paperclip maximizer, i.e., an unconstrained automa- run would be more likely to be sentient than those tion or optimization process? Wouldn’t that reduce run by random ETs because post-humans would have suffering because it would eliminate wild ETs as the a tendency to simulate things closer to themselves paperclipper spread throughout the galaxy, without in mind-space. They might run nature sims for aes- causing any additional suffering? thetic appreciation, lab sims for science experiments, Maybe, but if the paperclip maximizer is actually or pet sims for pets. generally intelligent, then it won’t stop at tiling the

4 Risks of Astronomical Future Suffering – Center on Long-Term Risk

solar system with paperclips. It will want to do sci- how that colonization proceeds. Evolutionary forces ence, perform lab experiments on sentient creatures, might distort compassionate human values into some- possibly run suffering subroutines, and so forth. It thing unrecognizable. Alternatively, a rogue AI will require lots of intelligent and potentially sen- might replace humans and optimize for arbitrary val- tient robots to coordinate and maintain its paperclip ues throughout the cosmos. In these cases, humans’ factories, energy harvesters, and mining operations, greater-than-average compassion doesn’t make much as well as scientists and engineers to design them. difference, so suppose that the value of these colo- And the paperclipping scenario would entail similar nization waves would be −1, just like for colonization black swans as a human-inspired AI. Paperclippers by random ETs. Let the probability be Q that these would presumably be less intrinsically humane than non-compassionate forces win control of Earth’s col- a "friendly AI", so some might cause significantly onization. Now the expected values are more suffering than a friendly AI, though others −31 · Q + −0.5 · (1 − Q) might cause less, especially the "minimizing" paper- clippers, e.g., cancer minimizers or death minimizers. for Earth-originating colonization versus If the paperclipper is not generally intelligent, I −1 · P have a hard time seeing how it could cause . In this case it would be like many other if Earth doesn’t colonize and leaves open the possi- catastrophic risks – deadly and destabilizing, but bility of later ET colonization. not capable of wiping out the human race. For concreteness, say that Q = 0.5. (That seems plausibly too low to me, given how many times Earth 6 What if human colonization is more has seen overhauls of hegemons in the past.) Then humane than ET colonization? Earth-originating colonization is better if and only if −1 · 0.5 + −0.5 · 0.5 > −1 · P If we knew for certain that ETs would colonize our region of the universe if Earth-originating intelli- −0.75 > −1 · P gence did not, then the question of whether humans P > 0.75. should try to colonize space becomes less obvious. Given uncertainty about the and As noted above, it’s plausible that humans are more , it seems hard to maintain a probability compassionate than a random ET civilization would greater than 75% that our future light cone would be. On the other hand, human-inspired computa- contain colonizing ETs if we don’t ourselves colonize, tions might also entail more of what we consider to although this section presents an interesting argu- count as suffering because the mind architectures of ment for thinking that the probability of future ETs the agents involved would be more familiar. And is quite high. having more agents in competition for the light cone What if rogue AIs result in a different magnitude might lead to dangerous outcomes. of disvalue from arbitrary ETs? Let H be the ex- But for the sake of argument, suppose an Earth- pected harm of colonization by a rogue AI. Assume originating colonization wave would be better than ETs are as likely to develop rogue AIs as humans the expected colonization wave of an ET civilization are. Then the disvalue of Earth-based colonization that would colonize later if we didn’t do so. In partic- is ular, suppose that if human values colonized space, H · Q + (−0.5) · (1 − Q), they would cause only −0.5 units of suffering, com- and the harm of ET colonization is pared with −1 units if random ETs colonized space. Then it would seem that as long as the probability P · (H · Q + (−1) · (1 − Q)). P of some other ETs coming later is bigger than 0.5, Again taking Q = 0.5, then Earth-based colonization then it’s better for humans to colonize and pre-empt has better expected value if the ETs from colonizing, since −0.5 > −1 · P for P > 0.5. H · 0.5 + −0.5 · 0.5 > P · (H · 0.5 + −1 · 0.5) However, this analysis forgets that even if Earth- H − 0.5 > P · (H − 1) originating intelligence does colonize space, it’s not (H − 0.5) P > , at all guaranteed that human values will control (H − 1)

5 Risks of Astronomical Future Suffering – Center on Long-Term Risk

Figure 1: Plot of threshold values for P as a function of H where the inequality flips around when we divide by Eventually, those who go on developing the tech- the negative number (H − 1). Figure1 represents nologies will hold most of the world’s power. These a plot of these threshold values for P as a function people will, if only by selection effect, have strong of H. desires to develop AI and colonize space. Even if H = 0 and a rogue AI caused no suffering, Resistance might not be completely futile. There’s it would still only be better for Earth-originating some small chance that suffering reducers could in- intelligence to colonize if P > 0.5, i.e., if the prob- fluence society in such a way as to prevent space ability of ETs colonizing in its place was at least colonization. But it would be better for suffering 50%. reducers, rather than fighting technologists, to com- These calculations involve many assumptions, and promise with them: We’ll let you spread into the it could turn out that Earth-based colonization has cosmos if you give more weight to our concerns about higher expected value given certain parameter val- future suffering. Rather than offering a very tiny ues. This is a main reason I maintain uncertainty chance of complete victory for suffering reducers, as to the sign of Earth-based space colonization. this cooperation approach offers a higher chance of However, this whole section was premised on human- getting an appreciable fraction of the total suffering inspired colonization being better than ET-inspired reduction that we want. In addition, compromise colonization, and the reverse might also be true, means that suffering reducers can also win in the since computations of the future are more likely to scenario ( 30% likely in my view) that technologi- be closer to what we most value and disvalue if cal development does prevent more suffering than it humans do the colonizing. causes even apart from considerations of strategic compromise with other people. 7 Why we should remain cooperative Ideally these compromises would take the form of robust bargaining arrangements. Some exam- If technological development and space colonization ples are possible even in the short term, such as if seem poised to cause astronomical amounts of suf- suffering reducers and space-colonization advocates fering, shouldn’t we do our best to stop them? Well, agree to cancel opposing funding in support of some it is worth having a discussion about the extent to commonly agreed-upon project instead. which we as a society want these outcomes, but my The strategic question of where to invest resources guess is that someone will continue them, and this to advance your values at any given time amounts to would be hard to curtail without extreme measures. a prisoner’s dilemma with other value systems, and

6 Risks of Astronomical Future Suffering – Center on Long-Term Risk

because we repeatedly make choices about where 8 Possible upsides to an intelligent to invest, what stances to adopt, and what policies future to push for, these prisoner’s dilemmas are iterated. In Robert Axelrod’s tournaments on the iterated 8.1 Black swans that don’t cut both prisoner’s dilemma, the best-performing strategies ways were always "nice," i.e., not the first to defect. Thus, Many speculative scenarios that would allow for suffering reducers should not be the first to defect vastly reducing suffering in the multiverse would against space colonizers. Of course, if it seems that also allow for vastly increasing it: When you can space colonizers show no movement toward suffering decrease the number of organisms that exist, you reduction, then we should also be "provocable" to can also increase the number, and those who favor temporary defection until the other side does begin creating more happiness / life / complexity / etc. to recognize our concerns. will tend to want to push for the increasing side. However, there may be some black swans that re- We who are nervous about space colonization ally are one-sided, in the sense that more knowledge stand a lot to gain from allying with its supporters – is most likely to result in a decrease of suffering. For in terms of thinking about what scenarios might hap- example: We might discover that certain routine pen and how to shape the future in better directions. physical operations map onto our conceptions of We also want to remain friends because this means suffering. People might be able to develop ways to pro-colonization people will take our ideas more seri- re-engineer those physical processes to reduce the ously. Even if space colonization happens, there will suffering they contain. If this could be done without remain many sub-questions on which suffering reduc- a big sacrifice to happiness or other values, most peo- ers want to have a say: e.g., not spreading wildlife, ple would be on board, assuming that present-day not creating suffering simulations/subroutines, etc. values have some share of representation in future decisions. We want to make sure suffering reducers don’t This may be a fairly big deal. I give nontrivial become a despised group. For example, think about probability (maybe ~10%?) that I would, upon suf- how eugenics is more taboo because of the Nazi ficient reflection, adopt a highly inclusive view of atrocities than it would have been otherwise. Anti- what counts as suffering, such that I would feel that technology people are sometimes smeared by associa- significant portions of the whole multiverse contain tion with the Unabomber. Animal supporters can be suffering-dense physical processes. After all, the tarnished by the violent tactics of a few, or even by mechanics of suffering can be seen as really simple the antics of PETA. We need to be cautious about when you think about them a certain way, and as something similar happening for suffering reduction. best I can tell, what makes animal suffering special Most people already care a lot about preventing are the bells and whistles that animal sentience in- suffering, and we don’t want people to start saying, volves over and above crude physics – things like "Oh, you care about preventing harm to powerless complex learning, thinking, memory, etc. But why creatures? What are you, one of those suffering can’t other physical objects in the multiverse be reducers?" where "suffering reducers" has become the bells and whistles that attend suffering by other such a bad name that it evokes automatic hatred. physical processes? This is all very speculative, but what understandings of the multiverse our descen- dants would arrive at we can only begin to imagine So not only is cooperation with colonization sup- right now. porters the more promising option, but it’s arguably the only net-positive option for us. Taking a more 8.2 Valuing reflection confrontational stance risks hardening the opposi- tion and turning people away from our message. If we care to some extent about moral reflection on Remember, preventing future suffering is something our own values, rather than assuming that suffering that everyone cares about, and we shouldn’t erode reduction of a particular flavor is undoubtedly the that fact by being excessively antagonistic. best way to go, then we have more reason to sup- 2Of course, what’s compelling to idealized-me would not necessarily be compelling to idealized-you. Value divergences may

7 Risks of Astronomical Future Suffering – Center on Long-Term Risk

port a technologically advanced future, at least if tems was encouraged by Carl Shulman. Initially it’s reflective. I resisted his claim, but – as has often proved the In an idealized scenario like coherent extrapolated case at least on factual and strategic questions – I volition (CEV) (Yudkowsky, 2004), say, if suffering eventually realized he was right and came around to reduction was the most compelling moral view, oth- his view. ers would see this fact.2 Indeed, all the arguments any moral philosopher has made would be put on References the table for consideration (plus many more that no philosopher has yet made), and people would have Bostrom, Nick. "The Future of Human Evolution." a chance to even experience extreme suffering, in Death and Anti-Death: Two Hundred Years a controlled way, in order to assess how bad it is After Kant, Fifty Years After Turing. Palo Alto, compared with other things. Perhaps there would CA: Ria UP, 2004. 339-71. Web. 3 Mar. 2016 be analytic approaches for predicting what people www.nickbostrom.com/fut/evolution.html. would say about how bad torture was without ac- Bostrom, Nick. "What Is a Singleton?" Linguistic tually torturing them to find out. And of course, and Philosophical Investigations 5.2 (2006): we could read through humanity’s historical record 48-54. Web. 3 Mar. 2016. and all the writings on the Internet to learn more www.nickbostrom.com/fut/singleton. html about what actual people have said about torture, Bostrom, Nick. "Astronomical Waste: The although we’d need to correct for will-to-live bias Opportunity Cost of Delayed Technological and deficits of accuracy when remembering emotions Development." Utilitas 15.3 (2003): 308-3014. in hindsight. But, importantly, in a CEV scenario, Web. 3 Mar. 2016. all of those qualifications can be taken into account www.nickbostrom.com/astronomical/waste. by people much smarter than ourselves. html. Of course, this rosy picture is not a likely future Pearce, David. ", the Fermi outcome. Historically, forces seize control because Paradox and the Hedonistic Imperative." The they best exert their power. It’s quite plausible that Hedonistic Imperative. N.p., n.d. Web. 3 Mar. someone will take over the future by disregarding 2016. www.hedweb.com/object32.htm the wishes of everyone else, rather than by com- Thiel, Inari, Neil W. Bergmann, and William Grey. bining and idealizing them. Or maybe concern for "A Case for Investigating the Ethics of Artificial the powerless will just fall by the wayside, because Life?" The University of New South Wales 1 it’s not really adaptive for powerful agents to care (2003): 276-87. Web. 3 Mar. 2016. about weak ones, unless there are strong, stable so- http://espace.library.uq.edu.au/view cial pressures to do so. This suggests that improving /UQ:10754/A_Case_for_Inves.pdf prospects for a reflective, tolerant future may be an Tomasik, Brian. "Do Artificial important undertaking. Rather than focusing on Reinforcement-Learning Agents Matter whether or not the future happens, I think it’s more Morally?" ArXiv.org. N.p., 30 Oct. 2014. Web. valuable for suffering reducers to focus on making 03 Mar. 2016. http://arxiv.org/ abs/1410 the future better if it happens – by encouraging com- .8233v1. promise, moral reflectiveness, philosophical wisdom, Yudkowsky, Eliezer. Coherent Extrapolated Volition. and altruism, all of which make everyone better off San Francisco, CA: The Singularity Institute, in expectation. 2004. Web. 3 Mar. 2016. http://intelligence.org/files/CEV.pdf. Acknowledgments

A question by Rob Wiblin first inspired this piece. The discussion of cooperation with other value sys-

remain after extrapolation. That said, there’s no alternative better than compromising using a CEV-like approach, because if I try to defect and push my particular values, you’ll just try to push yours, and we’ll both be worse off in expectation.

8