Simulation, Self-Extinction, and Philosophy in the Service of Human Civilization

Total Page:16

File Type:pdf, Size:1020Kb

Simulation, Self-Extinction, and Philosophy in the Service of Human Civilization AI & Soc (2016) 31:171–190 DOI 10.1007/s00146-015-0620-9 ORIGINAL ARTICLE Simulation, self-extinction, and philosophy in the service of human civilization Jeffrey White1 Received: 16 February 2015 / Accepted: 25 September 2015 / Published online: 30 October 2015 Ó Springer-Verlag London 2015 Abstract Nick Bostrom’s recently patched ‘‘simulation realistic simulations to get past self-extinction-level threats argument’’ (Bostrom in Philos Q 53:243–255, 2003; Bos- is projected. In the end, Bostrom’s basic argument for the trom and Kulczycki in Analysis 71:54–61, 2011) purports conviction that we exist now in a simulation is reaffirmed. to demonstrate the probability that we ‘‘live’’ now in an ‘‘ancestor simulation’’—that is as a simulation of a period Keywords Simulation Á Model Á Cognitive social prior to that in which a civilization more advanced than our science Á Democracy Á Brain in a vat Á Skepticism Á own—‘‘post-human’’—becomes able to simulate such a Global coordination problem state of affairs as ours. As such simulations under consid- eration resemble ‘‘brains in vats’’ (BIVs) and may appear open to similar objections, the paper begins by reviewing 1 Introduction objections to BIV-type proposals, specifically those due a presumed mad envatter. In counter example, we explore We must declare war on war, so the outcome will be the motivating rationale behind current work in the peace upon peace. development of psychologically realistic social simula- -Obama (2014) tions. Further concerns about rendering human cognition in Fathers, provoke not your children to anger. a computational medium are confronted through review of -Fetyukovich1 current dynamic systems models of cognitive agency. In these models, aspects of the human condition are repro- Current events have my head spinning. I wake daily to read duced that may in other forms be considered incomputable, on the affairs of the world, only to wonder if it is all a i.e., political voice, predictive planning, and consciousness. dream. But for the suffering, from wedding parties bombed The paper then argues that simulations afford a unique to civil activists singled out for government assassination, potential to secure a post-human future, and may be nec- evidence invites speculation that our globe might well be essary for a pre-post-human civilization like our own to part of a particularly violent and extraordinarily realistic achieve and to maintain a post-human situation. Long-s- video game, the product of a team of evil demons for their tanding philosophical interest in tools of this nature for exclusive enjoyment at the expense of anyone with Aristotle’s ‘‘statesman’’ and more recently for E.O. Wilson compassion and an eye to a flourishing future in which in the 1990s is observed. Self-extinction-level threats from such existential threats are finally resolved. At least, in State and individual levels of organization are compared, imagining it so, perceived reality is easier to accept. and a likely dependence on large-scale psychologically As fanciful as this may seem, Nick Bostrom’s recently patched ‘‘simulation argument’’ (Bostrom 2003; Bostrom and Kulczycki 2011) purports to demonstrate the proba- & Jeffrey White bility that we exist now in a simulation of similar [email protected] 1 Humanities and Social Sciences, Korean Advanced Institute 1 The straight-talking counselor from The Brothers Karamazov as of Science and Technology (KAIST), Daejeon, South Korea translated in Dostoyevsky and McDuff (2003, pp. 949–950). 123 172 AI & Soc (2016) 31:171–190 complexity, an ‘‘ancestor simulation’’ of a period prior to worry about our knowledge of the external world’’ and that in which a civilization more advanced than our own— axiological skepticism ‘‘is the concern that no genuine ‘‘post-human’’—becomes able to simulate such a state of value could attach’’ to a simulated existence (Cogburn and affairs as ours—‘‘human.’’ By this argument, one (or more) Silcox 2014, p. 563). Together, these forms of skepticism of the following propositions is (are) true: seem to encourage an attitude that simulated life is—on the probability conferred by Bostrom’s simulation argument— (1) The human species is very likely to go extinct not real, and this may encourage the sentiment that suf- before reaching a post-human stage. fering is—so long as it is not my own—inconsequential. (2) The fraction of post-human civilizations that are Indeed, the very word ‘‘simulation’’ carries the sense that interested in running a significant number of ancestor everything exists only as an ‘‘if’’ and is accompanied by the simulations is extremely small. feeling ‘‘as-if’’ life mattered, ‘‘as-if’’ justice mattered, and (3) We are almost certainly living in a computer so on and nothing more. We are ‘‘as-if’’ transported into the simulation (Bostrom and Kulczycki 2011, p. 54). Brothers Karamazov, confronted by a maddening—and Proposition 1 expresses confidence in humanity’s potential murderous—moral nihilism. Bostrom himself understands to destroy itself, as ‘‘we are likely to go extinct as a result this implication, but reasons that ‘‘properly understood … of the development of some powerful but dangerous the truth of (3) should have no tendency to make us ‘go technology’’ (Bostrom 2003, pp. 251–252). Accepting 1, crazy’’’ (Bostrom 2003, p. 255). What we have taken for however, also implies that some human civilizations may good should remain taken for good, and life should con- not go extinct or extinguish themselves. They become tinue as it has, simulated or not. ‘‘post-human,’’ representing a situation in which ‘‘hu- My suspicions are that Bostrom does not take our sim- mankind has acquired most of the technological capabil- ulated condition seriously enough. The following paper ities that one can currently show to be consistent with argues that the Bostrom-scale simulations under consider- physical laws and with material and energy constraints’’ ation may prove a necessity rather than a luxury for a (Bostrom 2003, p. 245). From within such a situation, it civilization with post-human aspirations. The next step in follows easily that realistic simulations of ‘‘ancestor’’ low- the fugue confronts objections deriving from similarities tech level 1 worlds such as our own are a matter of course. between simulations and brains in vats (BIVs). The third Post-human civilizations interested will create limitless and fourth steps build the complimentary case for psy- simulations, and as their simulations might also create chologically realistic simulations in the near term, while simulations, the ratio of simulated to natural worlds spun the fifth confronts a different form of skepticism with a out accordingly confirms near certainty in the third review of promising work in neurorobotics. Finally, the proposition, that this world, ours, is simulated. second half of the paper argues that the second of Bos- A ‘‘patch’’ offered in 2011 (Bostrom and Kulczycki trom’s three propositions should be modified on grounds 2011) aims to correct for concerns that too few simulating that pre-post-human civilizations like our own, ‘‘human,’’ populations survive proposition 1, and become 2, relative probably require the ongoing development of large-scale those surviving without such simulations. The concession psychologically realistic social simulations in order to that the patch offers is simply to accept the original con- achieve and then to maintain a post-human status, thereby clusion so long as simulating post-human worlds are not affirming his conclusion. unduly outnumbered by non-simulating worlds, in which case the simple probability delivers either ambiguous or opposite results. And with this, the reasoning driving 2 Skepticism and reality Bostrom’s simulation argument is quite clear—though there may be few civilizations which achieve a capacity to …the scientific spirit does not require us to blind produce simulations of the complexity in terms of which ourselves to the practical consequences which hang we find ourselves today, some which do achieve that upon the solution to not a few scientific problems. capacity will exercise it, and in exercising it will create -Rashdall (1914, p. 199) many more simulated ancestor worlds like our own than could ever have existed otherwise. So, given the proba- bility that simulated ancestor worlds far exceed natural How does it happen that a whole generation of sci- alternatives, we must accept proposition number 3. entific experts is blind to obvious facts? This is a contentious result for the skepticism that it -Dyson (2015) seems to represent, skepticism that may be analyzed into two types, epistemic and axiological (Cogburn and Silcox Confronted with the proposition that we exist as simula- 2014). Epistemic skepticism ‘‘is the traditional sort of tions, one may respond in a number of ways. One may 123 AI & Soc (2016) 31:171–190 173 object to Bostrom’s characterization of the nature of the that I have two hands’’ for example ‘‘remains undefeated’’ simulated condition, for example. He writes that ‘‘it would (Huemer 2000, p. 411). In the end, it is up to the skeptic to suffice for the generation of subjective experiences that the address this challenge from direct realism. My two hands computational processes of a human brain are structurally and what I make with them are much clearer and more replicated in suitably fine-grained detail, such as on the distinct than is any simulation; thus, the simulationist is level of individual synapses’’ (Bostrom 2003, p. 244).2 This either deluded, or wrong. characterization is close enough to envatted brain arche- But, what if the simulationist weakens the argument, in types to suffer similar faults, perhaps ending any further some way? Anticipating such a move, Davies (1997) has discussion about any simulated condition before it gets recalled Graeme Forbes’ (1985) contention that ‘‘the started. skeptic can evade Putnam’s argument and achieve all she To this end, Putnam (1982) famously argued for the has ever wanted by switching to the hypothesis that I am necessary falsity of the proposition that we are brains in ‘relevantly like a brain in a vat’’’ (as quoted in Davies vats (BIVs).
Recommended publications
  • An Evolutionary Heuristic for Human Enhancement
    18 TheWisdomofNature: An Evolutionary Heuristic for Human Enhancement Nick Bostrom and Anders Sandberg∗ Abstract Human beings are a marvel of evolved complexity. Such systems can be difficult to enhance. When we manipulate complex evolved systems, which are poorly understood, our interventions often fail or backfire. It can appear as if there is a ‘‘wisdom of nature’’ which we ignore at our peril. Sometimes the belief in nature’s wisdom—and corresponding doubts about the prudence of tampering with nature, especially human nature—manifest as diffusely moral objections against enhancement. Such objections may be expressed as intuitions about the superiority of the natural or the troublesomeness of hubris, or as an evaluative bias in favor of the status quo. This chapter explores the extent to which such prudence-derived anti-enhancement sentiments are justified. We develop a heuristic, inspired by the field of evolutionary medicine, for identifying promising human enhancement interventions. The heuristic incorporates the grains of truth contained in ‘‘nature knows best’’ attitudes while providing criteria for the special cases where we have reason to believe that it is feasible for us to improve on nature. 1.Introduction 1.1. The wisdom of nature, and the special problem of enhancement We marvel at the complexity of the human organism, how its various parts have evolved to solve intricate problems: the eye to collect and pre-process ∗ Oxford Future of Humanity Institute, Faculty of Philosophy and James Martin 21st Century School, Oxford University. Forthcoming in Enhancing Humans, ed. Julian Savulescu and Nick Bostrom (Oxford: Oxford University Press) 376 visual information, the immune system to fight infection and cancer, the lungs to oxygenate the blood.
    [Show full text]
  • Heidegger, Personhood and Technology
    Comparative Philosophy Volume 2, No. 2 (2011): 23-49 Open Access / ISSN 2151-6014 www.comparativephilosophy.org THE FUTURE OF HUMANITY: HEIDEGGER, PERSONHOOD AND TECHNOLOGY MAHON O‟BRIEN ABSTRACT: This paper argues that a number of entrenched posthumanist positions are seriously flawed as a result of their dependence on a technical interpretive approach that creates more problems than it solves. During the course of our discussion we consider in particular the question of personhood. After all, until we can determine what it means to be a person we cannot really discuss what it means to improve a person. What kinds of enhancements would even constitute improvements? This in turn leads to an examination of the technical model of analysis and the recurring tendency to approach notions like personhood using this technical model. In looking to sketch a Heideggerian account of personhood, we are reaffirming what we take to be a Platonic skepticism concerning technical models of inquiry when it comes to certain subjects. Finally we examine the question as to whether the posthumanist looks to apply technology‟s benefits in ways that we have reflectively determined to be useful or desirable or whether it is technology itself (or to speak as Heidegger would – the “essence” of technology) which prompts many posthumanists to rely on an excessively reductionist view of the human being. Keywords: Heidegger, posthumanism, technology, personhood, temporality A significant number of Posthumanists1 advocate the techno-scientific enhancement of various human cognitive and physical capacities. Recent trends in posthumanist theory have witnessed the collective emergence, in particular, of a series of analytically oriented philosophers as part of the Future of Humanity Institute at O‟BRIEN, MAHON: IRCHSS Postdoctoral Research Fellow, School of Philosophy, University College Dublin, Ireland.
    [Show full text]
  • Global Catastrophic Risks Survey
    GLOBAL CATASTROPHIC RISKS SURVEY (2008) Technical Report 2008/1 Published by Future of Humanity Institute, Oxford University Anders Sandberg and Nick Bostrom At the Global Catastrophic Risk Conference in Oxford (17‐20 July, 2008) an informal survey was circulated among participants, asking them to make their best guess at the chance that there will be disasters of different types before 2100. This report summarizes the main results. The median extinction risk estimates were: Risk At least 1 million At least 1 billion Human extinction dead dead Number killed by 25% 10% 5% molecular nanotech weapons. Total killed by 10% 5% 5% superintelligent AI. Total killed in all 98% 30% 4% wars (including civil wars). Number killed in 30% 10% 2% the single biggest engineered pandemic. Total killed in all 30% 10% 1% nuclear wars. Number killed in 5% 1% 0.5% the single biggest nanotech accident. Number killed in 60% 5% 0.05% the single biggest natural pandemic. Total killed in all 15% 1% 0.03% acts of nuclear terrorism. Overall risk of n/a n/a 19% extinction prior to 2100 These results should be taken with a grain of salt. Non‐responses have been omitted, although some might represent a statement of zero probability rather than no opinion. 1 There are likely to be many cognitive biases that affect the result, such as unpacking bias and the availability heuristic‒‐well as old‐fashioned optimism and pessimism. In appendix A the results are plotted with individual response distributions visible. Other Risks The list of risks was not intended to be inclusive of all the biggest risks.
    [Show full text]
  • Sharing the World with Digital Minds1
    Sharing the World with Digital Minds1 (2020). Draft. Version 1.8 Carl Shulman† & Nick Bostrom† [in Clarke, S. & Savulescu, J. (eds.): Rethinking Moral Status (Oxford University Press, forthcoming)] Abstract The minds of biological creatures occupy a small corner of a much larger space of possible minds that could be created once we master the technology of artificial intelligence. Yet many of our moral intuitions and practices are based on assumptions about human nature that need not hold for digital minds. This points to the need for moral reflection as we approach the era of advanced machine intelligence. Here we focus on one set of issues, which arise from the prospect of digital minds with superhumanly strong claims to resources and influence. These could arise from the vast collective benefits that mass-produced digital minds could derive from relatively small amounts of resources. Alternatively, they could arise from individual digital minds with superhuman moral status or ability to benefit from resources. Such beings could contribute immense value to the world, and failing to respect their interests could produce a moral catastrophe, while a naive way of respecting them could be disastrous for humanity. A sensible approach requires reforms of our moral norms and institutions along with advance planning regarding what kinds of digital minds we bring into existence. 1. Introduction Human biological nature imposes many practical limits on what can be done to promote somebody’s welfare. We can only live so long, feel so much joy, have so many children, and benefit so much from additional support and resources. Meanwhile, we require, in order to flourish, that a complex set of physical, psychological, and social conditions be met.
    [Show full text]
  • Global Catastrophic Risks
    Global Catastrophic Risks Edited by Nick Bostrom Milan M. Cirkovic OXPORD UNIVERSITY PRESS Contents Acknowledgements ν Foreword vii Martin J. Rees 1 Introduction 1 Nick Bostrom and Milan M. Cirkoviô 1.1 Why? 1 1.2 Taxonomy and organization 2 1.3 Part I: Background 7 1.4 Part II: Risks from nature 13 1.5 Part III: Risks from unintended consequences 15 1.6 Part IV: Risks from hostile acts 20 1.7 Conclusions and future directions 27 Part I Background 31 2 Long-term astrophysics! processes 33 Fred C. Adams 2.1 Introduction: physical eschatology 33 2.2 Fate of the Earth 34 2.3 Isolation of the local group 36 2.4 Collision with Andromeda 36 2.5 The end of stellar evolution 38 2.6 The era of degenerate remnants 39 2.7 The era of black holes 41 2.8 The Dark Era and beyond 41 2.9 life and information processing 43 2.10 Conclusion 44 Suggestions for further reading 45 References 45 3 Evolution theory and the future of humanity 48 Christopher Wills· 3.1 Introduction 48 3.2 The causes of evolutionary change 49 xiv Contents 3.3 Environmental changes and evolutionary changes 50 3.3.1 Extreme evolutionary changes 51 3.3.2 Ongoing evolutionary changes 53 3.3.3 Changes in the cultural environment 56 3.4 Ongoing human evolution 61 3.4.1 Behavioural evolution 61 3.4.2 The future of genetic engineering 63 3.4.3 The evolution of other species, including those on which we depend 64 3.5 Future evolutionary directions 65 3.5.1 Drastic and rapid climate change without changes in human behaviour 66 3.5.2 Drastic but slower environmental change accompanied by changes in human behaviour 66 3.5.3 Colonization of new environments by our species 67 Suggestions for further reading 68 References 69 4 Millennial tendencies in responses to apocalyptic threats 73 James J.
    [Show full text]
  • Human Genetic Enhancements: a Transhumanist Perspective
    Human Genetic Enhancements: A Transhumanist Perspective NICK BOSTROM Oxford University, Faculty of Philosophy, 10 Merton Street, Oxford, OX1 4JJ, U. K. [Preprint of paper published in the Journal of Value Inquiry, 2003, Vol. 37, No. 4, pp. 493-506] 1. What is Transhumanism? Transhumanism is a loosely defined movement that has developed gradually over the past two decades. It promotes an interdisciplinary approach to understanding and evaluating the opportunities for enhancing the human condition and the human organism opened up by the advancement of technology. Attention is given to both present technologies, like genetic engineering and information technology, and anticipated future ones, such as molecular nanotechnology and artificial intelligence.1 The enhancement options being discussed include radical extension of human health-span, eradication of disease, elimination of unnecessary suffering, and augmentation of human intellectual, physical, and emotional capacities.2 Other transhumanist themes include space colonization and the possibility of creating superintelligent machines, along with other potential developments that could profoundly alter the human condition. The ambit is not limited to gadgets and medicine, but encompasses also economic, social, institutional designs, cultural development, and psychological skills and techniques. Transhumanists view human nature as a work-in-progress, a half-baked beginning that we can learn to remold in desirable ways. Current humanity need not be the endpoint 1 of evolution. Transhumanists
    [Show full text]
  • F.3. the NEW POLITICS of ARTIFICIAL INTELLIGENCE [Preliminary Notes]
    F.3. THE NEW POLITICS OF ARTIFICIAL INTELLIGENCE [preliminary notes] MAIN MEMO pp 3-14 I. Introduction II. The Infrastructure: 13 key AI organizations III. Timeline: 2005-present IV. Key Leadership V. Open Letters VI. Media Coverage VII. Interests and Strategies VIII. Books and Other Media IX. Public Opinion X. Funders and Funding of AI Advocacy XI. The AI Advocacy Movement and the Techno-Eugenics Movement XII. The Socio-Cultural-Psychological Dimension XIII. Push-Back on the Feasibility of AI+ Superintelligence XIV. Provisional Concluding Comments ATTACHMENTS pp 15-78 ADDENDA pp 79-85 APPENDICES [not included in this pdf] ENDNOTES pp 86-88 REFERENCES pp 89-92 Richard Hayes July 2018 DRAFT: NOT FOR CIRCULATION OR CITATION F.3-1 ATTACHMENTS A. Definitions, usage, brief history and comments. B. Capsule information on the 13 key AI organizations. C. Concerns raised by key sets of the 13 AI organizations. D. Current development of AI by the mainstream tech industry E. Op-Ed: Transcending Complacency on Superintelligent Machines - 19 Apr 2014. F. Agenda for the invitational “Beneficial AI” conference - San Juan, Puerto Rico, Jan 2-5, 2015. G. An Open Letter on Maximizing the Societal Benefits of AI – 11 Jan 2015. H. Partnership on Artificial Intelligence to Benefit People and Society (PAI) – roster of partners. I. Influential mainstream policy-oriented initiatives on AI: Stanford (2016); White House (2016); AI NOW (2017). J. Agenda for the “Beneficial AI 2017” conference, Asilomar, CA, Jan 2-8, 2017. K. Participants at the 2015 and 2017 AI strategy conferences in Puerto Rico and Asilomar. L. Notes on participants at the Asilomar “Beneficial AI 2017” meeting.
    [Show full text]
  • Philos 121: January 24, 2019 Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence” (Skip Pp
    Philos 121: January 24, 2019 Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence” (skip pp. 7–13) Jeremy Bentham, Introduction to the Principles of Morals and Legislation, Ch. I–II Suppose we tell AI: “Make paper clips.” Unfortunately, AI has become advanced enough to design ever more intelligent AI, which in turn designs ever more intelligent AI, and so on. AI then tries to turn the universe into a paper-clip making factory, diverting all resources from food production into paper-clip production. Getting hungry, we try to turn it off. AI interprets us and our attempts to turn it off as obstacles to making paper clips. So it eliminates the obstacles… So… what should we tell AI to do, if we have only one chance to get it right? This is very close to the basic question of moral philosophy: In the most basic and general terms, what should we, or any other intelligent agent, do? The road to utilitarianism: First premise (Hedonism): What is good for us as an end is pleasure, and what is bad for us as an end is pain. Good as a means vs. good as an end: • Good as a means: good because it helps to bring about something else that is good. • But this chain of means has to end somewhere. There has to be something that is: • Good as an end: good because of what it is in itself and not because of what it brings about. What is good for us as an end? The answer at the end of every chain seems to be: pleasure and the absence of pain.
    [Show full text]
  • Existential Risk Prevention As Global Priority
    Global Policy Volume 4 . Issue 1 . February 2013 15 Existential Risk Prevention as Global Priority Nick Bostrom Research Article University of Oxford Abstract Existential risks are those that threaten the entire future of humanity. Many theories of value imply that even relatively small reductions in net existential risk have enormous expected value. Despite their importance, issues surrounding human-extinction risks and related hazards remain poorly understood. In this article, I clarify the concept of existential risk and develop an improved classification scheme. I discuss the relation between existential risks and basic issues in axiology, and show how existential risk reduction (via the maxipok rule) can serve as a strongly action-guiding principle for utilitarian concerns. I also show how the notion of existential risk suggests a new way of thinking about the ideal of sustainability. Policy Implications • Existential risk is a concept that can focus long-term global efforts and sustainability concerns. • The biggest existential risks are anthropogenic and related to potential future technologies. • A moral case can be made that existential risk reduction is strictly more important than any other global public good. • Sustainability should be reconceptualised in dynamic terms, as aiming for a sustainable trajectory rather than a sus- tainable state. • Some small existential risks can be mitigated today directly (e.g. asteroids) or indirectly (by building resilience and reserves to increase survivability in a range of extreme scenarios) but it is more important to build capacity to improve humanity’s ability to deal with the larger existential risks that will arise later in this century. This will require collective wisdom, technology foresight, and the ability when necessary to mobilise a strong global coordi- nated response to anticipated existential risks.
    [Show full text]
  • Desire, Time, and Ethical Weight
    Desire, Time, and Ethical Weight Dr. Nick Bostrom Department of Philosophy Yale University New Haven, Connecticut 06520 U. S. A. Fax: (203) 432-7950 Phone: (203) 500-0021 Email: [email protected] Desire, Time, and Ethical Weight 1. Introduction We want more than we can have. So we must somehow weigh our desires in order to determine which of them it is most important that we satisfy. When the objective is simply to maximize the satisfaction of one’s own current desires (“egoism of the present moment”), decision theory provides a well-worked out, albeit highly idealized, framework for conducting the requisite deliberation. It takes into account the strength of the various desires one has and the subjective probabilities of the possible outcomes of the actions one is choosing between (and, on causal decision theories, also one’s beliefs about the causal relations between actions and outcomes1). All that an egoist of the present moment has to do is to introspectively ascertain these parameters, calculate (or intuitively estimate) the expected utility of the various choices, and then pick the one with the highest expected utility. But what if one is not an egoist of the present moment? What if, for example, one’s objective is to maximize desire-satisfaction simpliciter, or even just satisfaction of one’s own desires, past, present, and future? Then there is a complication. For in order to get a measure of the total preference satisfaction in these cases, we have to compare preferences held at different times and for different durations. How to operationally compare the strengths of different people’s preferences, or the preferences of the same person at different times, is a notorious challenge for social choice theory.
    [Show full text]
  • Identifying and Assessing the Drivers of Global Catastrophic Risk: a Review and Proposal for the Global Challenges Foundation
    Identifying and Assessing the Drivers of Global Catastrophic Risk: A Review and Proposal for the Global Challenges Foundation Simon Beard and Phil Torres Summary The goal of this report is to review and assess methods and approaches for assessing the drivers of global catastrophic risk. The review contains five sections: (1) A conceptual overview setting out our understanding of the concept of global catastrophic risks ​ (GCRs), their drivers, and how they can be assessed. (3) A brief historical overview of the field of GCR research, indicating how our understanding of the drivers of GCRs has developed. (2) A summary of existing studies that seek to quantify the drivers of GCR by assessing the likelihood that different causes will precipitate a global catastrophe. (4) A critical evaluation of the usefulness of this research given the conceptual framework outlined in section 1 and a review of emerging conceptual, evaluative and risk assessment tools that may allow for better assessments of the drivers of GCRs in the future. (5) A proposal for how the Global Challenges Foundation could work to most productively improve our understanding of the drivers of GCRs given the findings of sections 2, 3, and 4. Further information on these topics will be included in three appendices, each of which is an as-yet-unpublished academic paper by the authors, alone or in collaboration with others. These are: (A) Transhumanism, Effective Altruism, and Systems Theory: A History and Analysis of Existential Risk Studies. (B) An Analysis and Evaluation of Methods Currently Used to Quantify the Likelihood of Existential Hazards. (C) Assessing Climate Change’s Contribution to Global Catastrophic Risk.
    [Show full text]
  • A Conversation with Stuart Russell on February 28, 2014
    A conversation with Stuart Russell on February 28, 2014 Participants • Stuart Russell –Professor of Computer Science and Smith-Zadeh Professor in Engineering, University of California, Berkeley; Advisory Board Member, Centre for the Study of Existential Risk (CSER) • Alexander Berger – Senior Research Analyst, GiveWell • Jacob Steinhardt – Graduate student, Computer Science, Stanford University Note: This set of notes was compiled by GiveWell and gives an overview of the major points made by Stuart Russell. Summary GiveWell spoke with Stuart Russell as part of its investigation of potential social implications of artificial intelligence (AI) research. Conversation topics included: assessing risks from super-intelligent systems, attitudes in the AI research community, and logical next steps in research. Incentives for AI development Significant advances in AI, once a distant theoretical possibility, are increasingly likely. The economic incentives for improvements to AI are immense. Individual companies or researchers can gain billions of dollars for marginal improvements in AI algorithms. As more consumers and industries use AI techniQues, the market pressure for advancements in AI will likely grow. The development of AI may mirror the development of nuclear weapons in that, once the technology exists, there will be a race to improve and deploy it. In such situations, regulation may struggle to keep up. Ethical issues related to AI Some ethical issues related to AI are already being discussed. For example, there are difficult issues around modern autonomous weapon systems. While official US policy is that drone strikes are only to be permitted with the authorization of a human operator, the current design and deployment of drones could be modified easily to allow the operator to send the drone to acQuire and eliminate human targets based on its programmed criteria.
    [Show full text]