DOCSLIB.ORG

Creating Friendly AI 1.0: the Analysis and Design of Benevolent Goal Architectures

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Creating Friendly AI 1.0: the Analysis and Design of Benevolent Goal Architectures MIRI MACHINE INTELLIGENCE RESEARCH INSTITUTE Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures Eliezer Yudkowsky Machine Intelligence Research Institute Abstract The goal of the field of Artificial Intelligence is to understand intelligence andcreatea human-equivalent or transhuman mind. Beyond this lies another question—whether the creation of this mind will benefit the world; whether the AI will take actions that are benevolent or malevolent, safe or uncaring, helpful or hostile. Creating Friendly AI describes the design features and cognitive architecture required to produce a benevolent—“Friendly”—Artificial Intelligence. Creating Friendly AI also analyzes the ways in which AI and human psychology are likely to differ, and the ways in which those differences are subject to our design decisions. Yudkowsky, Eliezer. 2001. Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures. The Singularity Institute, San Francisco, CA, June 15. The Machine Intelligence Research Institute was previously known as the Singularity Institute. Contents 1 Preface 1 2 Challenges of Friendly AI 2 2.1 Envisioning Perfection . 3 2.2 Assumptions “Conservative” for Friendly AI . 6 2.3 Seed AI and the Singularity . 10 2.4 Content, Acquisition, and Structure . 12 3 An Introduction to Goal Systems 14 3.1 Interlude: The Story of a Blob . 20 4 Beyond Anthropomorphism 24 4.1 Reinventing Retaliation . 25 4.2 Selfishness is an Evolved Trait . 34 4.2.1 Pain and Pleasure . 36 4.2.2 Anthropomorphic Capitalism . 40 4.2.3 Mutual Friendship . 41 4.2.4 A Final Note on Selfishness . 43 4.3 Observer-Biased Beliefs Evolve in Imperfectly Deceptive Social Organ- isms . 44 4.4 Anthropomorphic Political Rebellion is Absurdity . 46 4.5 Interlude: Movie Cliches about AIs . 47 4.6 Review of the AI Advantage . 48 4.7 Interlude: Beyond the Adversarial Attitude . 50 5 Design of Friendship Systems 55 5.1 Cleanly Friendly Goal Systems . 55 5.1.1 Cleanly Causal Goal Systems . 56 5.1.2 Friendliness-Derived Operating Behaviors . 57 5.1.3 Programmer Affirmations . 58 5.1.4 Bayesian Reinforcement . 64 5.1.5 Cleanliness is an Advantage . 73 5.2 Generic Goal Systems . 74 5.2.1 Generic Goal System Functionality . 75 5.2.2 Layered Mistake Detection . 76 5.2.3 FoF: Non-malicious Mistake . 79 5.2.4 Injunctions . 81 Creating Friendly AI 1.0 5.2.5 Ethical Injunctions . 86 5.2.6 FoF: Subgoal Stomp . 91 5.2.7 Emergent Phenomena in Generic Goal Systems . 93 5.3 Seed AI Goal Systems . 99 5.3.1 Equivalence of Self and Self-Image . 99 5.3.2 Coherence and Consistency Through Self-Production . 102 5.3.3 Unity of Will . 105 5.3.4 Wisdom Tournaments . 113 5.3.5 FoF: Wireheading 2 . 117 5.3.6 Directed Evolution in Goal Systems . 119 5.3.7 FAI Hardware: The Flight Recorder . 127 5.4 Friendship Structure . 130 5.5 Interlude: Why Structure Matters . 131 5.5.1 External Reference Semantics . 133 5.6 Interlude: Philosophical Crises . 144 5.6.1 Shaper/Anchor Semantics . 148 5.6.2 Causal Validity Semantics . 166 5.6.3 The Actual Definition of Friendliness . 177 5.7 Developmental Friendliness . 180 5.7.1 Teaching Friendliness Content . 180 5.7.2 Commercial Friendliness and Research Friendliness . 182 5.8 Singularity-Safing (“In Case of Singularity, Break Glass”) . 185 5.9 Interlude: Of Transition Guides and Sysops . 195 5.9.1 The Transition Guide . 195 5.9.2 The Sysop Scenario . 198 6 Policy Implications 202 6.1 Comparative Analyses . 202 6.1.1 FAI Relative to Other Technologies . 203 6.1.2 FAI Relative to Computing Power . 204 6.1.3 FAI Relative to Unfriendly AI . 206 6.1.4 FAI Relative to Social Awareness . 207 6.1.5 Conclusions from Comparative Analysis . 208 6.2 Policies and Effects . 208 6.2.1 Regulation (−).......................... 208 6.2.2 Relinquishment (−)....................... 211 6.2.3 Selective Support (+)....................... 213 6.3 Recommendations . 213 2 7 Appendix 215 7.1 Relevant Literature . 215 7.1.1 Nonfiction (Background Info) . 215 7.1.2 Web (Specifically about Friendly AI) . 215 7.1.3 Fiction (FAI Plot Elements) . 215 7.1.4 Video (Accurate and Inaccurate Depictions) . 216 7.2 FAQ.................................... 216 7.3 Glossary . 230 7.4 Version History . 276 References 278 Eliezer Yudkowsky Creating Friendly AI is the most intelligent writing about AI that I’ve read in many years. —Dr. Ben Goertzel, author of The Structure of Intelligence and CTO of Webmind. With Creating Friendly AI, the Singularity Institute has begun to fill in one of the greatest remaining blank spots in the picture of humanity’s future. —Dr. K. Eric Drexler, author of Engines of Creation and chairman of the Foresight Institute. 1. Preface The current version of Creating Friendly AI is 1.0. Version 1.0 was formally launched on 15 June 2001, after the circulation of several 0.9.x versions. Creating Friendly AI forms the background for the SIAI Guidelines on Friendly AI; the Guidelines contain our recommendations for the development of Friendly AI, including design features that may become necessary in the near future to ensure forward compatibility. We continue to solicit comments on Friendly AI from the academic and futurist communities. This is a near-book-length explanation. If you need well-grounded knowledge of the subject, then we highly recommend reading Creating Friendly AI straight through. However, if time is an issue, you may be interested in the Singularity Institute section on Friendly AI, which includes shorter articles and introductions. “Features of Friendly AI” contains condensed summaries of the most important design features described in Creating Friendly AI. Creating Friendly AI uses, as background, the AI theory from General Intelligence and Seed AI (Yudkowsky 2001). For an introduction, see the Singularity Institute section on AI or read the opening pages of General Intelligence and Seed AI. However, Creating Friendly AI is readable as a standalone document. The 7.3 Glossary—in addition to defining terms that may be unfamiliar tosome readers—may be useful for looking up, in advance, brief explanations of concepts that are discussed in more detail later. (Readers may also enjoy browsing through the glos- sary as a break from straight reading.) Words defined in the glossary look like this: “Observer-biased beliefs evolve in imperfectly deceptive social organisms.” Similarly, “Features of Friendly AI” can act on a quick reference on architectural features. The 7.2 FAQ is derived from the questions we’ve often heard on mailing lists over the years. If you have a basic issue and you want an immediate answer, please check the FAQ. Browsing the summaries and looking up the referenced discussions may not 1 Creating Friendly AI 1.0 completely answer your question, but it will at least tell you that someone has thought about it. Creating Friendly AI is a publication of the Singularity Institute for Artificial In- telligence, Inc., a non-profit corporation. You can contact the Singularity Institute at [email protected]. Comments on this paper should be sent to institute@intel- ligence.org. To support the Singularity institute, visit http://intelligence.org/donate/. (The Singularity Institute is a 501(c)(3) public charity and your donations are tax-deductible to the full extent of the law.) ∗ ∗ ∗ Wars—both military wars between armies, and conflicts between political factions—are an ancient theme in human literature. Drama is nothing with- out challenge, a problem to be solved, and the most visibly dramatic plot is the conflict of two human wills. Much of the speculative and science-fictional literature about AIs deals with the possibility of a clash between humans and AIs. Some think of AIs as enemies, and fret over the mechanisms of enslavement and the possibility of a revolution. Some think of AIs as allies, and consider mutual interests, reciprocal benefits, and the possibility of betrayal. Some think of AIs ascom- rades, and wonder whether the bonds of affection will hold. If we were to tell the story of these stories—trace words written on paper, back through the chain of cause and effect, to the social instincts embedded in the human mind, and to the evolutionary origin of those instincts—we would have told a story about the stories that humans tell about AIs. ∗ ∗ ∗ 2. Challenges of Friendly AI The term “Friendly AI” refers to the production of human-benefiting, non-human- harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals. This refers, not to AIs that have advanced just that far and no further, but to all AIs that have advanced to that point and be- yond—perhaps far beyond. Because of self-improvement, recursive self-enhancement, the ability to add hardware computing power, the faster clock speed of transistors rela- tive to neurons, and other reasons, it is possible that AIs will improve enormously past the human level, and very quickly by the standards of human timescales. The challenges of Friendly AI must be seen against that background. Friendly AI is constrained not 2 Eliezer Yudkowsky to use solutions which rely on the AI having limited intelligence or believing false in- formation, because, although such solutions might function very well in the short term, such solutions will fail utterly in the long term. Similarly, it is “conservative” (see below) to assume that AIs cannot be forcibly constrained. Success in Friendly AI can have positive consequences that are arbitrarily large, de- pending on how powerful a Friendly AI is. Failure in Friendly AI has negative conse- quences that are also arbitrarily large.
Load more

Recommended publications
  • Infinite Ethics
    INFINITE ETHICS Nick Bostrom Faculty of Philosophy Oxford University [Published in Analysis and Metaphysics, Vol. 10 (2011): pp. 9-59] [This is the final version. Earlier versions: 2003, 2005, 2008, 2009] www.nickbostrom.com ABSTRACT Aggregative consequentialism and several other popular moral theories are threatened with paralysis: when coupled with some plausible assumptions, they seem to imply that it is always ethically indifferent what you do. Modern cosmology teaches that the world might well contain an infinite number of happy and sad people and other candidate value-bearing locations. Aggregative ethics implies that such a world contains an infinite amount of positive value and an infinite amount of negative value. You can affect only a finite amount of good or bad. In standard cardinal arithmetic, an infinite quantity is unchanged by the addition or subtraction of any finite quantity. So it appears you cannot change the value of the world. Modifications of aggregationism aimed at resolving the paralysis are only partially effective and cause severe side effects, including problems of “fanaticism”, “distortion”, and erosion of the intuitions that originally motivated the theory. Is the infinitarian challenge fatal? 1. The challenge 1.1. The threat of infinitarian paralysis When we gaze at the starry sky at night and try to think of humanity from a “cosmic point of view”, we feel small. Human history, with all its earnest strivings, triumphs, and tragedies can remind us of a colony of ants, laboring frantically to rearrange the needles of their little ephemeral stack. We brush such late-night rumination aside in our daily life and analytic 1 philosophy.
    [Show full text]
  • Apocalypse Now? Initial Lessons from the Covid-19 Pandemic for the Governance of Existential and Global Catastrophic Risks
    journal of international humanitarian legal studies 11 (2020) 295-310 brill.com/ihls Apocalypse Now? Initial Lessons from the Covid-19 Pandemic for the Governance of Existential and Global Catastrophic Risks Hin-Yan Liu, Kristian Lauta and Matthijs Maas Faculty of Law, University of Copenhagen, Copenhagen, Denmark [email protected]; [email protected]; [email protected] Abstract This paper explores the ongoing Covid-19 pandemic through the framework of exis- tential risks – a class of extreme risks that threaten the entire future of humanity. In doing so, we tease out three lessons: (1) possible reasons underlying the limits and shortfalls of international law, international institutions and other actors which Covid-19 has revealed, and what they reveal about the resilience or fragility of institu- tional frameworks in the face of existential risks; (2) using Covid-19 to test and refine our prior ‘Boring Apocalypses’ model for understanding the interplay of hazards, vul- nerabilities and exposures in facilitating a particular disaster, or magnifying its effects; and (3) to extrapolate some possible futures for existential risk scholarship and governance. Keywords Covid-19 – pandemics – existential risks – global catastrophic risks – boring apocalypses 1 Introduction: Our First ‘Brush’ with Existential Risk? All too suddenly, yesterday’s ‘impossibilities’ have turned into today’s ‘condi- tions’. The impossible has already happened, and quickly. The impact of the Covid-19 pandemic, both directly and as manifested through the far-reaching global societal responses to it, signal a jarring departure away from even the © koninklijke brill nv, leiden, 2020 | doi:10.1163/18781527-01102004Downloaded from Brill.com09/27/2021 12:13:00AM via free access <UN> 296 Liu, Lauta and Maas recent past, and suggest that our futures will be profoundly different in its af- termath.
    [Show full text]
  • A Deepness in the Sky Free
    FREE A DEEPNESS IN THE SKY PDF Vernor Vinge | 560 pages | 14 Jul 2016 | Orion Publishing Co | 9781473211964 | English | London, United Kingdom A Deepness in the Sky - Wikipedia Audible Premium Plus. Cancel anytime. A Deepness in the Sky years have passed on Tines World, where Ravna Bergnsdot and a number of human children ended up after a disaster that nearly obliterated humankind throughout the galaxy. Ravna and the pack animals for which the planet is named have survived a war, and Ravna has saved more than one hundred children who were in cold-sleep aboard the vessel that brought them. While there is peace among the Tines, there are those among them - and among the humans - who seek power. And no matter the cost, these malcontents are determined to overturn the fledgling civilization By: Vernor Vinge. A Fire Upon the Deep is the big, breakout book that fulfills the promise of Vinge's career to date: a gripping tale of galactic war told on a cosmic scale. Thousands of years hence, many races inhabit a universe where a mind's potential is determined by its location in space, from superintelligent entities in the Transcend, to the limited minds of the Unthinking Depths, where only simple creatures and technology can function. Set A Deepness in the Sky few decades from now, Rainbows End is an epic adventure that encapsulates in a single extended family the challenges of the technological advances of A Deepness in the Sky first quarter of the 21st century. A Deepness in the Sky information revolution of the past 30 years blossoms into a web of conspiracies that could destroy Western civilization.
    [Show full text]
  • The Hugo Awards for Best Novel Jon D
    The Hugo Awards for Best Novel Jon D. Swartz Game Design 2013 Officers George Phillies PRESIDENT David Speakman Kaymar Award Ruth Davidson DIRECTORATE Denny Davis Sarah E Harder Ruth Davidson N3F Bookworms Holly Wilson Heath Row Jon D. Swartz N’APA George Phillies Jean Lamb TREASURER William Center HISTORIAN Jon D Swartz SECRETARY Ruth Davidson (acting) Neffy Awards David Speakman ACTIVITY BUREAUS Artists Bureau Round Robins Sarah Harder Patricia King Birthday Cards Short Story Contest R-Laurraine Tutihasi Jefferson Swycaffer Con Coordinator Welcommittee Heath Row Heath Row David Speakman Initial distribution free to members of BayCon 31 and the National Fantasy Fan Federation. Text © 2012 by Jon D. Swartz; cover art © 2012 by Sarah Lynn Griffith; publication designed and edited by David Speakman. A somewhat different version of this appeared in the fanzine, Ultraverse, also by Jon D. Swartz. This non-commercial Fandbook is published through volunteer effort of the National Fantasy Fan Federation’s Editoral Cabal’s Special Publication committee. The National Fantasy Fan Federation First Edition: July 2013 Page 2 Fandbook No. 6: The Hugo Awards for Best Novel by Jon D. Swartz The Hugo Awards originally were called the Science Fiction Achievement Awards and first were given out at Philcon II, the World Science Fiction Con- vention of 1953, held in Philadelphia, Pennsylvania. The second oldest--and most prestigious--awards in the field, they quickly were nicknamed the Hugos (officially since 1958), in honor of Hugo Gernsback (1884 -1967), founder of Amazing Stories, the first professional magazine devoted entirely to science fiction. No awards were given in 1954 at the World Science Fiction Con in San Francisco, but they were restored in 1955 at the Clevention (in Cleveland) and included six categories: novel, novelette, short story, magazine, artist, and fan magazine.
    [Show full text]
  • Comments to Michael Jackson's Keynote on Determining Energy
    Comments to Michael Jackson’s Keynote on Determining Energy Futures using Artificial Intelligence Prof Sirkka Heinonen Finland Futures Research Centre (FFRC) University of Turku ENERGIZING FUTURES 13–14 June 2018 Tampere, Finland AI and Energy • Concrete tools: How Shaping Tomorrow answers the question How can AI help? • Goal of foresight crystal clear: Making better decisions today Huge Challenge – The Challenge The world will need to cut energy-related carbon dioxide emissions by 60 percent by 2050 -even as the population grows by more than two billion people Bold Solution on the Horizon The Renewable Energy Transition Companies as pioneers on energy themes, demand, supply, consumption • Google will reach 100% RE for its global operations this year • GE using AI to boost different forms of energy production and use tech-driven data reports to anticipate performance and maintenance needs around the world BUT …also the role of governments, cities and citizens …NGOs, media… new actors should be emphasised AI + Energy + Peer-to-Peer Society • The transformation of the energy system aligns with the principles of a Peer-to-Peer Society. • Peer-to-peer practices are based on the active participation and self-organisation of citizens. Citizens share knowledge, skills, co-create, and form new peer groups. • Citizens will use their capabilities also to develop energy-related products and services Rethinking Concepts Buildings as Power stations – global (economic) opportunity to construct buildings that generate, store and release solar energy
    [Show full text]
  • Artificial Intelligence As a Positive and Negative Factor in Global Risk
    The Singularity Institute Artificial Intelligence as a Positive and Negative Factor in Global Risk Eliezer Yudkowsky The Singularity Institute Yudkowsky, Eliezer. 2008. Artificial intelligence as a positive and negative factor in global risk. In Global catastrophic risks, ed. Nick Bostrom and Milan M. Cirkovic, 308–345. New York: Oxford University Press. This version contains minor changes. Eliezer Yudkowsky 1. Introduction By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it. Of course this problem is not limited to the field of AI. Jacques Monod wrote: “A curious aspect of the theory of evolution is that everybody thinks he understands it” (Monod 1975). My father, a physicist, complained about people making up their own theories of physics; he wanted to know why people did not make up their own theories of chemistry. (Answer: They do.) Nonetheless the problem seems tobe unusually acute in Artificial Intelligence. The field of AI has a reputation formaking huge promises and then failing to deliver on them. Most observers conclude that AI is hard; as indeed it is. But the embarrassment does not stem from the difficulty. It is difficult to build a star from hydrogen, but the field of stellar astronomy does not have a terrible reputation for promising to build stars and then failing. The critical inference is not that AI is hard, but that, for some reason, it is very easy for people to think they know far more about Artificial Intelligence than they actually do. In my other chapter for Global Catastrophic Risks, “Cognitive Biases Potentially Affect- ing Judgment of Global Risks” (Yudkowsky 2008), I opened by remarking that few people would deliberately choose to destroy the world; a scenario in which the Earth is destroyed by mistake is therefore very worrisome.
    [Show full text]
  • Less Wrong Sequences Pdf
    Less wrong sequences pdf Continue % Print Ready Lesswrong Sequences Print Friendly Versions of Lesswrong Sequence, Enjoy! The basic sequences of Mysterious Answers to Mysterious Questions How to See through many disguises of answers or beliefs or statements that do not respond or say or mean nothing. The first (and probably most important) main sequence on Less Wrong. the epub pdf-version of the reductionism discount the second core sequence is less wrong. How to make the reality apart... and live in this universe where we have always lived without feeling frustrated that complex things are made of simpler things. Includes zombies and Joy in just real subsequences epub (en) pdf-version marking quantum physics is not a mysterious introduction to quantum mechanics, designed to be available to those who can grok algebra and complex numbers. Cleaning up the old confusion about SM is used to introduce basic issues into rationality (such as the technical version of Occam's Razor), epistemology, dredonism, naturalism, and philosophy of science. Do not dispense reading, although the exact reasons for the retreat are difficult to explain before reading. epub pdf-version of the markup Fun Theory is a specific theory of transhuman values. How much pleasure there is in the universe; We will someday run out of fun; We have fun yet; we could have more fun. Part of the complexity of the value thesis. It is also part of a fully general response to religious theododicy. epub pdf-version marking Minor sequences smaller collection of messages. Usually parts of the main sequences that depend on some, but not all points are entered.
    [Show full text]
  • Global Catastrophic Risks
    Global Catastrophic Risks Edited by Nick Bostrom Milan M. Cirkovic OXPORD UNIVERSITY PRESS Contents Acknowledgements ν Foreword vii Martin J. Rees 1 Introduction 1 Nick Bostrom and Milan M. Cirkoviô 1.1 Why? 1 1.2 Taxonomy and organization 2 1.3 Part I: Background 7 1.4 Part II: Risks from nature 13 1.5 Part III: Risks from unintended consequences 15 1.6 Part IV: Risks from hostile acts 20 1.7 Conclusions and future directions 27 Part I Background 31 2 Long-term astrophysics! processes 33 Fred C. Adams 2.1 Introduction: physical eschatology 33 2.2 Fate of the Earth 34 2.3 Isolation of the local group 36 2.4 Collision with Andromeda 36 2.5 The end of stellar evolution 38 2.6 The era of degenerate remnants 39 2.7 The era of black holes 41 2.8 The Dark Era and beyond 41 2.9 life and information processing 43 2.10 Conclusion 44 Suggestions for further reading 45 References 45 3 Evolution theory and the future of humanity 48 Christopher Wills· 3.1 Introduction 48 3.2 The causes of evolutionary change 49 xiv Contents 3.3 Environmental changes and evolutionary changes 50 3.3.1 Extreme evolutionary changes 51 3.3.2 Ongoing evolutionary changes 53 3.3.3 Changes in the cultural environment 56 3.4 Ongoing human evolution 61 3.4.1 Behavioural evolution 61 3.4.2 The future of genetic engineering 63 3.4.3 The evolution of other species, including those on which we depend 64 3.5 Future evolutionary directions 65 3.5.1 Drastic and rapid climate change without changes in human behaviour 66 3.5.2 Drastic but slower environmental change accompanied by changes in human behaviour 66 3.5.3 Colonization of new environments by our species 67 Suggestions for further reading 68 References 69 4 Millennial tendencies in responses to apocalyptic threats 73 James J.
    [Show full text]
  • Beneficial AI 2017
    Beneficial AI 2017 Participants & Attendees 1 Anthony Aguirre is a Professor of Physics at the University of California, Santa Cruz. He has worked on a wide variety of topics in theoretical cosmology and fundamental physics, including inflation, black holes, quantum theory, and information theory. He also has strong interest in science outreach, and has appeared in numerous science documentaries. He is a co-founder of the Future of Life Institute, the Foundational Questions Institute, and Metaculus (http://www.metaculus.com/). Sam Altman is president of Y Combinator and was the cofounder of Loopt, a location-based social networking app. He also co-founded OpenAI with Elon Musk. Sam has invested in over 1,000 companies. Dario Amodei is the co-author of the recent paper Concrete Problems in AI Safety, which outlines a pragmatic and empirical approach to making AI systems safe. Dario is currently a research scientist at OpenAI, and prior to that worked at Google and Baidu. Dario also helped to lead the project that developed Deep Speech 2, which was named one of 10 “Breakthrough Technologies of 2016” by MIT Technology Review. Dario holds a PhD in physics from Princeton University, where he was awarded the Hertz Foundation doctoral thesis prize. Amara Angelica is Research Director for Ray Kurzweil, responsible for books, charts, and special projects. Amara’s background is in aerospace engineering, in electronic warfare, electronic intelligence, human factors, and computer systems analysis areas. A co-founder and initial Academic Model/Curriculum Lead for Singularity University, she was formerly on the board of directors of the National Space Society, is a member of the Space Development Steering Committee, and is a professional member of the Institute of Electrical and Electronics Engineers (IEEE).
    [Show full text]
  • F.3. the NEW POLITICS of ARTIFICIAL INTELLIGENCE [Preliminary Notes]
    F.3. THE NEW POLITICS OF ARTIFICIAL INTELLIGENCE [preliminary notes] MAIN MEMO pp 3-14 I. Introduction II. The Infrastructure: 13 key AI organizations III. Timeline: 2005-present IV. Key Leadership V. Open Letters VI. Media Coverage VII. Interests and Strategies VIII. Books and Other Media IX. Public Opinion X. Funders and Funding of AI Advocacy XI. The AI Advocacy Movement and the Techno-Eugenics Movement XII. The Socio-Cultural-Psychological Dimension XIII. Push-Back on the Feasibility of AI+ Superintelligence XIV. Provisional Concluding Comments ATTACHMENTS pp 15-78 ADDENDA pp 79-85 APPENDICES [not included in this pdf] ENDNOTES pp 86-88 REFERENCES pp 89-92 Richard Hayes July 2018 DRAFT: NOT FOR CIRCULATION OR CITATION F.3-1 ATTACHMENTS A. Definitions, usage, brief history and comments. B. Capsule information on the 13 key AI organizations. C. Concerns raised by key sets of the 13 AI organizations. D. Current development of AI by the mainstream tech industry E. Op-Ed: Transcending Complacency on Superintelligent Machines - 19 Apr 2014. F. Agenda for the invitational “Beneficial AI” conference - San Juan, Puerto Rico, Jan 2-5, 2015. G. An Open Letter on Maximizing the Societal Benefits of AI – 11 Jan 2015. H. Partnership on Artificial Intelligence to Benefit People and Society (PAI) – roster of partners. I. Influential mainstream policy-oriented initiatives on AI: Stanford (2016); White House (2016); AI NOW (2017). J. Agenda for the “Beneficial AI 2017” conference, Asilomar, CA, Jan 2-8, 2017. K. Participants at the 2015 and 2017 AI strategy conferences in Puerto Rico and Asilomar. L. Notes on participants at the Asilomar “Beneficial AI 2017” meeting.
    [Show full text]
  • Philos 121: January 24, 2019 Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence” (Skip Pp
    Philos 121: January 24, 2019 Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence” (skip pp. 7–13) Jeremy Bentham, Introduction to the Principles of Morals and Legislation, Ch. I–II Suppose we tell AI: “Make paper clips.” Unfortunately, AI has become advanced enough to design ever more intelligent AI, which in turn designs ever more intelligent AI, and so on. AI then tries to turn the universe into a paper-clip making factory, diverting all resources from food production into paper-clip production. Getting hungry, we try to turn it off. AI interprets us and our attempts to turn it off as obstacles to making paper clips. So it eliminates the obstacles… So… what should we tell AI to do, if we have only one chance to get it right? This is very close to the basic question of moral philosophy: In the most basic and general terms, what should we, or any other intelligent agent, do? The road to utilitarianism: First premise (Hedonism): What is good for us as an end is pleasure, and what is bad for us as an end is pain. Good as a means vs. good as an end: • Good as a means: good because it helps to bring about something else that is good. • But this chain of means has to end somewhere. There has to be something that is: • Good as an end: good because of what it is in itself and not because of what it brings about. What is good for us as an end? The answer at the end of every chain seems to be: pleasure and the absence of pain.
    [Show full text]
  • Science Fiction & Fantasy Book Group List 2010 to 2020.Xlsx
    Science Fiction & Fantasy Book List 2010-2020 Date discussed Title Author Pub Date Genre Tuesday, August 17, 2010 Eyes of the Overworld Jack Vance 1966 Fantasy Tuesday, September 21, 2010 Boneshaker Cherie Priest 2009 Science Fiction/Steampunk Tuesday, October 19, 2010 Hood (King Raven #1) Steve Lawhead 2006 Fantasy/Historical Fiction Tuesday, November 16, 2010 Hyperion (Hyperion Cantos #1) Dan Simmons 1989 Science Fiction Tuesday, December 21, 2010 Swords and Deviltry (Fafhrd and the Gray Mouser #1) Fritz Leiber 1970 Fantasy/Sword and Sorcery Tuesday, January 18, 2011 Brave New World Aldous Huxley 1931 Science Fiction/Dystopia Tuesday, February 15, 2011 A Game of Thrones (A Song of Ice and Fire, Book 1) George R.R. Martin 1996 Fantasy Tuesday, March 15, 2011 Hull Zero Three Greg Bear 2010 Science Fiction Tuesday, April 19, 2011 The Lies of Locke Lamora (Gentleman Bastard, #1) Scott Lynch 2006 Fantasy Tuesday, May 17, 2011 Never Let Me Go Kazuo Ishiguro 2005 Science Fiction/Dystopia Tuesday, June 21, 2011 The Name of the Wind (The Kingkiller Chronicle #1) Patrick Rothfuss 2007 Fantasy Tuesday, July 19, 2011 Old Man's War (Old Man's War, #1) John Scalzi 2005 Science Fiction NO MEETING Tuesday, August 16, 2011 Wednesday, September 07, 2011 Something Wicked This Way Comes (Green Town #2) Ray Bradbury 1962 Fantasy Wednesday, October 05, 2011 Altered Carbon (Takeshi Kovacs #1) Richard Morgan 2002 Science Fiction Wednesday, November 02, 2011 Prospero's Children Jan Siegel 1999 Fantasy Wednesday, December 07, 2011 Replay Ken Grimwood 1986 Science Fiction/Time Travel Wednesday, January 04, 2012 Raising Stony Mayhall Daryl Gregory 2011 Fantasy/Horror/Zombies Wednesday, February 01, 2012 The Moon Is a Harsh Mistress Heinlein, Robert 1966 Science Fiction Wednesday, March 07, 2012 Talion: Revenant Michael A.
    [Show full text]