C O R P O R A T I O N YUNA HUH WONG, JOHN M. YURCHAK, ROBERT W. BUTTON, AARON FRANK, BURGESS LAIRD, OSONDE A. OSOBA, RANDALL STEEB, BENJAMIN N. HARRIS, SEBASTIAN JOON BAE Deterrence in the Age of Thinking Machines For more information on this publication, visit www.rand.org/t/RR2797 Library of Congress Cataloging-in-Publication Data is available for this publication. ISBN: 978-1-9774-0406-0 Published by the RAND Corporation, Santa Monica, Calif. © Copyright 2020 RAND Corporation R® is a registered trademark. Cover: Getty Images Plus. Limited Print and Electronic Distribution Rights This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited. Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial use. For information on reprint and linking permissions, please visit www.rand.org/pubs/permissions. The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. Support RAND Make a tax-deductible charitable contribution at www.rand.org/giving/contribute www.rand.org Preface We expect artificial intelligence (AI) and autonomous systems to sig- nificantly change the future battlefield. Militaries around the world are incorporating AI and autonomy into their organizational processes, command and control systems, logistics systems, and, of course, weapon systems themselves—with an aim toward leveraging current develop- ments from the commercial world. As AI and autonomy proliferate on the battlefield, an important question arises: How might deterrence be affected by the proliferation of AI and autonomous systems? Up to this point, deterrence has primarily involved humans attempting to affect the decision calculus and perceptions of other humans. But what hap- pens when decisionmaking processes are no longer fully under the con- trol of humans? In this report, we lay out some initial considerations and present ideas for how deterrence could change in the age of AI and autonomy. RAND Ventures The RAND Corporation is a research organization that develops solu- tions to public policy challenges to help make communities through- out the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest. RAND Ventures is a vehicle for investing in policy solutions. Philanthropic contributions support our ability to take the long view, tackle tough and often-controversial topics, and share our findings in innovative and compelling ways. RAND’s research findings and iii iv Deterrence in the Age of Thinking Machines recommendations are based on data and evidence, and therefore do not necessarily reflect the policy preferences or interests of its clients, donors, or supporters. Funding for this venture was made possible by the independent research and development provisions of RAND’s contracts for the operation of its U.S. Department of Defense federally funded research and development centers. International Security and Defense Policy Center This research was conducted within the International Security and Defense Policy Center of the RAND National Defense Research Insti- tute, a federally funded research and development center sponsored by the Office of the Secretary of Defense, the Joint Staff, the Unified Combatant Commands, the defense agencies, and the defense Intel- ligence Community. For more information on the International Security and Defense Policy Center, see www.rand.org/nsrd/ndri/centers/isdp or contact the director (contact information is provided on the webpage). Contents Preface ............................................................................. iii Figures and Tables ...............................................................vii Summary .......................................................................... ix Acknowledgments ...............................................................xv Abbreviations ................................................................... xvii CHAPTER ONE Introduction ...................................................................... 1 Background ......................................................................... 2 Why Is This Topic Important? .................................................. 5 Approach ............................................................................ 7 How to Read This Report ........................................................ 8 CHAPTER TWO Key Deterrence Concepts ......................................................11 Deterrence .........................................................................11 Escalation ..........................................................................13 Stability and Instability ...........................................................15 Security Dilemma .................................................................15 CHAPTER THREE Artificial Intelligence and Autonomous Systems ..........................17 Artificial Intelligence .............................................................17 Autonomy ......................................................................... 22 CHAPTER FOUR Potential Futures in a World of Proliferated Artificial Intelligence and Autonomous Systems ................................................ 27 Key Factors Affecting Deterrence in the Age of Thinking Machines .... 27 Creating One Future World .................................................... 36 v vi Deterrence in the Age of Thinking Machines CHAPTER FIVE A Wargame of Artificial Intelligence and Autonomous Systems .......39 Wargame Overview ...............................................................39 Wargame Events .................................................................. 43 Wargame Limitations ............................................................ 48 CHAPTER SIX Wargame Insights and Debates ...............................................51 Wargame Insights .................................................................51 Debates During the Game ......................................................53 CHAPTER SEVEN Implications for Deterrence....................................................59 Deterrence Concepts Revisited ..................................................59 Comparisons with Nuclear Weapon Proliferation ............................61 How Escalatory Dynamics May Change ......................................63 CHAPTER EIGHT Implications for Decisionmaking ............................................69 Inadvertent Engagement .........................................................69 Trust and Autonomy ..............................................................74 Time Scale and Decisionmaking ................................................75 Considerations When Building Autonomous Forces ........................ 80 CHAPTER NINE Conclusion and Areas for Further Research ................................83 Conclusion .........................................................................83 Areas for Further Research .......................................................83 APPENDIX General Morphological Analysis ............................................. 87 References ........................................................................93 Figures and Tables Figures 7.1. Understanding Required in Traditional Deterrence .............67 7.2. Understanding Required in Deterrence with AI and Autonomous Systems ................................................67 8.1. Boyd’s Observe, Orient, Decide, Act (OODA) Loop ...........76 8.2. Reaction Speeds and Type of Autonomy ..........................78 Tables 4.1. One Future World ...................................................37 5.1. Relative Distribution of Military Assets ......................... 42 7.1. Nuclear Weapon Proliferation Versus Autonomous System Proliferation ..........................................................62 7.2. Human and Machine Configurations and Potential Escalatory Dynamics ............................................... 64 7.3. Potential Machine Misinterpretation of Human Signaling .... 66 8.1. Inadvertent Engagements by Autonomous Systems .............72 8.2. Potential Advantages and Disadvantages of Military Autonomous Systems ................................................73 A.1. Morphological Field .................................................89 A.2. Cross-Consistency Analysis ........................................ 90 A.3. One Future World .................................................. 92 vii Summary Artificial intelligence (AI) and autonomous systems have the potential to change the future of warfare. The increased use of unmanned sys- tems on the battlefield, breakthroughs in commercial AI, and interest that many countries now have in AI and autonomous systems for mili- tary applications make it likely that such systems will be widely used in future conflicts. Yet what happens to deterrence and escalation when decisions can be made at machine speeds and when states can