This electronic thesis or dissertation has been downloaded from the King’s Research Portal at https://kclpure.kcl.ac.uk/portal/ Autonomous weapons and stability Scharre, Paul Awarding institution: King's College London The copyright of this thesis rests with the author and no quotation from it or information derived from it may be published without proper acknowledgement. END USER LICENCE AGREEMENT Unless another licence is stated on the immediately following page this work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International licence. https://creativecommons.org/licenses/by-nc-nd/4.0/ You are free to copy, distribute and transmit the work Under the following conditions: Attribution: You must attribute the work in the manner specified by the author (but not in any way that suggests that they endorse you or your use of the work). Non Commercial: You may not use this work for commercial purposes. No Derivative Works - You may not alter, transform, or build upon this work. Any of these conditions can be waived if you receive permission from the author. Your fair dealings and other rights are in no way affected by the above. Take down policy If you believe that this document breaches copyright please contact [email protected] providing details, and we will remove access to the work immediately and investigate your claim. Download date: 01. Oct. 2021 Autonomous Weapons and Stability Paul Scharre March 2020 Thesis Submitted in Fulfillment of the Requirements for the Degree of Doctor of Philosophy in the Department of War Studies Faculty of Social Science & Public Policy King’s College London 1 Copyright © 2019 by Paul Scharre The copyright of this thesis rests with the author and no quotation from it or information derived from it may be published without proper acknowledgement. 2 ABSTRACT Autonomous Weapons and Stability Paul Scharre Militaries around the globe are developing increasingly autonomous robotic weapons, raising the prospect of future fully autonomous weapons that could search for, select, and engage targets on their own. Autonomous weapons have been used in narrow circumstances to-date, such as defending ships and vehicles under human supervision. By examining over 50 case studies of autonomous military and cyber systems in use or currently in development around the globe, this thesis concludes that widespread use of fully autonomous weapons would be a novel development in war. Autonomous weapons raise important considerations for crisis stability, escalation control, and war termination that have largely been underexplored to-date. Fully autonomous weapons could affect stability in a number of ways. Autonomous weapons could increase stability by replacing human decision-making with automation, linking political leaders more directly to tactical actions. Accidents with autonomous weapons could lead to the opposite, however, causing unintended lethal engagements that escalate a crisis or make it more difficult to terminate conflict. Nine total mechanisms are explored by which fully autonomous weapons could affect stability, four of which would enhance stability and five of which would undermine stability. The significance of these mechanisms is assessed, and the most prominent factor is the extent to which autonomous weapons increase or decrease human control. If autonomous weapons were likely to operate consistent with human intent in realistic military operational environments, then widespread autonomous weapons would improve crisis stability, escalation control, and war termination by increasing political leaders’ control over their military forces and reducing the risk of accidents or unauthorized escalatory actions. If, however, unanticipated action by autonomous weapons is likely in realistic environments, then their use would undermine crisis stability, escalation control, and war termination because of the potential for unintended lethal engagements that could escalate conflicts or make it more difficult to end wars. The competing frameworks of normal accident theory and high-reliability organizations make different predictions about the likelihood of accidents with complex systems. 3 Over 25 case studies of military and non-military complex systems in real-world environments are used to test these competing theories. These include extensive U.S. experience with the Navy Aegis and Army Patriot air and missile defense systems and their associated safety track records. Experience with military and civilian autonomous systems suggest that three key conditions exist that will undermine reliability for fully autonomous weapons in wartime: (1) the absence of routine day-to-day experience under realistic conditions, since testing and peacetime environments will not perfectly replicate wartime conditions; (2) the presence of adversarial actors; and (3) the lack of human judgment to flexibly respond to novel situations. The thesis concludes that normal accident theory best describes the situation of fully autonomous weapons and that militaries are highly unlikely to be able to achieve high-reliability operations with fully autonomous weapons. Unanticipated lethal actions are likely to be normal consequences of using fully autonomous weapons and militaries can reduce but not entirely eliminate these risks. Recent and potential future advances in artificial intelligence and machine learning are likely to exacerbate these risks, even as they enable more capable systems. This stems from the challenge of accurately predicting the behavior of machine learning systems in complex environments, their poor performance in response to novel situations, and their vulnerability to various forms of spoofing and manipulation. The widespread deployment of fully autonomous weapons is therefore likely to undermine stability because of the risk of unintended lethal engagements. Four policy and regulatory options are explored to mitigate this risk. Their likelihood of success is assessed based on analysis of over 40 historical cases of attempts to control weapons and the specific features of autonomous weapons. The thesis concludes with three potentially viable regulatory approaches to mitigate risks from fully autonomous weapons. 4 TABLE OF CONTENTS Abstract ............................................................................................................................ 3 List of Tables, Images, and Illustrations ....................................................................... 7 Overview and Key Research Questions ........................................................................ 8 Introduction .................................................................................................................. 8 Topic Overview .......................................................................................................... 14 Background and Existing Literature ........................................................................... 17 Research Questions .................................................................................................... 41 Methodology ............................................................................................................... 43 Outline of the Report and Preview of Conclusions .................................................... 55 Part I: Autonomy and Autonomous Weapons ........................................................... 57 The AI and Robotics Revolution ................................................................................ 57 The Nature of Autonomy ............................................................................................ 66 A History of Automation and Autonomy in Weapons ............................................... 71 Part II: Autonomy in Weapons Today and Future Trends ...................................... 91 Autonomy in Robotic Platforms ................................................................................. 91 Autonomy in Missiles and Sensors .......................................................................... 101 Commercially-Available Components for Autonomous Weapons .......................... 112 Cyberspace and Autonomous Weapons ................................................................... 113 Part III: Stability and Autonomous Weapons ......................................................... 126 The Concept of Stability ........................................................................................... 127 Autonomous Weapons and Stability ........................................................................ 132 Part IV: Controlling Autonomous Weapons: Risk, Predictability, and Hazard .. 156 Understanding Accident Risk: Normal Accident Theory vs. High-Reliability Organizations ............................................................................................................ 156 Case Studies of Non-Military Autonomous Systems in Competitive and Uncontrolled Environments ............................................................................................................ 165 Case Studies in Complex Automated Military Weapon Systems: The Patriot and Aegis ......................................................................................................................... 180 Automation as a Source of Both Reliability and Failure .......................................... 200 5 Nuclear Weapons as a Case Study in Risk Management ......................................... 213 Summary: The Inevitability of