Autonomous Weapon Systems and Cyber Operations

The Weaponization of Increasingly Autonomous Technologies: Autonomous Weapon Systems and Cyber Operations 2 No. 7 UNIDIR RESOURCES Acknowledgements Support from UNIDIR’s core funders provides the foundation for all of the Institute’s activities. In addition, dedicated funding for the activities leading to this paper was received from the Governments of Canada, Germany, Ireland, the Netherlands and Switzerland. UNIDIR would like to acknowledge the thoughtful contributions of the participants in a November 2015 meeting on cyber, AI and increasingly autonomous technologies convened by UNIDIR: David Atkinson, John Borrie, Aude Fleurant, Adam Henschke, Sean Legassick, Patrick Lin, Ryder McKeown, Nils Melzer, Richard Moyes, Jean-Marc Rickli, Heather Roff, Eneken Tikk-Ringas, and Kerstin Vignard. Particular thanks are extended to Patrick Lin for his substantive input, meeting moderation, and synthesis. UNIDIR would also like to acknowledge the contributions of those experts and interviewees who have requested to remain unnamed. This report was drafted by Kerstin Vignard. About the Project “The Weaponization of Increasingly Autonomous Technologies” Given that governments have a responsibility to create or affirm sound policies about which uses of autonomy in weapon systems are legitimate—and that advances in relevant technologies are also creating pressure to do so—UNIDIR’s work in this area is focused on what is important for States to consider when establishing policy relating to the weaponization of increasingly autonomous technologies. See http://bit.ly/UNIDIR_Autonomy for Observation Papers, audio files from public events, and other materials. This is the seventh in a series of UNIDIR papers on the weaponization of increasingly autonomous technologies. UNIDIR has purposefully chosen to use the word “technologies” in order to encompass the broadest relevant categorization. In this paper, this categorization includes machines (inclusive of robots and weapons) and systems of machines (such as weapon systems), as well as the knowledge practices for designing, organizing and operating them. About UNIDIR The United Nations Institute for Disarmament Research—an autonomous institute within the United Nations—conducts research on disarmament and security. UNIDIR is based in Geneva, Switzerland, the centre for bilateral and multilateral disarmament and non-proliferation negotiations, and home of the Conference on Disarmament. The Institute explores current issues pertaining to the variety of existing and future armaments, as well as global diplomacy and local tensions and conflicts. Working with researchers, diplomats, government officials, NGOs and other institutions since 1980, UNIDIR acts as a bridge between the research community and governments. UNIDIR’s activities are funded by contributions from governments and foundations. Note The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of the Secretariat of the United Nations concerning the legal status of any country, territory, city or area, or of its authorities, or concerning the delimitation of its frontiers or boundaries. The views expressed in this publication are the sole responsibility of UNIDIR. They do not necessarily reflect the views or opinions of the United Nations or UNIDIR’s sponsors. www.unidir.org © UNIDIR 2017 Contents I. International discussions on autonomous weapon systems and cyber operations .......... 1 Box 1. Examples of increasing autonomy in cyber systems ................................................. 4 II. Overlap between the two domains .................................................................................... 5 Dependence on complex learning algorithms and artificial intelligence ............................ 5 Dominance of the private sector .......................................................................................... 5 Interest, capacity and experience of malicious actors ......................................................... 6 A challenge to traditional arms control approaches ............................................................ 6 Difficulties for testing and verification ................................................................................. 7 III. What sort of interplay is there between increasing autonomy in conventional systems and cyber operations? ......................................................................................................... 9 As a driver and as a countermeasure .................................................................................... 9 Unintended interactions and emergent behaviours ............................................................ 9 The vulnerability of increasingly autonomous weapon systems to cyber operations ....... 10 Exacerbating vulnerabilities already seen in conventional weapon systems ............. 11 Potential cyber vulnerabilities unique to—or particularly acute in—AWS ................. 12 IV. Conclusions ........................................................................................................................ 15 i Acronyms and abbreviations AEG Automatic Exploit Generation AI Artificial Intelligence AWS Autonomous Weapons Systems CCW Convention on Certain Conventional Weapons DARPA Defense Advanced Research Projects Agency DoD Department of Defense (United States) DoT&E Department of Defense’s Operational Test & Evaluation Directorate (United States) EMP Electromagnetic Pulse GGE Group of Governmental Experts ICRC International Committee of the Red Cross ICT Information and Communication Technology IHL International Humanitarian Law LAWS Lethal Autonomous Weapons Systems R&D Research and Development ii I. International discussions on autonomous weapon systems and cyber operations The international discussions on autonomous weapons systems (AWS) focus on conventional weapon systems. Other technologies have not been widely present in the discussion thus far.1 However, military interest in autonomy is not limited to purely conventional systems—autonomy is relevant for intangible cyber operations2 as well. Autonomy-enhancing technological innovations in both physical and digital systems are advancing at a rapid pace. Despite the clear relevance of autonomy for both areas, the international discussions on these issues are held in different multilateral forums with virtually no overlap between the participating experts and policy practitioners. While both subjects are being discussed in formats known as Groups of Governmental Experts (GGEs), the modalities and mode of operation of these groups are completely different. Starting in 2014, lethal autonomous weapon systems (LAWS) have been taken up as an issue in an arms control framework by the High Contracting Parties of the Convention on Certain Conventional Weapons (CCW). Since then, annual week-long “informal meetings of experts” have discussed and debated various concerns, positions and potential policy responses. In late 2016, the CCW established a formal GGE, which was set to meet in 2017 with a mandate to “explore and agree on possible recommendations on options related to emerging technologies in the area of LAWS”, and consider “identification of characteristics and elaboration of a working definition of LAWS”.3 They also noted that further consideration should be given to the “risks posed by cyber operations in relation to LAWS”. Within the CCW framework, International Humanitarian Law (IHL) has been at the forefront of the discussions—not only concerning the legality of LAWS, but also on issues of responsibility and accountability for the use of these weapons (with weapon reviews in the context of Article 36 of the Additional Protocol I to the Geneva Conventions receiving particular attention).4 There has also been a strong emphasis on emerging norms. Some have highlighted the necessity of human control or judgment when considering the development, deployment and use of LAWS. In this regard, the 1 Some governments have weighed the commonality between autonomous weapon systems and other autonomous technologies. For instance, the United States Department of Defense Directive 3000.09 on “Autonomy in Weapons Systems” specifically stipulates that the Directive does not apply to “autonomous or semi-autonomous cyber systems for cyberspace operations”. This implies an acknowledgement that, intuitively at least, these varying systems with greater degrees of autonomy may be considered together. According to people familiar with the development of the Directive, it was a pragmatic decision to exclude cyber, not a principled one about classification or category boundaries. Accounting for the special issues that arise in cyber would have delayed that directive, which was already urgently needed to clarify policy on emerging robotic systems. United States Department of Defense, “Autonomy in Weapons Systems”, Department of Defense Directive 3000.09, section 2.b, 21 November 2012. Available from http://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodd/300009p.pdf. 2 As there is not yet standard terminology in this field, this paper uses the term “cyber operations” rather than “cyber weapons”, “cyber arms”, “cyber bombs” or other descriptions. 3 Draft recommendations by the Informal Meetings of Experts are available from http://www.reachingcriticalwill.org/images/documents/Disarmament-fora/ccw/2016/meeting-experts- laws/documents/DraftRecommendations_15April_final.pdf; the Final Document

Load more