ACADEMY BRIEFING N°9 DEFENDING THE BOUNDARY CONSTRAINTS AND REQUIREMENTS ON THE USE OF AUTONOMOUS WEAPON SYSTEMS UNDER INTERNATIONAL HUMANITARIAN AND HUMAN RIGHTS LAW MAYA BREHM - MAY 2017 THE ACADEMY A JOINT CENTER OF DEFENDING THE BOUNDARY GEMENTS 3 GEMENTS D LE CONSTRAINTS AND REQUIREMENTS ON THE USE W OF AUTONOMOUS WEAPON SYSTEMS UNDER INTERNATIONAL CKNO A HUMANITARIAN AND HUMAN RIGHTS LAW ACKNOWLEDGEMENTS This Academy Briefing was researched and written by Maya Brehm, Researcher at the Geneva Academy of International Humanitarian Law and Human Rights. With thanks to Neil Davison, Merel Ekelhof, Sandra Krähenmann, Richard Moyes, Michael Siegrist and Ozlem Ulgen who provided helpful comments on an earlier draft, and to Munizha Ahmad-Cooke for her meticulous copy-editing. The Geneva Academy would like to thank the Swiss Federal Department of For- eign Affairs (FDFA) for its support to the Geneva Academy’s research on this issue. DISCLAIMER This Briefing is the work of the author. The views expressed in it do not necessar- ily reflect those of the project’s supporters or of anyone who provided input to, or commented on, drafts. The designation of states or territories does not imply any judgement by the Geneva Academy or the FDFA regarding the legal status of such states or territories, or their authorities and institutions, or the delimitation of their boundaries, or the status of any states or territories that border them. 5 4 CONTENTS ARY ARY D D OUN OUN B B THE THE KEY MESSAGES 6 ING ING D D EFEN EFEN D D 1. INTRODUCTION 7 2. AbOUT THIS STUDY 11 3. ENVISIONING AUTONOMOUS WEAPON SYSTEMS 13 A. HUMAN CONTROL AND THE USE OF FORCE 19 4. THE APPLICABLE LAW: IHL AND IHRL STANDARDS ON THE USE OF FORCE 23 A. HUMAN RIGHTS TREATY OBLIGATIONS ABROAD: AWS AND Extra-TERRITORIAL CONTROL 26 B. ANIMUS BELLIGERENDI: AWS AND THE INTENT TO WAGE WAR 30 C. THE BELLIGERENT NEXUS: AWS, CONTROL AND THE INTENT TO CONDUCT HOSTILITIES 33 D. PRELIMINARY FINDINGS ON THE APPLICABLE LAw 39 5. HUMAN RIGHTS REQUIREMENTS AND CONSTRAINTS ON THE USE OF AWS 42 A. AUTOMATED KILL ZONES: PREPARING THE GROUND FOR SENTRY-AWS? 42 B. THE DUTY TO INDIVIDUATE THE USE OF FORCE UNDER IHRL 45 C. THE SCOPE FOR CATEGORICAL KILLING UNDER IHL 49 D. ‘Non-LETHAL’ AUTONOMOUS INTERCEPTION 52 E. ALGORITHMIC TARGET CONSTRUCTION: A THREAT TO HUMAN RIGHTS AND HUMAN DIGNITY 56 1. SURVEILLANCE 57 2. SORTING PEOPLE 59 3. CALCULATED BLINDNESS 63 4. PROCESS MATTERS 65 6. CONCLUDING REMARKS 68 7 6 KEY MESSAGES 1. INTRODUCTION ARY ARY D D OUN OUN • Autonomous weapon systems (AWS) tend to be portrayed as ‘weapons of war’, ‘[W]hen the lethal decision is purely automatic, the only human agent directly identifiable B B 1 but international humanitarian law (IHL) would never be the sole, and in many in- as the efficient cause of death would turn out to be the victim …’ (G. Chamayou) THE THE stances, it would not be the primary legal frame of reference to assess the legality ING ING ‘Law depends on violence and uses it as a counterpunch to the allegedly more lethal and D D of their use. Consideration of international human rights law (IHRL) requirements destructive violence situated just beyond law’s boundaries. But the violence on which law 2 EFEN EFEN and constraints on the use of AWS must be a part of the debate on AWS, including in depends always threatens the values for which law stands.’ (A. Sarat) D D the framework of the 1980 Convention on Certain Conventional Weapons (CCW). Over recent years, there has been growing debate about the ethical, humanitarian, legal and security implications of autonomous weapon systems (AWS). The basic idea is that once activated, such weapon systems would detect, select and attack • Where IHL permits the ‘categorical’ targeting of security measures, including the targets without further human intervention. According to leading researchers in use of force, there is scope for the lawful use of an AWS. However, due to pro- the field of artificial intelligence (AI) and robotics, AI technology has ‘reached a cedural requirements and the need to individuate the use of force, this scope is point where the deployment of such systems is — practically if not legally — feasi- extremely limited under IHRL. IHRL requirements and constraints apply to the ble within years’.3 AWS are said to have the potential to revolutionize warfare (and use of an AWS in an armed conflict in so far as they are not displaced by IHL. policing, although that argument is seldom made). Whilst success in the quest for AI may bring unprecedented benefits to humanity, it is also argued to pose an ex- istential threat to humankind.4 • To safeguard human dignity and human rights, human agents must: A small number of states are actively engaged in research and development with the stated goal of increasing autonomy in weapon systems. Regarding the drivers — exercise the control necessary to determine, in a timely manner, what for this trend, commentators cite a perceived need to react to threats more quickly, legal rules govern applications of force by means of an AWS, and adapt process growing data much more efficiently (speeding up the targeting–decision operations as required cycle), improve performance in communications-denied environments, increase — remain involved in algorithmic targeting processes in a manner that en- persistence and endurance, and reduce the exposure of states’ own security forces 5 ables them to explain the reasoning underlying algorithmic decisions in to physical harm. concrete circumstances — be continuously and actively (personally) engaged in every instance of force application outside of the conduct of hostilities 1 G. Chamayou, Drone Theory, Penguin Books, 2015, p 211. 2 A. Sarat, ‘Situating Law Between the Realities of Violence and the Claims of Justice: An Introduction’, in — exercise active and constant (continuous or at least frequent, periodic) A. Sarat (ed), Law, Violence, and the Possibility of Justice, Princeton University Press, 2001, p 3. human control over every individual attack in the conduct of hostilities; 3 ‘Autonomous Weapons: An Open Letter from AI & Robotics Researchers’, Future of Life Institute, 28 they must appropriately bound every attack in spatio-temporal terms July 2015, http://futureoflife.org/open-letter-autonomous-weapons/. so as to enable them to recognize changing circumstances and adjust 4 S. Russell, D. Dewey and M. Tegmark, ‘Research Priorities for Robust and Beneficial Artificial Intelligence’, operations in a timely manner AI Magazine (2015) 105–114, http://futureoflife.org/data/documents/research_priorities.pdf. 5 ‘The changing character of war, fleeting nature of targets, and glut of big data requires the mili- tary to integrate machine learning into its targeting process to win wars’ (C. Lewis, ‘Capturing Flying Insects: A Machine Learning Approach to Targeting’, War on the Rocks, 6 Sep 2016, http://waronthe rocks.com/2016/09/capturing-flying-insects-a-machine-learning-approach-to-targeting/). Others have cautioned that increasing automation exacerbates, rather than addresses challenges linked to speed and data load, and have pointed out that the protection of a state’s own forces can be enhanced without au- tonomy in critical functions. See, e.g., United Nations Institute for Disarmament Research (UNIDIR), Framing Discussions on the Weaponization of Increasingly Autonomous Technologies, 2014, pp 5–6, http://www. unidir.ch/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autono mous-technologies-en-606.pdf. For a critical appraisal of the claim that AWS would help reduce cost, see R. R. Hoffmann, T. M. Cullen and J. K. Hawley, ‘The Myths and Costs of Autonomous Weapon Systems’, 72 Bulletin of the Atomic Scientists 4 (2016) 247–255, http://tandfonline.com/doi/ abs/10.1080/00963402.2016.1194619?journalCode=rbul20. The use of AWS can be expected to change the manner in which and the processes with international law and, in times of armed conflict, with IHL’.10 States Parties 9 8 by which human beings exercise control over the use of force and its consequenc- also agree that 1977 Additional Protocol I (AP I) to the Geneva Conventions impos- ARY es. Out of concern over serious negative ethical, humanitarian, legal and security es a legal obligation to determine whether the use of an AWS as a ‘new weapon, D UCTION implications, policy makers and commentators have emphasized that human be- means or method of warfare’ would in some or all circumstances be prohibited D OUN 6 11 B ings must retain ‘meaningful’, ‘appropriate’ or ‘effective’ control over weapons. under international law. INTRO THE What that involves, concretely, remains to be clarified. ING Views diverge, however, on the circumstances in which it would be legal to use an D From a legal perspective, the requirement for meaningful human control over AWS. The former UN Special Rapporteur on extrajudicial killings, Philip Alston, EFEN D AWS would seem to entail that human agents involved in the use of an AWS have among others, noted the difficulty that military personnel face in present practice the opportunity and capacity to assess compliance with applicable legal norms to distinguish between those who may be lawfully targeted and those who may not and to take all legally required steps to respect and ensure respect for the law, – ‘decision-making [that] requires the exercise of judgement, sometimes in rapidly including preventive and remedial measures. In what circumstances this is no changing circumstances and in a context which is