Defining the Scope of Autonomy

PRIO POLICY BRIEF 02 2014

VisitingAddress: BoxPO 9229Grønland, NO PeaceResearch Institute Oslo(PRIO) Defining the Scope of Hausmanns gate 7 Autonomy

- 0134 Oslo,Norway Issues for the Campaign to Stop Killer Robots

Twitter:PRIOUpdates Facebook: PRIO.org www.prio.org The Campaign to Stop Killer Robots was launched in April 2013

with the objective of achieving a ban on the development, produc-

tion and deployment of lethal autonomous weapons. In May 2014, the issue will be discussed by a UN expert meeting under the aus- pices of the Convention on Conventional Weapons (CCW) in Ge- neva. At this stage, it is inevitable that there will be much debate and discussion over the scope and meaning of any future prohibition. The Campaign is still being shaped, and what will be neces- ISBN:978 sary for its success is that over the next few years a group of states

978

82 and governments coalesce around a shared understanding of the

7288 7288 problem and its solutions.

541

540

- - 1

9 (online) 9 2 (print) 2 This policy brief provides a pragmatic theory of humanitarian dis-

armament and discusses a set of key challenges related to the ques-

tion of scope – that is, what exactly would be covered by a possible international agreement on lethal autonomous robots. The Cam- paign must tackle these issues if it is to move from a startup phase and into activism with sustained political impact. Attention is given to the Brimstone, a UK anti-tank weapon and one of the most advanced ‘fire and forget’ missiles in use, whose degree of autonomy is also a matter of some contention. Nicholas Marsh Peace Research Institute Oslo (PRIO)

A Pragmatic Theory of Successful Hu- tial). lethal fully autonomous robot is the decisionmaking capability of the machine – that it manitarian Disarmament If it is not possible to persuade states of the is able to navigate around its environment and necessity of a prohibition, the Campaign will A humanitarian disarmament treaty is de- discriminate between targets. most likely not find support from the states signed to reduce human suffering rather than that will be required to form a coalition and The above definitions used by the Campaign, to manage the affairs of great powers. In the negotiate a successful treaty. Noel Sharkey and Christof Heyns coalesce experience of this author, two things are re- around a definition of machine which is capa- quired for a successful campaign for a hu- Defining the Threshold of Autonomy ble of lethal force, and can autonomously manitarian disarmament treaty. move around the battlefield and select a tar- One of the most contentious issues is likely to  First, the campaign needs to stigmatize the get. However, this is not a universally shared concern the threshold at which a weapon type of weapon concerned by showing that it understanding of the essential characteristics system is deemed to be ‘fully autonomous’. causes an unacceptable level of harm. Doing of a lethal autonomous weapon. The UK Discussions on this issue will necessarily be so involves highlighting aspects such as the Ministry of Defence defines an ‘autonomous both technical and intensely pragmatic. The propensity of the weapon to cause death and system’ in the following manner: threshold that is set would determine which injury among civilians. A key element of the systems are banned and which are allowed to An autonomous system is capable of understand- campaigns to ban anti-personnel land mines continue in operation. ing higher level intent and direction. From this or cluster munitions was the message that understanding and its perception of its environ- these weapons remain lethal long after the Setting the threshold of autonomy is going to ment, such a system is able to take appropriate fighting has ended, and so pose a deadly risk involve significant debate because machine action to bring about a desired state.... Autono- to civilians. decisionmaking exists on a continuum. There mous systems will, in effect, be self-aware and is a grey area in which systems can be said to  The second task is to persuade govern- their response to inputs indistinguishable from, or be partly autonomous, and there is no consen- ments that they do not need the weapon in even superior to, that of a manned aircraft. As sus as to exactly what counts as a fully auton- question. States are being asked to voluntarily such, they must be capable of achieving the same omous machine. A key task for the Campaign give up the use of a weapon, and usually level of situational understanding as a human.7 will be to create consensus on this issue prohibitive treaties also cover development, among both NGOs and the states that would Sharkey criticizes the UK Ministry of Defence production, stockpiling and trade. For a gov- have to negotiate and then implement a ban. definition8 on the grounds that not only does ernment to abjure a weapon it needs to first the technology to enable human-like cogni- accept that the weapon is not militarily essen- Scholar Noel Sharkey has defined a lethal tion not exist, but it is very unlikely to exist in tial. fully autonomous robot as one that operates the foreseeable future. The point for Sharkey in an open and unstructured environment; A key element of the campaigns to ban cluster is that full autonomy occurs when machines receives information from sensors; and pro- munitions and landmines involved spreading have sole responsibility for selecting targets cesses the information in order to move, the message that these weapons were not and using weapons – which could happen in select targets and fire – all without human essential – that alternatives existed, the role machines with decisionmaking capabilities far supervision.3 The Human Rights Watch re- envisioned for them no longer existed, or that below levels of human cognition. Using the port Losing Humanity: The Case Against their use is so abhorrent as to be counterpro- above definition, the UK Ministry of Defence Killer Robots similarly defines a fully auton- ductive. A counter example is the case of could employ machines with the ability to omous robot as being one that is ‘capable of nuclear weapons. No sensible person doubts move, locate and select targets, and then fire selecting targets and delivering force without their devastating effects upon civilians. None- at them, without defining them as ‘autono- any human input or interaction’.4 Later, the theless, nuclear weapons remain in state mous’ if they lacked a particular aspect of report refers to a fully autonomous weapon arsenals because the nuclear states still view human cognition. The next section examines that would identify targets and trigger itself, them as being necessary. Nuclear disarma- how these definitional disputes apply to spe- and also to Sharkey’s definition noted above. 5 ment will therefore not occur until the gov- cific weapon systems. ernments of those states are persuaded that Likewise, Christof Heyns, the UN Special such weapons cannot be used under any Rapporteur on extrajudicial, summary or The Case of ‘Fire and Forget’ Weap- circumstances; that they are considered mili- arbitrary executions, considered the technolo- ons: The Brimstone tarily useless.2 gy and defined lethal autonomous robots as This section highlights the ambiguous status being ‘weapon systems that, once activated, The Campaign to Stop Killer Robots needs to of a particular weapon, the Brimstone anti- can select and engage targets without further strike a balance between a prohibition with a tank missile, one of a class of ‘fire and forget’ intervention by a human operator. The im- wide ranging scope and a pragmatic accom- weapons that includes missiles with autono- portant element is that the robot has an au- modation of the interests of likely State sup- mous capabilities. tonomous “choice” regarding selection of a porters of the Campaign (who need to be target and the use of lethal force’.6 The Brimstone is a ‘fire and forget’ weapon reassured that they won’t have to give up system – one of a range of weapons that are something they perceive to be militarily essen- The emphasis in all these definitions of a designed to be used precisely as that term

PRIO POLICY BRIEF 02 2014 www.prio.org

suggests. Most are missiles that guide them- its ability to autonomously select targets was relevant to the current debate prompted by the selves to their target once launched, allowing ill-suited to contemporary operations – espe- Campaign to Stop Killer Robots. Five areas of the firer to concentrate on other tasks, such as cially Afghanistan. There, because of the particular concern are presented below: escaping enemy retaliation. Some simply fly conflict’s complex nature, rules of engage-  First, at the technical level, is the detailed to preset coordinates, while others actively ment required that a human monitor the consideration of exactly what counts as lethal look for targets. engagement right up until impact of the mis- and fully autonomous. The status of ‘fire and sile. A ‘fire and forget’ missile was inappro- The Brimstone, a UK-built anti-tank weapon, forget’ weapons has been included in media priate in such an environment.12 In response, is currently one of the most advanced ‘fire and articles and commentary on the issue of au- the UK Ministry of Defence commissioned an forget’ missiles. Development started in the tonomous lethal weapons.16 This policy brief urgent update so that the missile could also be 1990s, and the system has been in service argues that the Brimstone and other weapons guided with a laser seeker that would guide since 2005. A description of it provided by the cross the definitional threshold provided by the missile to a target directly selected by a UK’s Royal Air Force states that: the Campaign and other commentators such human operator. The ability of the Brimstone as Noel Sharkey and Christof Heyns. Brimstone is a fully autonomous, fire-and-forget, to autonomously select targets with its radar, anti-armour weapon, effective against all known however, has been used. For example, the first  Second, a narrow definition of autonomy – and projected armoured threats.... During the use of the autonomous radar seeker was re- such as the one used by the UK Ministry of search phase of the engagement, Brimstone’s ported on 16 September 2011: over Libya, a Defence, which specifies ‘human like capabili- [radar] seeker searches for targets in its path, salvo of 24 missiles using radar guidance was ties’ – would exclude not only all current comparing them to a known target signature in used to destroy seven or eight vehicles travel- technology, but likely all machine deci- its memory. The missile automatically rejects ling in a convoy.13 sionmaking in existence during the foreseea- returns which do not match (such as cars, buses, ble future. One potential hurdle for the Cam- The Brimstone is not unique. There are tens buildings) and continues searching and compar- paign is likely to be states that support the of different types of ‘fire and forget’ missiles ing until it identifies a valid target. The missiles idea of a ban, but set the threshold for auton- in existence. Many just have the capability to can be programmed not to search for targets until omy so high that it will not affect any of the fly to preset coordinates and thus do not select they reach a given point, allowing them to safely robot systems they wish to deploy. targets without human intervention. Never- overfly friendly forces, or only to accept targets in theless, the Brimstone is not the only missile  Third, a governmental preference for an a designated box area, thus avoiding collateral to have an autonomous target-selection capa- excessively high threshold of autonomy raises damage.9 bility. In particular, some missiles used to a paradox. A key motivation for the Campaign

attack shipping actively search and identify is that robots cannot make the ethical and These capabilities appear to place the Brim- targets. The missiles frequently have ranges of contextual assessments that humans can. stone within the above-mentioned definition over 100 kilometres, and thus enemy ships However, a robot with a decisionmaking of an autonomous lethal robot as defined by will have moved during the period between capability as advanced as human cognition the Campaign to Stop Killer Robots, as well as launch and the missile reaching the target could presumably make such judgements. by Noel Sharkey and Christof Heyns. It would area. not fall squarely within the UK Ministry of  Fourth, it is one thing for the Campaign to Defence’s definition, however, as while the There is not sufficient space in this brief to attempt to preemptively stop the development Brimstone can distinguish between potential examine in detail the exact nature of the tech- of lethal autonomous weapons that do not targets, it is unable to make human-like as- nology. However, I will highlight two claims currently exist: it is hard to argue that a weap- sessments of intent. The UK Ministry of made by the manufacturers of such weapons on is military essential when it has not yet Defence position document on autonomous that indicate the development of autonomous even been developed. It will be much more systems, however, does mention that the target selection. The Norwegian Joint Strike difficult, though not impossible, to get states Brimstone is an example of a weapon that Missile is advertised by its manufacturer the to give up arms that are already deployed. At ‘probably’ achieves a degree of autonomous Kongsberg Group as having ‘Autonomous the very least, if the Brimstone and other ‘fire operation in certain environments.10 The Target Recognition with identification of and forget’ missiles are assumed to fall under specific question of whether the Brimstone is targets to ship class level’ and also employs a the definition used by the Campaign, then a fully autonomous weapon is also considered library of potential targets.14 Similarly, the considerable effort may need to be expended briefly in an article by the UK Royal Aeronau- company MBDA advertises its MM40 BLOCK on explaining why they are not to be consid- tical Society’s Air Power Group. This reiter- 3 Exocet missile by stating ‘terminal guidance ered militarily essential. For pragmatic rea- ates that the Brimstone does not meet the UK relies on a sophisticated J-band active seeker sons, it might be possible to negotiate that the armed forces definition of an autonomous to discriminate and select targets at sea’.15 scope of an international ban should exclude system, but does not comment on the wider certain systems even if they cross the thresh- debate.11 Issues for Campaigners and Policy- old of full autonomy. But, even so, creating makers such a loophole would be a difficult decision. The operational use of the Brimstone actually highlights the very issues raised by the Cam- The Brimstone and other ‘fire and forget’ Finally, the focus of the Campaign to Stop paign. Originally designed for the Cold War, weapons with similar capabilities are clearly Killer Robots has been very much on weapons

platforms – air- or ground-based robots such as unmanned aerial vehicles – that if fully 29 January 2014. The author thanks the other mem- 11 Aerospace (2013) ‘The ‘drone’ debate the ethics of autonomous could decide to fire weapons. bers of the panel – Christian Holmboe Ruge, Alexan- armed unmanned aircraft’. November issue, pp. 14-19. The reality of current autonomous lethal der Harang and Simon O’Connor – and members of URL: http://aerosociety.com/Assets/Docs/ Publica- systems is much more prosaic. The Brim- tions/The_Ethics_of_Armed_UAS.pdf (accessed 4 the audience for their insightful comments. Kristin stone and other ‘fire and forget’ missiles do April 2014). Bergtora Sandvik provided very astute feedback on a not look like robots as popularly conceived, draft. The author also appreciates the assistance 12 See Markowitz, Mike and John D. Gresham (2012) even if they meet the definition provided by provided by Maral Mirshahi in editing the brief prior Dual-Mode Brimstone Missile Proves Itself in Com- the Campaign, Noel Sharkey and Christof to publication. bat. Defense Media Network, 26 April. URL: Heyns. This mirrors the general development http://www.defensemedianetwork.com/stories/dual- of automation. In decades gone by, science- 2 Lewis, Jeffrey (2013) Poison Control – Why Can’t mode-brimstone-missile-proves-itself-in-combat/ fiction authors conceived of intelligent hu- We Get Rid of Nukes the Same Way We Got Rid of (accessed 2 February 2014). manoid robots that would mimic the activities Chemical Weapons? Foreign Policy, 14 October. URL: of humans. In fact, technology has developed http://www.foreignpolicy.com/articles/2013/ 13 UK Ministry of Defence (2011) RAF Conducts very differently. For example, instead of hav- 10/14/poison_control_nuclear_chemical_weapons Precision Strikes over Libya. Press Release, 16 Sep- ing one general-purpose robot that would take (accessed 27 March 2014). tember. URL: https://www.gov.uk/government/news/ over people’s everyday tasks, there are now raf-conducts-precision-strikes-over-libya (accessed 2 3 Sharkey, Noel (2012) Automating Warfare: Lessons numerous examples of machines that use February 2014). autonomous decisionmaking to fulfil specific Learned from the Drones. Journal of Law, Information 14 limited tasks: for example, a smartphone that & Science 21(2): p. 2. Kongsberg Gruppen (2013) Kongsberg and Boeing Complete Joint Strike Missile (JSM) Check on F/A-18 accepts spoken instructions and recommends 4 Human Rights Watch (2012) Losing Humanity: The Super Hornet. Press Release, 29 November. URL: music; robots that will clean a room without Case Against Killer Robots, p.2. direct human control; and cars equipped with http://www.kongsberg.com/en/kog/news/2013/novem 5 sensors that detect an incoming cyclist or Ibid, p.6. ber/kongsberg-and-boeing-complete-joint-strike- missile/ (accessed 2 February 2014). pedestrian and automatically brake to prevent 6 Heyns, Christof (2013) Report of the Special Rappor- a collision. The battlefield is following a simi- teur on Extrajudicial, Summary or Arbitrary Executions. 15 MBDA Missile Systems (2014) EXOCET MM40 – lar trajectory. Instead of Terminator-like hu- A/HRC/23/47. New York: United Nations, pp. 6–7. New Generation Long Range Anti-Ship Missile Sys- manoid machines, the lethal autonomous tem. URL: http://www.mbda- 7 robots of today, and most likely the future, are UK Ministry of Defence (2011) The UK Approach to systems.com/mediagallery /files/exocet_mm40_ds.pdf those in which sensors and artificial deci- Unmanned Aircraft Systems, pp. 2-2 to 2-3. (accessed 5 March 2014). sionmaking are applied to a specific and lim- 8 Sharkey, Noel (2012) Automating Warfare: Lessons 16 ited role. The challenge for the Campaign will Wired (2008) Robot Wars Continues. 31 March. Learned from the Drones. Journal of Law, Information URL: http://www.wired.com/dangerroom/ be to capture the public imagination in order & Science 21(2): pp. 2-3. to build a sufficiently large coalition able to 2008/03/the-robot-war-c/ (accessed 5 March 2014). 9 Handy, Brian (ed.) (2007) Royal Air Force Aircraft & prohibit such a heterogeneous array of lethal autonomous weapons. Weapons. DCC(RAF) Publications, p. 87. URL: http://www.raf.mod.uk/rafcms/mediafiles/0186cc2a_1 143_ec82_2ef2bffff37857da.pdf (accessed 5 March Notes 2014). 10 UK Ministry of Defence (2011) Joint Doctrine 2/11: 1 This PRIO Policy Brief is based upon remarks made The UK Approach to Unmanned Aircraft Systems, p. 5-4. by Nicholas Marsh at a seminar on autonomous lethal URL: http://dronewarsuk.files.wordpress.com/ robots organized by the Norwegian Centre for 2011/04/uk-approach-to-uav.pdf (accessed 27 March Humanitarian Studies, which took place at PRIO on 2014)

THE AUTHOR THE PROJECT PRIO

Nicholas Marsh is a Researcher at PRIO and This policy brief is a product of the Critical The Peace Research Institute Oslo (PRIO) is a since 2001 has worked there on the trade, Humanitarian Technology Project. The pro- non-profit peace research institute (estab- production, use and governance of weapons. ject examines the influence of new technolo- lished in 1959) whose overarching purpose is His current research interests include the use gies upon humanitarian action, and aims to to conduct research on the conditions for of weapons by non-state parties involved in inform the development of a Norwegian poli- peaceful relations between states, groups and civil war, and the international trade in small cy on humanitarian technology. For more people. The institute is independent, interna- arms and light weapoweapons.ns. Email: [email protected] information about the project, please visit: tional and interdisciplinary, and explores is- www.humanitarianstudies.no. sues related to all facets of peace and conflict.

PRIO POLICY BRIEF 02 2014 www.prio.org