Defending the Boundary: Constraints And

Academy briefing n°9

Defending the Boundary

Constraints and Requirements on the Use of Autonomous Weapon Systems Under International Humanitarian and Human Rights Law

Maya Brehm - MAy 2017

The academy a joint center of of theirboundaries,orthestatusanystatesterritoriesthatborderthem. such statesorterritories,theirauthoritiesandinstitutions,thedelimitation any judgement by the Geneva Academy or the FDFA regarding the legalstatusof or commentedon,drafts.Thedesignationofstatesterritoriesdoesnotimply ily reflectthoseoftheproject’ssupportersoranyonewhoprovidedinputto, This Briefingistheworkofauthor.Theviewsexpressedinitdonotnecessar- eign Affairs(FDFA)foritssupporttotheGenevaAcademy’sresearchonthisissue. The GenevaAcademywouldliketothanktheSwissFederalDepartmentofFor- draft, andtoMunizhaAhmad-Cookeforhermeticulouscopy-editing. Michael SiegristandOzlemUlgenwhoprovidedhelpfulcommentsonanearlier With thankstoNeilDavison,MerelEkelhof,SandraKrähenmann,RichardMoyes, the GenevaAcademyofInternationalHumanitarianLawandHumanRights. This Academy Briefing was researched and written by Maya Brehm, Researcher at Disclaimer A ckno of Autonomous Weapon S wledgements Defending the Bound C onstraints and Humanitarian and ystems Under International Requirements on the Use Human Rights La ary w

Acknowledgements 3 DEFENDING THE BOUNDARY 4 c 6. 4. 3. 2. e 4. c a 3. 2. introduction 1. key Messages

5. extra- a a

c envisioning Autonomous Weapon S human Rights Requirements and aB the Applicable La oncluding out this S D. B. B. D. . . . Remarks . . . tud animus Belligerendi: A algorithmic T the S the Belligerent Nexus: A

preliminaryFindings on the Applicable La the Duty to Indiviuate the Use of F human RightsTreaty ObligationsAbroa d human C automated

and ‘ to C Non-Lethal’ AutonomousInterception w : IHL

onduct Hostilities Human Dignity surveillance 1. cope for Categorical Killing under IHL Calculated Territorial C and S Process ontrol and IHRL Kill Zones: Preparing the Ground orting arget C C S onstraints on the Use ofA tand ystems

People Matters ontrol the Use ofF onstruction: A ard Blindness

WS WS

s on the Use of F and

, C

ontrol and the Intent to Wage War orce

orce under IHRL Threat to Human Rights

the Intent C : A orce w WS WS ontents for Sentry- A

and

WS ?

49 68 42 42 26 30 45 65 59 56 63 39 52 23 33 19 57 13 11 6 7

DEFENDING THE BOUNDARY 5 DEFENDING THE BOUNDARY 6 • • • use extremely To safeguard humandignityandrights,agentsmust: cedural use the of stances, and but Where Autonomous their framework international — — — — constraints of of — — — — force, an IHL use. requirements it exercise be concrete circumstances ables legal remain operations asrequired exercise force application outsideof theconduct of hostilities human they operations inatimelymanner so would AWS limited permits as continuously Consideration there weapon must rules them to of on in involved control not humanitarian active the the enable the an under is the govern appropriately to be scope 1980 armed control use and systems ‘categorical’ explain the over and IHRL. them in and of the of applications Convention primary algorithmic AWS for constant conflict international every necessary actively need (AWS) law the IHRL to the must recognize bound reasoning (IHL) individual lawful targeting legal to requirements in tend (continuous be (personally) individuate on targeting so of to would frame a every Key Messages Certain human use part force far to determine, changing underlying of attack be as of never of of attack security by portrayed they processes an the rights reference Conventional or the engaged and means in AWS. be debate at circumstances are use in the in constraints least the law measures, algorithmic a spatio-temporal of However, not conduct of timely as in to sole, (IHRL) in on an frequent, force, a assess ‘weapons displaced every AWS, Weapons manner AWS, and manner, requirements including of apply this

decisions due including instance the in and and hostilities; periodic) many scope that of legality by to (CCW). to adjust adapt terms what war’, pro- IHL. en- the the in- of in in is

mous-technologies-en-606.pdf unidir.ch/files/publications/pdfs/framing-discussions-on-the-weaponization-of-increasingly-autono Discussions ontheWeaponization of Increasingly Autonomous Technologies, Systems’, tonomy abs/10.1080/00963402.2016.1194619?journalCode=rbul20. cost, data cautioned rocks.com/2016/09/capturing-flying-insects-a-machine-learning-approach-to-targeting/). 5 AI Magazine(2015) 105–114, http://futureoflife.org/data/documents/research_priorities.pdf. Insects: 4 July 2015, http://futureoflife.org/open-letter-autonomous-weapons/. tary 3 SaratA. (ed), Law, Violence, andthePossibility of Justice, Princeton University Press, 2001, p3. istential threattohumankind. AI maybringunprecedentedbenefitstohumanity,itisalsoarguedposeanex- policing, althoughthatargumentisseldommade).Whilstsuccessinthequestfor 2 G.Chamayou, Drone Theory, Penguin Books,2015, p211. 1 to physicalharm. persistence andendurance,reducetheexposureofstates’ownsecurityforces cycle), improveperformanceincommunications-deniedenvironments,increase process growingdatamuchmoreefficiently(speedingupthetargeting–decision for thistrend,commentatorsciteaperceivedneedtoreactthreatsmorequickly, the statedgoalofincreasingautonomyinweaponsystems.Regardingdrivers A smallnumberofstatesareactivelyengagedinresearchanddevelopmentwith ble withinyears’. point wherethedeploymentofsuchsystemsis—practicallyifnotlegallyfeasi- the fieldofartificialintelligence(AI)androbotics,AItechnologyhas‘reacheda targets withoutfurtherhumanintervention.Accordingtoleadingresearchersin idea isthatonceactivated,suchweaponsystemswoulddetect,selectandattack legal andsecurityimplicationsofautonomousweaponsystems(AWS).Thebasic Over recentyears,therehasbeengrowingdebateabouttheethical,humanitarian, depends always threatens thevalues forwhichlawstands.’(A.Sarat) law which on violence the But boundaries. law’s beyond just situated violence destructive and lethal more allegedly the to counterpunch a as it uses and violence on depends ‘Law as theefficientcauseofdeathwouldturnouttobevictim…’(G.Chamayou) identifiable directly agent human only the automatic, purely is decision lethal the ‘[W]hen S. A. ‘Autonomous ‘The to load, Russell, see Sarat, integrate in A changing R. critical that and Machine 72 ‘Situating R. D. have Hoffmann, increasing Bulletin of theAtomic Scientists 4 Dewey Weapons: functions. machine character Learning pointed 3 Law 5 and AWSaresaidtohavethepotentialrevolutionizewarfare(and automation Between T. M. learning See, An out M. of Tegmark, Approach . Open Cullen war, e.g., that For the into United fleeting the Letter a exacerbates, 4 and ‘Research Realities critical to its protection Targeting’, J. Nations from targeting nature K. appraisal Hawley, of Priorities AI 1. Introduction Violence rather Institute of & of process War ontheRocks, Robotics a targets, state’s (2016) ‘The than for of and for Robust the Myths addresses to Disarmament own Researchers’, the and 247–255, win claim Claims forces and and glut wars’ Beneficial that Costs challenges of 6 of can http://tandfonline.com/doi/ (C. Sep big Research 2014, Justice: Future AWS be of Lewis, data 2 2016, enhanced Autonomous Artificial pp would linked of An (UNIDIR), requires 5–6, ‘Capturing Life http://waronthe Introduction’, to Others Intelligence’, http://www. help 1 Institute, without speed the Weapon Framing reduce Flying have mili- and au- 28 in

DEFENDING THE BOUNDARY 7 DEFENDING THE BOUNDARY 8 http://www.dod.mil/dodgc/images/law_war_manual15.pdf. USDoD, Law of War Manual,June2015, s6.5.9.2, p329, 9 Weapons Which May beDeemed to beExcessively Injuriousorto Have Indiscriminate Effects. Fifth objectives 8 Indiscriminate Effects, UNdocCCW/CONF.V/10, 23December 2016. Working_Paper_031315.pdf; Security on the-discussion-forward-en-615.pdf; http://www.unidir.ch/files/publications/pdfs/considering-how-meaningful-human-control-might-move- Technologies: Considering How Meaningful HumanControl Might Move theDiscussion Forward , ReportLAWS_2016_AdvancedVersion.pdf; www.unog.ch/80256EDD006B8954/(httpAssets)/DDC13B243BA863E6C1257FDB00380A88/$file/ to Parties wp-content/uploads/2013/11/Autonomous-weapons-memo-for-CCW.pdf; Convention Meaningful Weapons Systems: Implications of Increasing Autonomy intheCritical Functionsof Weapons, Meeting, 7 November21 2012, s4(a),http://www.dtic.mil/whs/directives/corres/pdf/300009p.pdf of autonomyinweaponsystems. At present,noruleofinternationallawspecificallyprohibitsorrestrictstheuse Parties that ‘… any use of force, including through[AWS], must strictly comply 1980 ConventiononCertainConventionalWeapons(CCW), meetings on ‘lethal autonomous weapon systems’ held in the framework of the longer the case wasa point of contention among participants in informal expert including preventiveandremedialmeasures.Inwhatcircumstancesthisisno and totakealllegallyrequiredstepsrespectensureforthelaw, the opportunityandcapacitytoassesscompliancewithapplicablelegalnorms AWS wouldseemtoentailthathumanagentsinvolvedintheuseofanhave From alegalperspective,therequirementformeaningfulhumancontrolover What thatinvolves,concretely,remainstobeclarified. 6 ings mustretain‘meaningful’,‘appropriate’or‘effective’controloverweapons. implications, policymakersandcommentatorshaveemphasizedthathumanbe- es. Outofconcernoverseriousnegativeethical,humanitarian,legalandsecurity by whichhumanbeingsexercisecontrolovertheuseofforceanditsconsequenc- The useofAWScanbeexpectedtochangethemannerinwhichandprocesses autonomous-weapons-systems.html; rules ofinternationallawapplicableinarmedconflict’. promote disarmament and the ‘codification and progressive development of the to ‘prohibitorrestrictfurthertheuseofcertainconventionalweapons’inorder chaired emerging the Article The Preamble, Review Use decided CCW by (CNAS), Versoix, Systems 36, and of France, Human on technologies Conference Certain held Structuring 1980 Certain purposes to March Switzerland, formalize informal (LAWS) and Control inWeapon Systems: APrimer, Conventional Convention Conventional in of 2015, of in Debate April the submitted the the these expert International 15–16 High https://s3.amazonaws.com/files.cnas.org/documents/Ethical_Autonomy_ area Convention’, 2015 on on Weapons discussions United meetings Contracting Weapons of Autonomous Report Prohibitions March by and lethal the 9 States Committee ThereisgeneralagreementamongCCWStates UNIDIR, April of Which 2016, to autonomous Chairperson the on and (CCW), be Parties 2016, Department 2016 ‘Lethal Weapons or established August chaired May The Weaponization of Increasingly Autonomous Restrictions 14–15 of Informal chaired Be to Autonomous the weapons the 2016, Deemed (Advanced by Working Systems, November of Convention India Red a by Meeting Defense Group p on Germany. 7, to Cross systems (Final Paper, Memorandum https://shop.icrc.org/publications/ 8 Be the Version),

of Weapon 2013, of (US Excessively Governmental M. on (ICRC), Document Experts Use Center (LAWS) DoD), C. Prohibitions In 7 http://www.article36.org/ aUNtreatyaiming Horowitz of April December Systems’ Autonomous Weapon on Directive Certain for for in Injurious Lethal (Advance 2016, the a Delegates . Experts and Report, New or context in Conventional 2016, Autonomous no §15, Restrictions May P. or American 3000.09, Version), Scharre, ‘related to http:// Expert States to of 2014, 2014, Have the the 6

the under A/HRC/23/47, 12 files/insight/SIPRIInsight1501.pdf. UN docA/HRC/23/47, supra fn 13,§63. 15 UNdocA/65/321, supra fn12,§48. 14 Summary orArbitrary Executions, 2 September UNdocA/71/372, 2016, §§75–83. 1977 Weapons, Means andMethods of Warfare: Measures to ImplementArticle 36of Additional Protocol Iof International 13 A/65/321, 23August 2010, §39. of Increasing Autonomy inWeapon Systems, Weapons Degrees 11 autonomous functionsfalls withinthedefinitionof a“mine”...itwould beregulated assuch’. Based” LAWS+MX_CountryPaper+Switzerland.pdf. unog.ch/80256EDD006B8954/(httpAssets)/D2D66A9C427958D6C1257F8700415473/$file/2016_ ple ofhumanity. life-and-death decisionscouldbeincompatiblewithhumandignityandtheprinci- tion oflawintoalgorithms,andcautionedthattakinghumandeliberationout in translating context-dependent, value-based judgements implicit in the applica- pliance withinternationalhumanrightsandhumanitarianlaw’. technologies areoptimizedintermsoftheircapacitytopromotemoreeffectivecom- sideration to the‘ways in whichproactive steps can betaken to ensurethat[robotic] Lethal 10 of complyingwiththerequirementsIHL’, UN SpecialRapporteurshavearguedthattotheextentAWSarenot‘capable ‘proactive steps’shouldinvolvelegallybindingmeasuresattheinternationallevel. Participants in ongoing multilateralpolicy discussions aredividedon whether with internationallawand,intimesofarmedconflict,IHL’. rization’. changing circumstancesandinacontextwhichisnotreadilysusceptibleofcatego- – ‘decision-making [that] requires the exercise of judgement, sometimes in rapidly to distinguishbetweenthosewhomaybelawfullytargetedandnot among others,notedthedifficultythatmilitarypersonnelfaceinpresentpractice AWS. TheformerUNSpecialRapporteuronextrajudicialkillings,PhilipAlston, Views diverge,however,onthecircumstancesinwhichitwouldbelegaltousean under internationallaw. means ormethodofwarfare’wouldinsomeallcircumstancesbeprohibited es alegalobligationtodeterminewhethertheuseofanAWSas‘newweapon, also agreethat1977AdditionalProtocolI(API)totheGenevaConventionsimpos- in the Protection Art Interim E.g., ‘CCW , 2006, customary Autonomous US Approach 36, of Report Systems DoD Meeting 12 Report 1977 Autonomy’, p Alston’ssuccessor,ChristofHeyns,elaboratedonthechallengesinvolved Peace 4). 9 of Law of War Manual, of April IHL, to For Victims Protocol of (LAWS), of Weapons Research the LAWS’, 13 a Experts: binding the Alstoncalledonthe international community to giveurgentcon- 2013, discussion, Special Statement of Special Additional Informal Geneva, International §§63–74 Institute Systems all Possible 11 Rapporteur parties

Rapporteur see Working by supra 13–17 (LAWS), (SIPRI), V. to and Challenges Switzerland, to Boulanin, the Armed In April fn an 89–97; on Paper SIPRI addition, Geneva 9, on November armed Geneva, Extrajudicial, 2015, s Extrajudicial, Conflicts 6.5.9.2, Implementing Article36Weapon Reviews intheLight Insights to submitted Report CCW conflict International Conventions 15 11–15 other

15 2015, April p or,morebroadly,thatthey‘require (AP Meeting on of April 329, weapon Summary (ICRC, by Summary Peace the https://www.sipri.org/sites/default/files/ I). 2015. Switzerland, 2016, Arguably, ‘to of Special Humanitarian A Guideto theLegal Review of New of and the treaties See 12 Experts or 30 or August Security extent also Arbitrary March Arbitrary Rapporteur a CCW may corresponding ‘Towards 14 on a 1949, 2016, Law no Meeting apply. weapon Lethal 10 Executions, Executions, 2015/1, StatesParties Due §8, on and a As “Compliance- to of Autonomous Extrajudicial, http://www. system pointed Relating duty Experts Stockholm Increasing UN UN exists with doc doc out on to

INTRODUCTION 9 INTRODUCTION 10 of ‘capable LAWS+MX_ChallengestoIHL_Statements_United+Kingdom.pdf, unog.ch/80256EDD006B8954/(httpAssets)/37B0481990BC31DAC1257F940053D2AE/$file/2016_ Lethal Geneva, 21005A2A95/$file/LAWSPaper_Pakistan_CCW.pdf; 2015 lutions 20 Journal (2013) 233. Kacwv Autonomous 21 LawAndEthics_r2_FINAL.pdf. University, be moreappropriate, guidance onlegalreviews,bestpracticesoramanualIHLinterpretation)would arguing thatmorelimitedrestrictionsornon-legallybindingmeasures(suchas by somegovernmentsandscientists. 18 The Campaign to Stop KillerRobots, http://www.stopkillerrobots.org/. 17 Management of Assemblies, UNdocA/HRC/31/66, 4February 2016, §67(f). 19 Robots’, http://icrac.net/call/. 16 no meaningfulhumancontrol’, coalition iscampaigningforapreventivebanon‘killerrobots’, ‘accommodate’ issuesraisedbyincreasingautonomyinweaponsystems. War-Algorithm Accountability, Work and How theLaws of War Can, the Weapon and Accountability-Appendices-Only-Searchable-August-2016.pdf Association Experts See, Loop”: See, Armed Joint K. K. and ; Autonomous in Anderson Anderson, International 13–17 e.g., of Systems’, e.g., military 2016 Report Autonomous dealing and Conflict, 2013, on ‘Inputs Weapons April Statement CCW Lethal the and technology. of D. 90 2015, with by p Special Weapons August Committee meetings the Reisner M. Pakistan’, International Law Studies Systems’, 23, Autonomous an Weapon Waxman, http://unog.ch/80256EDD006B8954/(httpAssets)/14636F3813F314DBC1257E Special by 20 evolution Rapporteur orconsiderthattheexistinglegalframeworkissufficientto http://media.hoover.org/sites/default/files/documents/Anderson-Waxman_ 2016, the For and Systems of Research CCW for Working United a Rapporteur experts, Systems Law andEthicsfor Autonomous Weapon Systems: Why aBan Won’t M. compilation p Robot Meeting in 150, Weapons Waxman, A (LAWS), automation’ on 16 Kingdom Briefing, National theyshouldbeprohibited.Aglobalcivilsociety Paper http://blogs.harvard.edu/pilac/files/2016/09/War-Algorithm- see and Arms Extrajudicial, of on 18 Appendix the of Experts Geneva, Othershaverejectedabanas‘misguided’, ‘Adapting submitted Systems Control, (2014) the states’ of Security Harvard Law ‘Elements as Great Rights it on of Summary II 395; 11–15 positions has ‘The Lethal (LAWS), to Armed the and by Britain Law . to ‘successfully D. M. Scientists’ the April expressing Supporting Law A. Freedom Law Autonomous School N. or as Lewis, Holy Conflict’, and Geneva, Schmitt 2016, of Essay, Arbitrary reflected See, Armed Northern Call Program G. accommodated’ of confidence 7 Hoover 4 Blum the and CCW to Peaceful April 11–15 Weapons Harvard NationalSecurity Executions in Conflict Ban 17 J. their Prohibition Meeting Ireland, and on 2016, acallsupported S. Institution, April Autonomous Thurnher, International statements N. Assembly that Systems to http://bit.ly/2e K. on 2016, CCW previous of Autonomous Modirzadeh, IHL 21 the Experts of Stanford Meeting (LAWS), ‘“Out will and Proper http:// Lethal Lethal at evo- Law the on be of of 19

access toanddefendadelimitedzone,forexample,aroundmilitary camp,adeten- The useofautonomoustechnologiestosecureaperimeteror boundaryanddeny pose tohumandignityandtheprotectionofrights. gorithm-based decisionmakinggenerally,anduseofforcemeasures specifically, rare, anumberofhumanrightscasesprovideinsightsintothe challengesthatal- of HumanRights(ECtHR).Althoughcaselawonautomated killingisrelatively draws onhumanrightsjurisprudence,especiallycaselawof theEuropeanCourt The studyanalysesinternationallegalinstrumentsandscholarly writings,and interplay betweenIHRLandIHLis,however,beyondthescope ofthispaper. lawful useofanAWS.Acomprehensivetreatmentthecomplexandunsettled ty of IHL affects obligations arising under IHRL, thereby affecting the scope for the may affectthelawapplicabletouseofforce,andexploreshowapplicabili- rently with IHL. The study critically examines how theemployment of an AWS hostilities paradigm,andthatIHRLappliestotheconductofconcur- in someinstancesfallwithinalawenforcementandothersconductof premises thatduringanarmedconflict,theuseofforcebymeansAWSmay poses, bothduringandoutsideofanarmedconflict.Theanalysisisbasedonthe of anAWSinrelationtotheconducthostilitiesandforlawenforcementpur- the primaryframe of referencetoassessthelegality of AWSuse.It looks at theuse This studytakesastepbackandcriticallyexaminesthepresuppositionthatIHLis use ofweaponsasmeanswarfare. rather thanpolicing,andbecausediscussionswithintheCCWarelimitedto tors andpolicymakersenvisiontheuseofAWSincontextmilitarycombat, May https://www.hrw.org/sites/default/files/reports/arms0514_ForUpload_0.pdf 2014, at 22 of AWSonhumanrightsprotection. the ruleondistinction.Comparablylittleattentionhasbeengiventoimpact IHL rulesontheconductofhostilities,inparticular,keytargeting,suchas tions raisedbyAWS.Thefocusofscholarlyinquiryhasbeenoncompliancewith There isarichandrapidlygrowingbodyofliteratureaddressingmanylegalques- place ontheuseofforcebymeansanAWS. person human the of protection the for standards legal that international constraints and requirements the of understanding the deepen humanity,to safeguard aims that study norms this legal international promoting and with compliance ensuring at aimed measures ulatory reg- potential on discussions multilateral supporting to view a With documents/act30/1401/2015/en/; Weapons Systems: Five Key Human RightsIssues for Consideration, Law Harvard C. Enforcement’, Heyns, Law ‘Human School 38 Human Rights Quarterly (IHRC), Rights Shaking theFoundations: The HumanRightsImplications of KillerRobots, and Human the 2. Ab Use Rights of 22 Autonomous 2 Thisisprobablybecausemanycommenta- Watch (2016) out this S (HRW) 350–378; Weapons and the Amnesty 2015, Systems International https://www.amnesty.org/en/ International, (AWS) Human . During tud Autonomous Rights Domestic Clinic y

DEFENDING THE BOUNDARY 11 ABOUT THIS STUDY 12 scientific Paradigms of Application , Gesetzgebung, and limitedthereforeanaprioriacceptableapplication. to secureaboundarytendsbeportrayedbyproponentsofAWSasdefensive regarded asundesirablebymostcommentators,theuseofanautonomoussystem could beusedtoadministerviolenceinaspatiallyunboundedmanner,tends Whereas a ‘hunter-killerscenario’, where an AWS with mobile weapon platforms 24 Law: Unruly Law, CambridgeUniversity Press, 2013, pp1–14. 23 and whoseexistenceisuncertain,withinaknowableregulatedcontext. has theadvantageofsituatingtechnologieswhosecharacteristicsareunknown backdrop tothelegaldiscussion.ConcentratingonaconcreteapplicationofAWS tion centreoralonganinternationalborder(so-called‘sentry-AWS’)providesthe and humandignity. process ofautonomous,algorithmictargetconstructionthreatens humanrights praisals ofpresentsecuritypractices,thefinalpartstudyelucidateshow degrading treatment,andtherighttoaneffectiveremedy.Drawingoncriticalap- right not to be discriminated against and not to be subjected to cruel, inhuman or rights tolife,freedomofmovement,libertyandsecurityperson,privacy,the ments andconstraintsontheuseofsentry-AWS.Thefocusischallengesto on automatedkilling,thelastpartofstudyinvestigatesIHRLandIHLrequire- of violencetoanongoingarmedconflict.Againstthebackdropjurisprudence triggering aninternationalarmedconflict(IAC)andthebelligerentnexusofact bates: extraterritorialobligationsunderhumanrightstreaties,thethresholdfor and IHLstandardsontheuseofforceincontextthreeunsettledlegalde- and controlmanifestedwhenanAWSisusedimpactontheapplicabilityofIHRL cation inaconcretesituation,thestudyexamineshowchangeshumanintent force ispresented.Astheapplicabilityofalegalruleprerequisiteforitsappli- concern tothestudy.Followingthis,legalframeworkgoverninguseof AWS. Thisdiscussionalsoservestocircumscribepracticesandtechnologiesof AWS, humanagencyandcontrol,theapplicationoflegalrulesinuse The study is organized as follows: it starts out by discussing ways of envisioning war andpeacebetweenhostilitieslawenforcementaremanifested. also becausebordersarezonesofcontestationwherethejuridicaldividesbetween only becausethisbringsintoquestionhowhumanbeingsexercisecontrol,but border technology deployed to control (movement, in particular) is interesting not ‘Robotic (2013) 137–138, http://www.tandfonline.com/doi/abs/10.1080/15027570.2013.818399. On A. M. the Weapons discourses, Johnson in politics Human-Machine Springer Nomos and and of see Verlagsgesellschaft, Democratic treating Science S. J. Axinn, Weber, Translations’, certain + ‘The Business Decision-Making’, ‘Black-Boxing Morality aspects 2013, Media in M. as of 211–230. Carrier B. Autonomous (un)known in Organisms, V., E. 2011, and Hilgendorf See A. pp also or Nordmann Robots’, 409–429; Exploiting (un)knowable F. and Johns, J. 12 P. (eds), Non-Legality inInternational E. the Journal of Military Ethics 24 Günther Datteri LookingatAWSas in Unpredictable: Science intheContext juridical and (eds), G. and Robotik und Tamburrini, techno- Control 23 2

ifsh.de/file-IFAR/pdf_deutsch/IFAR-WP20.pdf scriptive Institute Multidimensional Autonomy RiskAssessment (MARA) inWeapons Systems, 27 no weapons’. 29 human-supervised autonomous weapon systems). 28 https://www.law.upenn.edu/live/files/3884-cummings-the-human-role-in-autonomous-weapons. 26 the-Scope.pdf. University, Issues that 30 wp-content/uploads/2016/06/27-humans-drones-marra-mcneil.pdf. Weapons’, Conventional losing-humanity/case-against-killer-robots. ‘Meaningful debate. modes and locales ofhuman agency intheuseof force andshould be part of the other (highly)automatedtechnologiesofferimportantinsightsintothechanging mines, torpedoes,sentryguns,automatedanti-missilesystems,armeddronesand discussions onAWS,othersstressthatpastandpresentviolentpracticesinvolving mous systemswillbecriticalforthedebateaboutfutureAWS. they havea(hu)manin,onoroutof‘theloop’. sion-making cycleandtodistinguishamongweaponsystemsbasedonwhether common approachistofocusontheroleofhumanagentsinmilitarydeci- ‘on-the-loop’ systemsaresometimestermed‘fullyautonomous’weapons. with taxonomiesdescribingdegreesofautonomyinweaponsystems. 25 response. they identifyasproblematicandtheirorientationtowardapotentialregulatory agency andweaponsindifferentways.Thisaffectswhattechnologiesorpractices Rather, participantsinmultilateralpolicydiscussionsframeissuesofautonomy, rect nomenclature. cor- the on agree to failure a to due simply not is understanding mon it may not be constructive to tightly define the term. The lack of a com- There is no common understanding of what an AWS is and at this stage Autonomous Weapon S 3000.09, HRW, M. US such W. For See, L. in DoD, C. for 26 characteristics a Cummings, technologies Losing Humanity:The Case Against KillerRobots, e.g., Governance Marra Drawing Someconsiderthatthedifferentiationbetweenautomatedandautono- brief Briefing 25 Peace Human Law of War Manual, https://globalsecurity.asu.edu/sites/default/files/files/Control-or-Judgment-Understanding- supra Weapons Whereassomeseektoexcludeexistingweaponsystemsfrompolicy M. and discussion Dickow, Research on fn Control Paper ‘The S. experience 6, would Studies K. (CCW), of Glossary, Human McNeil, systems A. for of or and or Dahlmann, proposed Geneva, Appropriate no should Delegates supra Security Role with Understanding “The Loop”: HumansandtheNext Drone Generations, Part to 50, gauge fn existing in fall II 12–16 The 9, Autonomous C. definitions (distinguishing Policy within at s Alwardt, Human 6.5.9.1, Brookings their (proposing the technologies December at the autonomous the Review p Judgment? ambit of F. 329, Weapon 3. Envisioning University Institution, AWS Sauer 2012, between a 2016, describes 29 framework of Conference to ‘Man-out-of-the-loop’andcertain a within https://www.hrw.org/report/2012/11/19/ inform and Design potential capabilities). Global The of autonomous, N. 2012, Hamburg, mines the the Necessary and to Schörnig, Security of future debate quantify CCW Deployment’, https://www.brookings.edu/ as the See ‘rudimentary December context, ystems legal Working Initiative, Convention Limits semi-autonomous, on First Steps Towards a also 27 and Othersoperate AWS restriction US compute on see [undated], does Paper DoD, 2015, Arizona 28 Autonomous autonomous Another on H. not Directive 30 on M. https:// key no Certain Irre- mean AWS. State Roff, and p de- 20, 5,

DEFENDING THE BOUNDARY 13 DEFENDING THE BOUNDARY 14 37 decide, act,team’, http://www.acq.osd.mil/dsb/reports/2010s/DSBSS15.pdf. able 36 Cross (IRRC) 886(2012) 787–799. ceptualizes ‘Future shaping with activeorpassivesensorstonavigateanddetectobjects,motionpatterns. as mobileroboticcomponents(e.g.unmannedair,groundornavalvehicles)equipped (eds), 35 Anderson andWaxman, Law andEthicsfor Autonomous Weapon Systems, supra fn20, p2. 34 of Chicago Press, 2015, p4). 31 with sensors,algorithmsandeffectors. tonomy andAI),itissufficienttodescribeanAWSschematicallyasaweaponsystem For thepurposesofthispaper(andside-steppingcomplicateddiscussionsaboutau- as aweaponsystemwithautonomyin‘criticalfunctions’. International Committee of the Red Cross (ICRC) has proposed to define an AWS spective of whether a weapon system is described as (fully) autonomous or not, the 33 E.g., Article36,Structuring Debate onAutonomous Weapons Systems, supra fn6. 32 Expert Meeting, Geneva, Switzerland, 26–28 March November 2014, p62. 2014, ducting warinevermoremoralandlegalways. embrace thenarrativethatincreasingautonomyinweaponsystemsenablescon- preventive prohibitionasawayofshapingtechnologicaldevelopments. commentators tendtochallengeclaimsabouttheinevitabilityofAWSandseea mensions ofmilitarizedrationalitiesandtechnologiesintocivilianspheres.These automated killingandproblematizetheexpansionofspatialtemporaldi- Others, includingtheauthorofthisstudy,situateAWSwithinabroadertrend AWS tendstobeportrayedasinevitable,andtheirpreventiveprohibitionfutile. society doesoroughttoplayinshapingthatevolution. cio-technical imaginariesabouthowmilitarytechnologyevolvesandwhatrole AWS reflectsthatparticipantsinthedebatedrawonandgeneratedifferentso- More fundamentally,perhaps,thelackofagreementonwhatisproblematicabout located ineitherthehumanormachine. approach tendtounderstandautonomousagencyasrelational,ratherthanbeing of meaningful human control is gaining increasing traction. ly, anapproachcenteringonthequestionofwhetherAWSoperatesoutside sense ofspatially,temporallyorcausallyproximate,humanintervention.Final- be abletodetect,track,selectandattack(e.g.fireat)atargetwithoutdirect,inthe tect makers and Applications, feelings Sensors persons, E.g. ICRC, Defense An through Dreamscapes of Modernity:Sociotechnical Imaginariesandthe Fabrication of Power, Imperfect: and imaginary N. scientific about Autonomous Weapon Systems: Technical, Military, Legal andHumanitarian Aspects, E. ‘technologies can other see, Sharkey, Science science something. include Springer e.g., and actors Science, can and Board ‘The ‘Detectors policy be electro-optical, draw critical technology, International Evitability described Imaginaries Technology, (DSB), developments. on of to and Humans’, Summer Study onAutonomy, the of as as generate infrared, Publishing, help Autonomous development and a well shared Socio-technical the produce in as 36 J. radar Imaginations about imaginaries Such a system can include stationary as well Suchasystemcanincludestationaryaswell Fraden, vision 2016, Robot systems of or what autonomous sonar. pp (an Handbook of ModernSensors: Physics, Designs, Warfare’, imaginaries 271–333, 34 ought to imagined of Amongthisgroup,theadventof of For inform Modernity’, meaning Report, a to recent doi: 94 systems’ be and future), encode International Review of theRed 10.1007/978-3-319-19303-8_7. attained US 33 31 in survey justify Somecommentators Suchasystemwould in DoD, a 32 in S. society. visions Proponents of this symbols terms Jasanoff of June their (see, sensors of Scientists, of 2016, actions, e.g., and what The ‘sense, and 35 S. used associated University p Jasanoff, is S-H. thereby 11, Report, 37 attain- think/ to policy

con- Kim de- 19, See also Anderson etal,‘Adapting theLaw of Armed Conflict to Autonomous Weapon Systems’, supra fn operated andcontrolled andof theoperational context and conditions). 43 issuu.com/nato_ccd_coe/docs/tallinnmanual. International Law Applicable to Cyber Warfare, org/amw/Commentary%20on%20the%20HPCR%20Manual.pdf; ‘Evitable Culture andtheHumanities 41 Routledge, 2006. International Law Applicable to AirandMissile Warfare 40 plant mines’ (ICRC, Autonomous Weapon Systems, ExpertMeeting Report (2016), supra fn6,p 21). development, is‘designed to carry outrescue missions’ but‘could also beused to lay smoke screens and Applicable to Air and Missile Warfare research.org/amw/HPCR%20Manual.pdf; B. 42 Press, pp168–169. 2014, against whom and why’. a weaponisnotthenatureofobjectitself,butratherhowwasused, lenges arise, Harrison Dinniss stresses that ‘the key factors that determine its use as context ofhostileactivitiesinthecyberdomainwheresimilardefinitionalchal- of force. what technologiesare,theyareforandhowimplicatedintheuse ments, mechanisms,equipmentorsoftware,canformaweaponsystem. 38 a ‘non-weaponized’autonomoussystemisnotstraightforward. autonomously selecttargetsofattack.Nevertheless,differentiatinganAWSfrom (armed drones)thatcannavigateorrundiagnosticswithouthumanintervention, Neither autonomouscarsnorremote-controlled,unmanned,weaponplatforms target, andsomesortofcommunicationsystemthatallowsforhumaninteraction. ception library. tified bycomparingsensordatawithtargettypescontainedinadatabaseorper- enabling thedetection,trackingandclassificationofobjects.Targetscanbeiden- The collectedsensordataisprocessedbycomputationallyintensivealgorithms loss offunctionality. vere mentalsufferingordeathofpersons,damagetodestructionobjects design, on (system),weaponsarecommonlydescribedinlegalpracticeasdevicesthatby shape eachother.Intheabsenceofaninternationallyagreeddefinitionaweap- html. http://www.govtech.com/transportation/FBI-Says-Autonomous-Vehicles-Could-Be-Lethal-Weapons. 39 (eds), SpringerHandbookof Robotics , 2ndedn, SpringerInternational Publishing,2016, p1613. Rappert, 397 Consider See, W. H. Program Critiquing According Harrison H. (recognizing use orintendedarecapableofcausingincapacitation,injury,illness,se- e.g., Conflicts, Chun 40 Controlling theWeapons of War: Politics, Persuasion, andtheProhibition of Inhumanity, Technologiesandpracticesofviolencearemutuallyconstitutive the M. on the headline to and Dinniss, Roscini, Humanitarian a Inevitable 38 doctrine speaker N. that Finally,thesystemincludesaweaponormunitionto‘engage’ Papanikolopoulos, ‘FBI Cyber Warfare and the Laws of War, Cyber Operations andtheUse of Force inInternational Law, whether 41 (online Says Aweapon,togetherwithotherdevices,materials,instru- at of Technologies? 43 an technological Policy Autonomous Given the tightening connection between surveillance ICRC a 7 system January and (Commentary expert ‘Robot HPCR, Commentary ontheManualInternational Law Conflict operates The Vehicles neutrality 2014) meeting, Cambridge Science Surveillance Research 18–27, on autonomously Could Platforma-M, in AMW and the Be University (AMW doi: and at Fiction Lethal Cambridge Manual), context 10.1177/1743872113509443. Harvard M. Security’, Manual), Schmitt Weapons’, is, a of Press, Russian inter of Robotic 2010, University University AWS, in (ed), alia, 2013, 2009, B. GT Magazine, system p I. Warfare Siciliano 39 a 55, Tallinn Manualonthe Kerr pp function Itisdisputable §A(1)(ff), (HPCR), Manualon Press, http://ihlresearch. Oxford 141–142, reportedly and and and 17 2012, of K. 42 University Generally, July O. IHL’, http://ihl how Inthe Szilagyi, https:// Khatib under 2014, p Law it 70. is

Envisioning Autonomous Weapon Systems 15 DEFENDING THE BOUNDARY 16 ligence, cidents’, ‘prob[e]maneuversbyenemysquads’or‘interceptintruders’. obviate theneedtodispatchhumansecuritypersonnelrespond‘emergingin- will reduce manpower requirements. hope thatthereducedneedforapermanentphysicalpresenceofhumanguards combining targetdetection,identificationandthecapabilitytofire. 44 the DemilitarizedZone(DMZ)betweenNorthandSouthKorea. human intervention.OnesystemofthistypeisDoDaam’sSuperaEgisIIdeployedin ployed, butnoneofthemiscurrentlyselectingandattackingtargetswithoutdirect Sentry systemswithautonomouscapabilitiesincriticalfunctionsarealreadyde- pect that‘thecarefulinsertionofautomaticandautonomoustechnologies’ to surveyandguardboundariespatrolareas.Partisansofsuchtechnologiesex- crossing alineorenteringexitinganarea.Sentrysystemsarebeingadvertised One ofthefunctionsenvisagedforAWSistopreventpeopleorvehiclesfrom and targetingtechnologiespractices, missile launcher. missile launcher. options, witha12.7mmmachinegun,40mmgrenadelauncherorsurface-to-air kilometers atnight.Accordingtothemanufacturer,itcanbeequipped,amongother tised as being able to detect humans from 3 kilometers away in daylight and from 2.2 focus onthematerialitiesofweaponcontrol. a regulatorycategoryintheframeworkofCCW,wheredeliberationstendto AWS beginsandends.Thispresentsachallengetotheconstructionofas when their use or intended use constitutes an AWS, that is, where and when an ponents areintangibleandcanbegeographicallydistributed,itisfarfromclear to Technology’, and https://www.wired.com/2007/06/for_years_and_y/; and Security Operational Success, 46 ChunandPapanikolopoulos, ‘Robot Surveillance andSecurity’, supra fn38,p1606. 45 Criminology 3(2011) 239–245. ‘Surveillance War: 48 Shoopetal,MobileDetection Assessment andResponse Systems (MDARS), supra fn46. 47 dtic/tr/fulltext/u2/a449408.pdf. DoDaam Systems Ltd, ‘Super aEgis II’,http://www.dodaam.com/eng/sub2/menu2_1_4.php 49 2008, http://defense-update.com/20081127_sentrytech.html. C Technologies for Autonomous Area Denial, Border ‘Surveillance ontrol and close B. knowledge K. Defense N. Everyday Skibba, H. Shachtman, surveillance the Kindervater, temporal 47(3) and and Update, Mobile Detection Assessment andResponse Systems (MDARS): AForce Protection, Physical Geographies production Annihilation’, Violence Security Dialogue and ‘Robo-Snipers, ‘Lethal and ‘The Perimeter Security reconnaissance spatial from Emergence and of Presence: Space the in Afar: decisions gap Chamayou, War “Auto and (2016) between The of Remotely on Naval capabilities Lethal Politics on Kill Terror’, 224 Drone Theory, life Zones” Warfare the 47 Surveillance: (describing The other goal is to ‘close the kill chain’ by Controlled and of 41 two 44 Drones ‘are B. Antipode death to andconsideringthatsomeAWScom- … Systems Shoop, linked a Protect practice supra ‘lethal Sentries and Watching have 1 M. directly Center, Liminal (2009) fn Israeli surveillance’ become Johnston, in 1, Assume which pp and to San Security-Scapes’, 49–69; Borders’, 37–45; targeted Killing one Diego, mechanisms Guard 49 R. Thesystemisadver- as and Goehring, T. L. in a Wired Wall Amoore, http://www.dtic.mil/ Roles’, practice killing the the 48 and History same.’ , 46 of 15 6 in 27 Theyalso J. where T. ‘Algorithmic surveillance April an Theoretical Moneyhun November Monahan, . of See attempt 45 Drone will 2007, intel- also 53 R.Hughes,‘IDFDeploys SentryTech onGaza Border’, Jane’sDefence Weekly, 6June2007. 52 creasing thegun’sautomatic functionality”’(Parkin, ‘KillerRobots’, supra fn50). 57 Hughes,‘IDFDeploys SentryTech onGaza Border’, supra fn52. 56 Shachtman,‘Robo-Snipers’, supra fn46. 55 Hughes,‘IDFDeploys SentryTech onGaza Border’, supra fn52. 54 Station’, http://www.rafael.co.il/5700-744-en/Marketing.aspx. thermal cameras,andisequipped‘withauto-targetacquisitioncapture’. Ben GurionAirport.TheGuardiumcarriesvarioussensors,includingvideoand Israel along its borders with Gaza and Lebanon and has previously been used at cal] package’. sor-acquired targetwithoutverificationthroughtheweaponstation[electro-opti- however, ahumanoperatorpullsthetriggerand‘operatorcannotengagesen- on distanttargets’. A newervariantcandeliverlong-rangeanti-tankguidedmissiles,enabling‘strikes station can also carry machine guns of a larger calibre or a 40mm grenade launcher. weapon station,mountingamachinegunof5.56or7.62mmcalibre.The 50 interactions withtheenvironment. predetermined area.Inthelattercase,patrollingpathsemergebasedonrobots’ human operators, or swarms of mobile unitscan be left to self-organize within a as orperimeters.Theserobotscanfollowpatrolpathsdeterminedinadvanceby Sentry systemscancomprisemobileunitsthatallowthemtopatrolspecificare- sor-to-shooter weaponplatforms. (‘Sees-Fires’) SentryTechsystemcomprisesremotelyoperated,pre-positionedsen- Another sentrysystemisdeployedbyIsraelalongitsborderwithGaza.The‘Roeh-Yoreh’ gets, A human operator can specify the perimeter within which the system scans for tar- to ‘navigatealonethroughcities’or‘patrolborders’. developed byG-Niusfollowspre-programmedroutes.Theunitissaidtobeable tion’, butinpresentpracticeahumanoperatorunlocksthesystem’sfiringability. ‘had an auto-firing system’, enabling it to target and attack ‘without human interven- moving target’andtoissueawarningtargetbeforeanattack.Theoriginalversion “closed-loop” system–nohumaninterventionrequired’. and intervention 51 com/future/story/20150715-killer-robots-the-soldiers-that-never-sleep. 58 com/chapter/10.1007%2F978-3-540-89076-8_29#page-1. Simulation, Modeling,andProgramming for Autonomous Robots, foxnews.com/story/2008/04/28/israeli-military-unveils-armed-patrol-robot.html. A Associated F. Experiments RAFAEL S. 50 step Legras, Parkin, and the system reportedly has the capability to ‘identify, track and destroy a andthesystemreportedlyhascapabilityto‘identify,trackdestroya taken is Advanced ‘Killer A. ‘a Press, 56 temporary Glad, due with Robots: ‘Israeli to Operators’, O. Defence 54 clients’ Accordingtoonesource‘Theidea,ultimately,ishavea Simonin state’. Military The concerns. Systems Soldiers in and ‘Their Unveils S. F. Carpin, aim Ltd, Charpillet, that 52 For TheseareequippedwithRafael’sMini-Samson Armed 57 is TheGuardium,anunmannedgroundvehicle ‘Samson DoDaam Never I. now Noda, Patrol ‘Authority to Sleep’, Mini make engineers, E. Robot’, Pagello, BBC Future, RWS the Sharing Fox News, product Compact Springer, 58 however, M. Itiscurrentlydeployedby Reggiani in 16 a “smarter” 55 Stabilized July 28 Swarm 2008. the Inpresentpractice, April and 2015, requirement https://link.springer. of by 2008, O. Remote UAVs: http://www.bbc. von focusing http://www. Stryk Simulation of Weapon human on (eds), 51 “in- 53

Envisioning Autonomous Weapon Systems 17 DEFENDING THE BOUNDARY 18 Shoop etal.,MobileDetection Assessment andResponse Systems (MDARS), supra fn46,3. 67 General DynamicsRobotic Systems, ‘MDARS’, supra fn62. 66 Hughes,‘IDFDeploys SentryTech onGaza Border’, supra fn52. 65 Parkin, ‘Killer Robots’, supra fn50. 64 nicintifada.net/content/israels-video-game-killing-technology/8919. 63 pillowfists/mdars_general_dynamics_robotic_systems 62 Shoopetal,MobileDetection Assessment andResponse Systems (MDARS), supra fn46,3. 61 www.newscientist.com/article/mg22229664-400-armed-russian-robocops-to-defend-missile-bases/). warehouses andstoragesitesaswell‘nuclearsites’. (MDARS) provides‘automatedintrusiondetection’inUSDepartmentofDefense es elsewhere in the world’. numerous otherunspecifiedairports,powerplants,pipelinesandmilitaryairbas- United ArabEmirates…theRoyalPalaceinAbuDhabi,anarmouryQatarand use innumerouslocationstheMiddleEast,includingthreeairbases erator-controlled’, ‘non-lethalgunpod’. 60 guardium.htm. 59 security industriesaroundtheglobe’. al environments. They appear to be in high demand in ‘the burgeoning homeland Autonomous patrolandsentrysystemsareadvertisedforuseindiverseoperation- weapons’. It canbeequippedwithavarietyofremotelycontrolled‘lethalorlessthanlethal physical securitymissionsinamorehostileenvironment’. defined perimeteronsemitounstructuredterraininsupportofforceprotection/ grated RapidResponseEquipment(FIRRE)isintendedforoperations‘outsidea tion intothe‘tacticalunstructuredenvironment’.TheUSArmy’sFamilyofInte- semi-structured facilities’,follow-onprojectsaimtoexpandthescopeofapplica- and itsterritories’. the battlespaceorforhomelandsecurityandborderpatroleffortsacrossUS supply depots, andportfacilities’,as well as to support ‘force protectioneffortsin ‘random patrols around inventory-sensitivewarehouses, air-fields, ammunition detection andincidentassessmentsubsystems. closed securityareawhoseboundariesarepre-programmed.Itincludesmotion capable ofautonomousmovementwithin‘adefinedareaoperation’or‘anen- camps inexpeditionary/peacekeepingoperations’. transportable andcan,forexample,bedeployedtoprotecttemporaryforward/base tions’ www.public.navy.mil/spawar/Pacific/Robotics/Pages/MDARS.aspx. have ‘Enguard! J. SPAWAR General reportedly (D. Cook, Hambling, ‘Israel’s 59 Dynamics Systems Similarly,GDSR’sMobileDetectionAssessmentandResponseSystem Introducing announced ‘Armed Video Center Robotic 66 WhereastheMDARSwasinitiallyconceivedfor‘structured/ the Russian Game ‘that Pacific, Guardium Systems, mobile Killing Robocops 64 ‘Mobile Sentry Tech exists as a mobile station that is ‘easily Technology’, ‘MDARS’ robots UGV’, Detection to Defense Update, would Defend 63 Brochure, TheSuperaEgisIIisreportedly‘inactive 62 The Electronic Intifada, be Assessment Missile standing TechyLib, 61 Bases’, TheMDARSisfittedwithan‘op- https://defense-update.com/products/g/ 65 and guard TheMDARSisadvertisedfor The https://www.techylib.com/en/view/ New Scientist, Response 60 Russian over Theroboticplatformis 67 13 five July System Strategic ballistic 23 2010, April (MDARS)’, missile https://electro Missile 2014, installa- https:// http:// Forces in time,thecontrolofthosewhoemploythem, harmful effects of weapons must not be unforeseeable or escape, either in spaceor ganized violence.Thelatteris,forinstance,reflectedintheIHLrequirementthat ing requirementforthemoralacceptability,politicallegitimacyandlegalityofor- That weaponsandtheirconsequencesarecontrolledcontrollableisalong-stand- to exercisecontroloverthenaturalandhumanworld,aswelldeployviolence. in lawenforcementoperations. use offorcedemandingthatstateagents‘placethefloweventsundertheircontrol’ 68 can leadtounpredictabilityinitsfunctioning. it operates,anyunforeseenchangetothatenvironment,oroperationoutsideofit, iour. If an AWS functions on the basis of a model of the environment within which ers ofanAWSareinprincipleunabletopredictandcontrolcompletelyitsbehav- tered and the every with selectingandfiringattargetsmakesitmoredifficultfortheusertopredict legal dutiesandbeaccountablefortheconsequences.Forone,taskingamachine ability and,byextension,thatofthestateonwhosebehalftheyact,toperform Changes in how human beings exercise control in the use of weapons affect their security practicesandtechnologies. controllability ofunknownfuturethreatsareimportantdriversalgorithm-based As advertisementsforsentry-AWSillustrate,culturesofcontrolandabeliefinthe ‘plans andanyotherformofprescriptivespecificationpresupposecompetencies responsibility underinternationallawfortheconsequencesofarmedviolence. quirement and, at the same time, it plays a role in delimiting the boundaries of state tonomous underlying 69 A in edu/ONR_report.pdf at Are and Advisory grammed emergent behaviours). 71 Judgment, 17December 2009, §114. 70 icrc.org/customary-ihl/eng/docs/v2_rul_rule71. 72 on theResponsibility of States for Internationally Wrongful Acts (UNGARes 56/83, 12December 2001). stances . Human C Time California by This ICRC, K. Art DSB, European Nature Abney, particular target,theprecisemomentandlocationwhereviolenceisadminis- precluding of 14, is Opinion, Autonomous Weapon Systems, Summer Study onAutonomy, them system War; causal reflected, 1956 Polytechnic Indiscriminate’, Autonomous Military Robotics: Risk,Ethics,andDesign, Court to International concrete environment within which violent effects are produced. Us- ICRC linkages’ 8 that wrongfulness. ontrol and do’ July (describing for of Draft would as 1996, example, Human State a so ‘sorely Rules ICRC, as Court ‘sense §35. University, the to Rights See, in for predict Customary outdated’ For of ‘common the provisions e.g., the 70 Justice more supra the Use of F Theexerciseofcontrolbystateagentsisalegalre- (ECtHR), state Expert Limitation San ‘geopolitical Arts 68 sources, misconception belief, IHL (ICJ), fn of Humanbeingshavelongusedtechnologies Luis 8 on Meeting and the 36, Mikayil Mammadov vAzerbaijan, Database the Obispo, Legality of theThreat or Use of Nuclear Weapons, given of pp world 23(1), see events’, attribution the 80–81 Report 69 ‘Practice 72 the Dangers 2001 (ICRC aswellinIHRLstandardsonthe and December that AsSuchmanandWeberexplain, orce complexity with (envisaging (2016), build robots International CIHL of Relating a incurred conduct Ethics view an Database), 2008, supra will internal of to the + to do by programs ‘safeguard p Emerging to fn Law Rule only development 8, the a 6, https://ihl-databases. representation http://ethics.calpoly. state Commission p 71 what Civilian App 8. – and Sciences P. U.S. and Weapons we no Lin, Population potentially interests’). in have of 4762/05, G. circum- Articles an 71 of Group Bekey That pro- au- the

Envisioning Autonomous Weapon Systems 19 DEFENDING THE BOUNDARY 20 “Right Value Goes Learning intelligence.org/files/obsolete/ValueLearningProblem.pdf 75 rationality thatstrives for AIsystems thatoperate autonomously inopenandcomplex environments). 77 and context-dependent’, p1). 76 read/ai-deep-learning-ethics-right-to-explanation. 85. legal preceptsisdeeplycontentious. equately encodedsoastoensurethattheconsequencesofAWSusecomplywith interactions ofsystemcomponentswitheachotherandtheirenvironment, example, areusedwherecollectivebehaviourcanemergefromtheself-organized is compoundedifcomplexdistributedsystems,basedonswarmintelligencefor today, (the ‘realworld’).Howmachines‘makedecisions’andlearnisnotwellunderstood AWS aremadetolearnsoasfunctionindynamic,unstructuredenvironments 73 itations onpermissible‘human-machineconfigurations’. the lawpresupposesameasureofhumanagencyinuseforcethatplaceslim- In additiontoconstraintsbasedonethicalandotherimperatives,compliancewith and in situ forms of interaction that they can never fully specify’. a keyquestioninthedebateonAWS. man controlisnolongerexercisedinameaningfulorappropriatemanneris,thus, who riskbeingadverselyaffectedbyalgorithm-based decisions. Atwhat point hu- C. 423 74 discrepancies between actualsystem performance andoperator expectations). Meeting operations’ 1462; 78 bit.ly/1Nx8eLb. supra com/files.cnas.org/documents/CNAS_TheComingSwarm_Scharre.pdf; the social being capacities draws autonomous agency asrelational). Kress ability DSB, M. L. Suchman See, Suchman (explaining B. Wrong, P. fn Learning Dorigo theory an Suchman attention to Goodman 75 Scharre, (eds), of 36, Summer Study onAutonomy, e.g., (WHI Explanation”’, andtheunderlyingpremisethatarepresentationofourworldcanbead- intrinsic of for Experts as pp humans and We has ‘The and action and Autonomous Weapons Systems: Law, Ethics, Policy, ‘many 2016), 83–87 Problem’, how and Robotics ontheBattlefieldPart II:The Coming Swarm , to Won’t M. and effectively Weber, capacity Weber, Concept the Birattari, on J. the that to autonomous New (noting S. Lethal Weber, relations Be Paper control vision Flaxman, arise Machine ‘Human-Machine ‘Human-Machine Able York, of of ‘Swarm challenged Autonomous that “Meaningful an presented ‘Human-Machine of out directly’ to between 28 emergent entity, Ask ‘the system ‘European Intelligence of Intelligence’, June particular It hundreds-to-thousands supra (p Why’, the whether at 2016, human 86)). behaviors Weapons Human Autonomies’, behaviour Autonomies’, 2016 premise Union fn 76 Motherboard, Research 77 See Autonomies’, https://arxiv.org/abs/1606.08813; Thisisamajorconcern,notleastforthose 2Scholarpedia socio-technical 36,

beings ICML human Control”’, also Regulations p Systems will that has 18 Workshop Weber, and supra Institute, change supra (recognizing or (‘Human become autonomy 6 machine, Working machines, in (LAWS), July 9 fn of fn systems.’ ‘Black-Boxing N. (2007) on over 73, individual on 73, Technical 2016, Bhuta, part Cambridge goals Algorithmic CNAS, DSB, pp Human can Paper p Geneva, shifting the time 1462, of supporting 78 85–86. https://motherboard.vice.com/ are 78 The be S. the Summer Study onAutonomy, potential 2014, (arguing Legalobligationsare due platforms Beck, Report submitted adequately doi: Interpretability complex, leitmotif Organisms’, concept University the 13–17 See to Decision-Making https://s3.amazonaws. 10.4249/scholarpedia. R. J. 73 focus an learning’, no for also that Geiss, The challenge Pearson, April would understanding of of 2015-4, by culturally ‘surprise N. understood ‘contemporary instead Press, a ‘configuration’ supra Austria, 2015, new H.-Y. Soares, in be leading ‘When Machine 74 2016, techno- https:// fn beyond Liu http:// during to laden, orif and 23, CCW ‘The and the as AI to of p p a D. 82 they can perform more ethically thanhumansoldiers are capable of’). p rules’, ed 2016, judgment’ guing Reactive Robot Architecture , (eds), Robotik undGesetzgebung, NomosVerlagsgesellschaft, 2013, 239–260. 582.443+01+DOC+PDF+V0//EN different Reasons’, The 83 Responsibility’, 2 Politica &Società (2014) 265–292, doi:10.4476/77103. in theirresult, implies Epistemology, HermeneuticsandJurisprudence, normative – the mous proportionality, orientation facts, Need for Meaningful HumanControl, in and although pressed able legal judgements(asopposedtoalgorithmiccalculations). 79 addressed tohumanbeings. Policy, an entityaccountableforharmdoneorinfringementsofthelaw. devoid ofintentionality. centric perspective,anAWSisartefact–objectmadeforacertainpurpose, can bemisleading, Weapons Experts obligations law 81 Zawieska+SS.pdf. ch/80256EDD006B8954/(httpAssets)/369A75B470A5A368C1257E290041E20B/$file/23+Karolina+ Rights that ‘Do 80 assumption’ decision to of IHRListhat‘the use lethalforce must bereasonable andtaken by ahuman’). ‘Wishful in 7, zu Purves, any any autonomous http://www.cc.gatech.edu/ai/robot-lab/online-publications/formalizationv35.pdf construction In discussion Robots of See, On the See, ‘a Vor- to On and weapons that contrast, and event.’ p and Cambridge manifest given war many that the on to Advisory the Mnemonics in 12, view, perhaps und e.g., 18 R. Systems if on origin ‘even that imagine the has Lethal on particular, ascription rules it Equal complex Jenkins pitfalls Ethical Theory andMoral Practice http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//NONSGML+COMPARL+PE- situation’, the See that Nachteilen is they the E. Use on see see to ‘will even for presumption robots’). of a recognized that basis Lieblich on also, Committee be Autonomous the sometimes sophisticated weapons University computers Draft a Humans? of are instance). them R. ‘could not of and socially and if viewed conducting gives enough Autonomous locus of C. US ‘Towards robot that anthropomorphization Unlawful’, be 80 as Report, B. moral Arkin, der applying On Autonomous DoD, thisauthorespousestheviewthatanAWSdoesnotmake and not able it is J. ‘human of themselves; that acceptable with the ‘decisions’ Press, meaning Einführung Strawser, Anthropomorphic Suggestions on of required algorithmic unwittingly, apply Technical be agency 82 E. Law of War Manual, agency Governing Lethal Behavior: Embedding EthicsinaHybridDeliberative/ Weapons to European robot (contemplating human a Itcannotmeaningfullybetreatedasaholderofrightsor Issues ‘status attacks fact scepticism made Benvenisti, in “Compliance-Based” standards, independently Weapons 79 moral 2016, rules no N. Althoughtheuseofanthropomorphizinglanguage and is (see, intersects Killing to for ‘Autonomous of Bhuta meaning are agency’ 97 for question’, Report not of einer ... (e.g. machines, Systems judgment Parliament, calculations pp made. law to Public determining AIV course, the extensionally impose e.g., the if the Systems’, 252–253 which ‘The Machines’, applying are et in calculating elektronischen / 4 right Springer electronic is Terminology GIT-GVU-07-11, P. kind no with effect of Consider, International the al apply (2015) reflected socially Nerhot (LAWS), see Obligation an an supra 26 obligations is per (eds), see Machines, reasons’, of Committee context inanimate the not can indeterminate (pointing supra S. Approach CAVV, that law IHL thing direct se P. 136 Science 851–872, indistinguishable Beck, ‘codifiability codifiable, fn (ed), an constructed personhood be Autonomous Weapons Systems: Law, Ethics, M. e.g., do Geneva, for is in an 9, fn AISB Quarterly October artillery understood that in Asaro, equivalent Person’, of to not a rendering participation Law, at Moral AWS Law, Interpretation andReality: Essays in s ‘Über 22, Advisory on range Georgia object out LAWS’, + on AWS, 6.5.9.3, Exercise to least aim is Business doi: persons. fn Legal LAWS’, 13–17 Autonomous Weapon Systems: The would that capable ‘Determinism, i.e. legal Judgment, Sinn projectile 2015, 53, to of debate’. in and the for see could 10.1007/s10677-015-9563-y it Presentation, to Council them Institute predict p while E. IHL Affairs, 362 as cannot Discretion: April norm ‘at und next from 330, 81 apply supra that N. Hilgendorf the Media p in an These (2013) of provisions); Fromthisanthropo- not least 26, (stating Sharkey ‘morally hostilities it For Unsinn interpretive replicating 10 2015, ‘application trajectory), any in according and human the 2015/2103(INL), on assume is be the fn asserting Dordrecht, of application 83 years’. Machine the a rules possible (‘I International 10, 14–22; captured fact right Acting brief law Technology, Why http://www.unog. that am and most von CCW and deficient’). p moral or do an Heyns, 3 or are convinced The legal overview, to human ‘an Statusfragen J. , Autonomous one that for act K. reality L. Agency, of “obligation” and §16 sophisticat jus inbello not Meeting 1990, which frequently, P. by statement to Zawieska, judgment Suchman, unspoken law’. the involving outcome is Günther autono- ‘Human specific accept impose a (noting 31 Affairs ‘hard- moral 2007, list has Right For p that ‘The May This and (ar see 2). of of a a - - -

Envisioning Autonomous Weapon Systems 21 Envisioning Autonomous Weapon Systems 22 88 in USArmy FieldManualsonLandmines,Booby Traps andIEDs’, 46Political Geography (2015) 41–53. Remote Warfare, Responsibility 87 Prohibitions orRestrictions onthe Use of Mines,Booby-Traps andOtherDevices (CCW AmPII). 86 sional autonomy riskassessment scores’ (M.Dickow etal,First Steps, supra fn28). M. ‘levels supra Research Personnel MinesandontheirDestruction (APMBT). Robot human indicators’ 85 through technicalmeasures,andclearingthemafteraspecifiedlapseoftime. of mines,fencingandperimeter-markingminedareas,renderingminesinoperable 1996 CCWAmendedProtocolIIacceptresponsibilityforrecordingthelocation a measureofcontrolovertheviolenteffectslandmines,StatesPartiesto ized inthecomprehensivelegalbanonanti-personnelminesadopted in 1997. poral restrictions and procedural requirements – anormative development formal- and livelihoodscannotbeadequatelycontrolledthroughtheProtocol’sspatio-tem- have concludedthattheadverse impact ofanti-personnel landmines onhumanlives should bedrawncancontroversialandchangeovertime.Anumberofstates The exampleoflandminesalsodemonstrates,however,thatwheretheseboundaries force isdirectedatandthatmaybeincidentallyaffected. duced, andwhoorwhatisharmed–bothinrespectofpersonsobjectsthat which forceisused,whenandwhereappliedorviolenteffectsarepro- restrictions andrequirementspertainingtothereasonswhymannerin In presentpractice, human controltakestheform of technicaland normative in specifiedspatio-temporalboundariesandaccordingtopredefinedparameters. be absolute.Today,itisacceptedthatweaponsproduceeffects‘ontheirown’with- This exampleillustratesthathumancontroloverweaponeffectsdoesnotneedto no 3ontheAfrican Charter onHumanandPeoples’ Rights:The Rightto Life (Article4), 84 der toensurethathumanbeingsretainmeaningfulcontrolintheuseofforce. Legal normsareonewaytoformallylimithuman–machineconfigurationsinor- machine Bolton, As Aspects Art On Arts of Weapons’, fn control. reflected 1, the human autonomy 6), Paper 3(2), ‘From 1997 and evolving of ‘dynamic for ‘space-time The 5(2)(a)-(b) how Minefields supervisory Convention 166, in 2 Edward Computer-Guided Politica &Società African use in human available spatial the diligence’ of Elgar selection such to and Commission partitions’ control’ on control Minespace: dimension Press, new at the 10(1) (P. SSRN, Lethal of Prohibition technologies (N. is (2014) Margulies, forthcoming. and (Article human exercised Sharkey, on An https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2734900), of Force Technical Human Archeology killing 305–324, targets 36, in of ‘Making ‘Towards over should Structuring Debate on Autonomous Weapons Systems, Armed the with Roger and Annex, or weapons doi: Use, of Peoples’ ‘remote the follow Autonomous Williams Conflicts’, a the 10.4476/77105), Principle Stockpiling, use 1996 Changing the can of Rights and University force Amended established in be for J. autonomously Weapons Architecture (AfCommHPR), conceptualized Ohlin the Production should 85 Human Forexample,toretain or School (ed), Protocol perhaps rules be Accountable: Research Handbookon subject Supervisory of and of violent of Autonomous 2015, Law, General Comment in even international II Transfer to terms to Legal s devices’, the ‘multidimen- meaningful F, Command Control §35: of CCW of 88 Studies ‘proxy Killing 86 Anti- law.’ ‘Any

see on of 84 87

‘derive fromtheinherentdignityofhumanperson’ 89 use offorce. er situationswouldhavetobeassessedprincipallyagainstIHRLstandardsonthe frame ofreferencefortheuseAWSinsomesituations,whereastheiroth- If ourexperiencewitharmeddronesisanyguide,IHLwouldbethedominantlegal the securingofapowerplant. or border international an of patrolling the including activities, other are being advertised for use in military combat situations as well as for Yet, as the examples given above illustrate, autonomous sentry systems The legal debate on AWS has thus far focused on compliance with IHL. a lawfularrestortopreventtheescapeofpersonlawfullydetained, or‘inaction law enforcementaim, such as to defend a person from unlawful violence,toeffect sable, unavoidable)andstrictlyproportionatetotheachievement ofalegitimate on HumanRights(ECHR),theuseofforcemustbeabsolutely necessary(indispen- over individuals,objectsorterritory. store publicsecurity,lawandorderortootherwiseexerciseits authorityorpower any otherterritorialorextraterritorialmeasuretakenbyastatetomaintainre- law enforcementoperations,suchasthedispersalofariot,bordergovernanceor ment Officials (BPUFF), provide the normative framework for the use of force in notably the 1990 BasicPrincipleson the Useof Force and Firearmsby Law Enforce- security andprivacy,fromdiscrimination.IHRLstandardsontheuseofforce, from arbitrarydeprivationoflife,interferencewiththerightstoliberty, protection of humandignity is the common aimof IHRL and IHL. victims of armed violence, including by placing constraints on the use of force. The Albeit indifferentways,bothIHLandIHRLaimtosafeguardhumanityprotect and difficulttodetermine. N.Melzer, Targeted KillinginInternational Law, Oxford University Press, 2008, p90. 92 and Political Rights(ICCPR). 91 Heyns, ‘HumanRightsandthe Use of Autonomous Weapons Systems’, supra fn22,367. 90 Law of International Law ‘Missile ‘Lethal 2(2014) 317–339, doi: 10.1093/jcsl/kru001. Art The Autonomous Defence 1, jus adbellum 1948 IHL 89 Universal Shields: Whichsetofstandardsappliesinagivensituationcanbecontested Weapons 1 implications and (2015) Automated 4. The Applicable La Declaration 37–52, and of Jus AWS and IHRL of http://scholarlycommons.law.case.edu/jil/vol47/iss1/7; ad Human Anticipatory are Bellum beyond 92 the Use ofF InthecontextofEuropeanConvention Rights; Proportionality’, the Self-Defence?’, S Preamble, scope tand of this 1966 47 19 paper. Journal of Conflict and Security Case Western Reserve Journal International 91 andprotecteveryone ard On this 90 topic, Human rights Covenant orce s on see, H. F. w on M. Grimal, Roff, Civil :

DEFENDING THE BOUNDARY 23 DEFENDING THE BOUNDARY 24 tional Rights, The Role of theHumanRightsCouncil, (BPUFF). 94 App no18984/91, Judgment,27 September 1995,§148. the suspectedoffender); force isused,tominimizethehazarditposeshumanlife(ofbystandersand 93 to securetherightlifewithintheirjurisdiction. ing torefrainfromtheunlawfultakingoflife,butalsohavingtakepositivesteps Human rightsprotectionoflifeandphysicalintegritynotonlyentailsstateshav- lawfully taken for the purpose of quelling a riot or insurrection’. 100 Principle2,BPUFF;ECtHR, Güleç vTurkey, Appno21593/93, Judgment,27 July1998,§71. 99 Principle5(b),BPUFF;ECtHR, Ergi vTurkey Judgment,28 July1998,§80., Appno23818/94, 98 57948/00 and57949/00, Judgment, 24 February 2005, §171. 95 it proportionate) andallother means are insufficientto achieve thatobjective (makingitnecessary)’). use offorcetosecureaccountability. effective officialinvestigationwhenindividualshavebeenkilledasaresultofthe extent possible,recoursetolethalforceandincidentallossoflife, general securityset-upandspecificoperationssoastominimize,thegreatest against animminent(orgrave)threatofdeathseriousinjury). potentially lethalforcemaynotbeusedexceptasalastresortinordertoprotect suit oftheseaims,however(andacknowledgingthatthereisdebateonthispoint), weapons to allow for a differentiated use of force; respect of some rights. take measuresthatderogatefromtheirobligationsunderhumanrightstreatiesin in timeofwarora‘publicemergencythreateningthelifenation’,statesmay life, isnotabsolute.TotakeaccountofthedifficultiessecuringrespectforIHRL However, thelegalprotectionofhumanrights,includingagainstdeprivation a humanrightsviolation. assessment ofthesurroundingcircumstances’; including bymakingtheuseofpotentiallylethalforcedependentona‘careful ulatory frameworkandproceduresthatstrictlycontrollimittheuseofforce, tions’ assumedbystatesarethedutytoputinplaceanappropriatelegalandreg- derogation. Yet,deathsresultingfromlawful‘actsofwar’arenot‘arbitrary’depri- 97 ECtHR, Nachova etalvBulgaria , Appnos43577/98 and43579/98, Judgment,6July2005, §96. 96 Rights Treaties’, PhDThesis no949,Graduate Institute of International Studies, Geneva, 2012. 101 United Kingdom , Appno55721/07, Grand Chamber, Judgment,7July2011, §164. international law andare not discriminatory ICCPR). (Art4, the

situation, McCann etal, Art For Principle States ECtHR, lethal 2016, 2(2), a For conceptualization may use McKerr v The United Kingdom, European a pp and 9, brief of only 1990 11–14. supra provided force discussion, derogate Basic Convention See fn … 93, is also that 101 of Principles §194; prohibited 98 Other rights, including the right to life, are not subject to from positive AfCommHPR, see providesecurityforceswithadequateequipmentand such on ECtHR, S. their Human measures Geneva Maslen, on obligations, unless obligations the Isayeva, Yusupova andBazayeva vRussia, App Rights 100 General Comment no3, Use Academy Use of Force inLaw Enforcement andtheRightto Life: it no are Thefailuretofulfillapositiveobligationis is of 28883/95, strictly see not (ECHR). to Force the S. inconsistent of Krähenmann, 96 unavoidable International and extent ECtHR, toplan,organizeandcontrolthe 4 99 Firearms May 95 and conduct some form of an Amongthe‘positiveobliga- strictly McCann etalvThe United Kingdom, with 2001, supra in ‘Positive Humanitarian by their required order §111; fn Law 84, other to ECtHR, Obligations Enforcement 94 s by protect 93 App E, 97 obligations Even in pur- the §27) Law and,ifsuch Al-Skeini v The nos exigencies life and (‘the 57947/00, in (making Officials Human Human inten- under of prohibit inabstractotothatwhichisactuallyrequiredconcreto’. sessment thatservestolimitmilitaryactions‘fromwhichpositiveIHLdoesnot ty andconsiderationsorprinciplesofhumanity. the conductofhostilities,whichseektostrikeabalancebetweenmilitarynecessi- to avoid and, at any rate, minimize civilian harm. proportionate attacks,anditrequiresthatallfeasibleprecautionsaretakeninattack attacks on civilians and civilian objects, and the launching of indiscriminate or dis- flexible enoughtotakeaccountofpracticaldifficultiesthatstatesmayencounter applicability of IHL does not eclipse states’ obligations under IHRL. The latter is circumstances’. necessary fortheaccomplishmentofalegitimatemilitarypurposeinprevailing provisions ofIHL,nomoredeath,injury,ordestructionbecausedthanisactually ‘considerations ofhumanityrequirethat,withintheparameterssetbyspecific degree of force that may be used in any given attack cannot be determined 102 to bereconciled. of armedconflict.Consequently,IHLandIHRLcanapplyconcurrentlyneed It iswidelyrecognizedtodaythathumanrightsprotectiondoesnotceaseintimes vations oflife. manner thattakesaccountofIHLrules. erned bylawenforcementstandards,butIHRListobeinterpretedandappliedina authority or power that does IHL prohibitstheuseofcertainmeansandmethodswarfare, Elementary Considerations of Humanity, 103 penalty, provided for in,e.g., Art6(2),ICCPR. https://ssrn.com/abstract=2089115. On I. trol, 104 (Riga), available atSSRN, https://ssrn.com/abstract=2912002. jury Consideration ‘Human 105 70 https://ihl-databases.icrc.org/customary-ihl/eng/docs/v1_rul). and71, G. Humanitarian Law 106 non-international armed conflicts (NIACs) (ibid,Rules 1, 11, 14and15). 108 Melzer, Targeted Killingin International Law , supra fn92, p297.107 0990.pdf Human Rights’, 37 Israel Yearbook onHumanRights(2007) 115–161. 109 Situations of Armed Conflict’, 40Israel Law Review 2(2007) 311. Kerr Gaggioli connections see or N. These On Notably This (eds), For C. unnecessary Melzer, P. Dignity Droege, the . a M. is and Robot Law, rules discussion Asaro, explicit normative weapons Of Interpretive Guidance ontheNotion of Direct Participation inHostilities UnderInternational between R. Humanity”?’, in are ‘The 102 (DPH 106 Kolb, ‘Jus suffering, an Intimesofarmedconflict,suchactsaregovernedbyIHLruleson in 108 of Inthissense,military necessity demands acontext-dependentas- Interplay Guidance), Edward Nascendi: on Age Art that part concept Insituationsofarmedconflict,anyexercisebystatestheir ‘A the how Right 15(2), are of of including notions 2017, Elgar customary Autonomous by IHL of Between Robotic to ECHR. ICRC, nature ‘humanity’ On European and Life not amount to the conduct of hostilities, remains gov- Publishing, of AWS, weapons 2009, ANU Another humanity in IHRL Weapons IHL indiscriminate International Armed the College Society p Weapons: and and can 77, 2016, Martens that exception 109 https://www.icrc.org/eng/assets/files/other/icrc-002- apply and and its Conflict? be of Andeveninsituationsofhostilities,the of render pp relationship applied the Law human International clause in Humanitarian Are or 267–386, Martens both that Research are The death 103 we 105 jointly and dignity

judicial international are Contribution Whereas the precise kindand in to doi: inevitable Clause’, the of Danger Law the Paper in 10.4337/9781783476732.00024. in a notion Law executions nature a ‘Martens relation (ESIL) in complementary no and of (ICRC R. armed of of 12–19, 2016 Calo, to 104 107 Losing meaningful the Human to clause’, cause aswelldirect CIHL based AWS, conflicts A. Annual European available

M. an Database, superfluous Rights

Froomkin see on see “Elementary fashion, human Conference (IACs) the M. O. ex ante, at Court Law

Ulgen, Zagor, death SSRN, Rules

con- and and see in- of in

The Applicable Law: IHL and IHRL Standards on the Use of Force 25 DEFENDING THE BOUNDARY 26 112 org/eng/assets/files/publications/icrc-002-4171.pdf. in caseof“absolutenecessity”foralegitimatepurpose.’ attacks against certain categories of persons in absolute terms, that is to say, even is “absolutelynecessary”foralegitimatepurpose’,whereas‘IHL…prohibitsdirect keeping inmindthatIHRL‘allowstheuseoflethalforceagainstanyonewherethis its regulationoftheuseforcethanalawenforcementone.However,itisworth of lifebecauseaconducthostilitiesframeworkisgenerallymorepermissivein ity ofIHLcanchangeandinsomecasessignificantlydiminishthelegalprotection 111 developed for situationsof armed conflicts’). G. Enforcement Paradigms, 110 tion oflethalforceabroad,inplaceswherestateagentsarenot physicallypresent. ly. Armeddrones(possiblecomponentsofanAWS)alreadyallow fortheapplica- an AWSandtheplacewhereviolenceisexperienced,including extraterritorial- Autonomy inweaponsystemsallowsforincreasingdistance betweentheuserof ability toassessthelegalityofAWSuseinlightexistinglaw. there isnowidelysharedagreementaboutwhatlawapplies,thenthislimitsour control andintentplayapivotalroleinthesecontroversies;second,becauseif These controversies are instructive for the debate on AWS. First, because issues of • • • three controversiesarebrieflyexposedbelow: to ongoinglegaldebate.Althoughthisdebatecannotbedefinitelysettledhere, cantly affectsthescopeforlawfuluseofAWS.Thesequestionsarealsosubject What legalrulesapplytotheuseofforce,andhowIHRLIHLinteract,signifi- mative contentoftheseparadigmsdiffersinimportantrespects. crucial impactonthehumanitarianconsequencesofanoperationsincenor- The distinctionbetweenlawenforcementandconductofhostilitiescanhavea burden onstates. in exercisingtheirauthoritysuchsituationssoasnottoimposeanimpossible IHL provides stronger protection against thedestruction of civilianproperty thanIHRL. Extra- A . Human RightsTreaty Obligations Abroad Gaggioli, For Melzer, Ibid, tute useofa‘meanswarfare’,governedbytheconducthostilitiesparadigm? During anarmedconflict,whendoestheuseofforcebymeansAWSconsti- Can anAWStriggerIAC‘onitsown’,bringingIHLintooperation? dictional linkonitsownfortheextraterritorialapplicationofIHRLtreaties? Does theuseofforceabroadbymeansanAWSestablishasufficientjuris- a 129 discussion Territorial C The Use of Force inArmed Conflicts: Interplay Between theConduct of Hostilities andLaw Targeted KillinginInternational Law , (noting of 110 that the Report, several distinguishing Expert aspects ontrol Meeting, features of positive January supra of and obligations fn 2012, differences 92, p ICRC, 384. ‘can November Likewise, between 112 be flexibly the in 2013, : A 111 certain applied, two Theapplicabil- WS https://www.icrc. ‘paradigms’, circumstances, and adapted

and see 115 default/files/reports/arms0415_ForUpload_0.pdf). the Gap:The Lackof Accountability for KillerRobots, rially. settled. acts thatproduceeffectsthere,whetherlawfullyorunlawfully,isnotdefinitely human rightstreatiesapplywhenastateperformsactsoutsideofitsterritoryor accepted that the notion of jurisdiction is primarily territorial. To what extent one 116 ECtHR, Sargsyan vAzerbaijan, Appno40167/06, Grand Chamber, Judgment,16June2015, §§126–131). the remedies tovictimsintheformoftreaty-basedhumanrightsmechanisms. rights Age”’, 29 September 1999,AnnualReport 1999,§37. jurisdiction 114 Melzer, Targeted KillinginInternational Law, supra fn92, p211. 113 one Under humanrightstreaties,statespartiesassumeobligationstosecureevery- ventional righttolife’, under customaryIHRL–bindingonallstates‘virtuallycoincidewiththecon- Although the conditions and modalities governing the lawful use of lethal force of theirobligationswhentheyaffectindividualsabroad. foreign stateagents’custody). the ‘controlandauthority’ofaforeignstate(suchaswhenpersonistakeninto tory ofanotherstatebringsanindividualintothe‘physicalpowerandcontrol’or outside itsnationalterritory, where astate,throughmilitaryaction,exerciseseffectiveoverallcontrolofanarea tionally admittedtheextraterritorialapplicationofECHRincircumstances Judgment, v Uruguay, Judgment, 1 Commission exceptional tion 118 ECtHR, Loizidou vTurkey, Appno15318/89, Grand Chamber, Judgment,18December 1996,§56. 117 war-remains-inside-the-court-room-jurisdiction-under-echr/. under in (2012) connection courts treaty that Note M. However, Al-Skeini, within their jurisdiction See, ECHR’, http://www.ohchr.org/Documents/Issues/Privacy/United%20States.pdf Milanovic, treaty law has practical relevance, not least in terms of the availability of 115 the 121–139, that e.g., to of 29 Comm 12 can circumstances The trend is towardassertingthat states remain bound by at least some on another, territorial UK HumanRightsBlog, the with the ‘United May March victims supra also Human user ‘Al-Skeini wording no an doi: 2005, arise and armed 2010, fn R.12/52, States state, of state Rights 10.1093/ejil/chr102; 100, when AWS that limit §91; of and §67; as exercises conflict 113 Response provisions §§136–137; (IACommHR), some UN its use well a ascertainingtheextraterritorialapplicabilityofhuman Al-Jedda ECtHR, state UN responsibility the rights and freedoms guaranteed by the treaty. It is doc abroad 117 states, as or 118 11 Human jurisdiction supp to has military orwheretheexerciseofpublicpowersonterri- supra-national Medvedyev etalvFrance, September on in the ECtHR, lost can the the Strasbourg’, D. Coard etal vUnited States, no OHCHR Rights Hart, US occupation. scope effective be to 40 or Öcalan vTurkey, respect for expected (A/36/40) 2016, ‘War Committee competence, Questionnaire of example, April human application 23 control and Remains https://ukhumanrightsblog.com/2016/09/11/ 2015, In European Journalof International Law) (EJIL to such rights ensure at reject face over (HRCttee), Sergio Euben Lopez Burgos 176 pp which cases, on App Inside of App formidable mechanisms 27–29, that part (1981), respect human the 116 Report no bears no there the of “Right the TheECtHRhasexcep- 46221/99, https://www.hrw.org/sites/ its 3394/03, §12(1)–(3); for ICCPR rights . Court the is Questions no territory challenges human to a (HRW 109/99, burden rebuttable Privacy applies treaties Room: Grand Grand rights and as Inter-American pertaining to in may Case extraterrito- in differs Jurisdiction IHRC, the show presump- accessing Chamber, Chamber, 114 (see, happen 10.951, Digital Mind from that e.g., to

The Applicable Law: IHL and IHRL Standards on the Use of Force 27 DEFENDING THE BOUNDARY 28 at control contracting swiftly for Defence &Ors through thecheckpoint’. diction ‘forthepurposeofassertingauthorityandcontroloverpersonspassing assumed responsibilityforprovidingsecurityinthatareaandexerciseditsjuris- south-eastern IraqoccurredwithinthejurisdictionofNetherlandsasithad Netherlands, theECtHRfoundthatadeathoccurringatvehiclecheckpointin cised the public powers necessary to maintain security. the courseof...securityoperations’takingplaceinanareawhereUKexer- United Kingdom (UK) ‘exercised authority and control over individuals killed in abroad donotexercisetherequiredcontroloverarea,norphysical the purviewofhumanrightstreatiesonbasisthatstatesusingarmeddrones reasoning adoptedinAl-Skeini would, however, exclude drone operations from agents ofthestateis,thus,notanecessarycondition.AccordingtoMilanovic, of thevictims. der areafellwithinthejurisdictionofTurkeyirrespectivepreciselocation deaths causedbyfiredischargedfromTurkishhelicoptersintheTurkey-Iranbor- conducting the airstrikes. abroad did not bring the affected people within the jurisdiction of the NATO forces al vBelgium et al,theECtHR controversially held that bombardmentfromthe air render 124 §152. 2014, 119 the checkpointwasnotmannedbyDutchsoldiers. Jaloud, the ECtHR considered that the Netherlands exercised jurisdiction although sensor orweaponsrangetoestablishajurisdictionallink?Itisnoteworthythatin amount tosufficientcontrolofanareaand/oroverindividualswithintheAWS’ In lightoftheabove,wouldextraterritorialuseforcebymeansanAWS own, asufficientlinktoextendjurisdictionextraterritorially. Controversy persists about whether theuseof force abroadconstitutes,on its 123 Al-Skeini, supra fn100, §149. 122 ECtHR, Pad etalvTurkey , Appno60167/00, Decision on Admissibility, 121 28June2007, §§54-55. cross-border operations amounted to theexercise of effective control of anarea inNorthernIraq). Civ/2016/811.html 120 Court Room’, supra fn116). §117). at App jected 12 in the 1km the December no See ECtHR, ECtHR, According exercise relevant that despatching the would in addition 31821/96, England Netherlands any party Banković et alvBelgium al, Jaloud vThe Netherlands, 2001, be of time such to non-justiciable, uses effective 121 Judgment, and to (grappling [2016] the them §71. to control the InAl-SkeinivTheUnitedKingdomtheECtHRdeterminedthat physical Wales observe Dutch responsible: use The would EWCA control could of 16 Court with implicated Government, force). force 123 and November arguably 120 whereas Civ be the or of In contrast, the Court considered in a later case that authority advise, itself: 811 equated of Hart Appeal question legal App governments (09 give soldiers such 2004, points App ‘[a]lthough this no September (Civil rested authority with rise no a of did 47708/08, position §82 out whether cornering 52207/99, to Division), territorial not with a a (no denied justiciable principled Netherlands (Ibid, imply 2016), the would jurisdiction the a Grand Al-Saadoon &OrsvThe Secretary of State control Iraqi §44). Grand having a group ECHR hierarchical imply http://www.bailii.org/ew/cases/EWCA/ 124 killing’ problem security Chamber, Similarly, Chamber, military Thedirectinvolvementof of 122 of due applies control ‘that a civilians Similarly, in (Hart, nature to with forces’ a failure relationship personnel whenever sniper over ECtHR, Judgment, Decision ‘War requiring up and 119 (Jaloud, the to an InBankovićet picking Remains Issa etal v Turkey, extent prove airspace, alley-way on had and Jaloud v The such 20 an Admissibility, supra off that that wherever been element November Inside as a and Turkish civilian fn results before would there 123, the re- of a veillance combinedwithinstantweaponsdeliverythattransform theconceptof the targetselectionprocess,butcapabilitiesofcontrolofferedbypersistentsur - On theotherhand,inRosén’saccountitisnotdirecthumanintervention selected bythesystem. cess andthathumanagentsmaythereforeneitherseenorknowspecifictargets that humaninterventionispurposefullyremovedfromthetargetselectionpro- gations. WhetherthesamecanbesaidofAWSisdoubtful,however,considering of proximity’ and itisthecapability of ‘seeingandknowing’thatmaytriggerobli- drones, which involves a strong aspect of control. For Rosén, drones are a‘medium within thejurisdictionofstate,but‘proximityandvisibility’enabledby tive, itisnotthedeliberateselectionofindividualsastargetsthatbringsthem may triggertheextraterritorialapplicationof[ECHR]’. for exercisingadegreeofcontrolandauthorityoverterritoriespersonsthat ‘surveillance andcontrolcapabilitiesofdronetechnology…suggestacapability Rosén, consideringdroneattacksmoregenerally,contends,incontrast,thatthe jurisdictional link. use offorcebymeansanAWSmay,thus,not(inallcases)establishtherequired target (a)specificindividual(s). the righttolifeoftargetedperson’. sufficient factualcontroltoassumelegalresponsibilityforitsfailure“respect” ficient factualcontrolorpowertocarryoutatargetedkillingwillalsoexercise human intervention.Melzer,ontheotherhand,arguesthat‘aStateexercisingsuf- either, irrespectiveofwhethertargetsareselectedandattackedwithorwithout standpoint, theuseofanAWSwouldnotestablishrequiredjurisdictionallink nition ofnominalidentities,itisnotindividualbutgeneric. targets ispredicatedonanalgorithmicanalysisofpatternsratherthantherecog- equated with‘targetedkilling’isquestionable.Totheextentthat‘identification’of 126 Milanovic, ‘Al-Skeini andAl-Jedda inStrasbourg’, supra fn116. 125 over individualswithinthatarea,akintotakingthemintocustody. targeting them’. to killindividuallyselectedpersonswhoarenotinthephysicalcustodyofthose is limitedtotheuseoflethalforce‘withintent,premeditationanddeliberation State instant jurisdiction “effective control”?’) mostly Journal of Conflict &Security Law 1(2014) 114, 117 ‘Iflongsurveillance (asking(at 121): periods(thattoday 130 Chamayou, Drone Theory , supra fn1, p42. 129 See ibidfor anoverview. 128 Melzer, Targeted KillinginInternational Law, supra fn92, p5. 127 or person inquestion’). Melzer, also F. lie weapons Rosén, has ahead … Targeted Killingin International Law, must a ‘Extremely positive of delivery be targeted 127 determined Mostotherdefinitionsoftargetedkillingsalsorequireintentto obligation does Stealthy or not signature by to imply and actively reference 128 Incredibly WhethertargetingbymeansofanAWScanbe killings) an “protect” intense to 126 the supra combined Close: Importantly,though,Melzer’sstatement level form the fn Drones, right of 92, of with control “effective p to 139 enforcement life Control actually (noting of control”, individuals and 129 130 exercised that Theextraterritorial Fromthisperspec- Legal capability then ‘the outside Responsibility’, extent over what 125 in Fromthis its the the constitutes to territorial territory which form 19 of a

The Applicable Law: IHL and IHRL Standards on the Use of Force 29 DEFENDING THE BOUNDARY 30 and From USDrone Practices inPakistan, tual” 136 Living-Under-Drones.pdf. trol EJIL: Talk!,EJIL: Global 135 of-the-right-to-privacy/. panopticon-part-ii/ 4 is notcomplete. even iftheresultsaffectthosefoundbeyondborderswherecontroloverterritory significantly broadenedtoensurethosewhoexercisepowerbearresponsibility, Similarly, LieblichandBenvenistiobservethattheconceptofcontrolhasbeen excellence’, evenifnotalwaysacceptedassuchininternationaljurisprudence. force applications. introducing anewweapontechnologythatreduceshumancontroloverspecific must not be allowed to evade their responsibilities under human rights treaties by ritorial jurisdictionisalsocongruentwiththebroaderconsiderationthatstates 134 Ibid,p264. 133 LieblichandBenvenisti, ‘The Obligationto Exercise Discretion’, supra fn81, p263–264. 132 Ibid,122. 131 trigger awar’. One concernthatissometimesvoicedaboutAWStheycould‘accidentally responsibility underIHRLandtriggerextraterritorialhumanrightsobligations. to triggertheextraterritorialapplicabilityofhumanrightstreaties. the meresurveillanceofindividualsabroadamountsto‘virtualcontrol’sufficient fringements committedwithsecretmasssurveillanceprograms,somesuggestthat human rightsobligationsinthedigitalage.Confrontedwithmassiveprivacy- This orientationissupportedbyargumentsinfavourofareconceptualization arises inrelationtonon-internationalarmedconflicts(NIACs), atopicexplored ger anIAC‘onitsown’andtherebybringIHLintoapplication?Thequestionalso removal ofhumanagentsfromspecificforceapplications,couldsuchanAWStrig- are insufficientlyrestrictedtopreventitfromfiringatforeignsoldiers.Giventhe international borderwhosetargetingparametersorsensorandweaponsranges people within the jurisdiction of the state using the AWS. any rate, they cannot influence and may not be able to escape, must bring these that areunknowntothosewithinitssensorandweaponsrange,which,at being madethetargetofattackbyamachinefunctioningaccordingtoparameters exposing individualstosurveillancecoupledwiththethreatorperceivedriskof B. B. November [‘virtual Animus Belligerendi livelihoods, controller’). E.g., A. On Justice Peters, the A. 21 control’ Krishnan, significant 2013, Clinic November ‘Surveillance see See 136 (arguing due International 132 Considerasentry-AWSdeployedintimesofpeacetosecurean (NYU also http://www.ejiltalk.org/surveillance-without-borders-the-unlawfulness-of-the-nsa- Killer Robots: Legality andEthicality of Autonomous Weapons, Theydescribe‘theprocessoftargeting’as‘aformcontrolpar adverse to C. 2013, School mere Nyst, that without ‘[i]t impacts surveillance] http://www.ejiltalk.org/interference-based-jurisdiction-over-violations- ‘Interference-Based Human of is Borders: Law), not September of Rights : A too Living Under Drones: Death, Injury, and Trauma to Civilians armed The far-fetched WS might and Unlawfulness 2012, drones and also Conflict Jurisdiction trigger http://chrgj.org/wp-content/uploads/2012/10/ in on the the Intent to Wage War Resolution peoples’ cyber-age of the Over the human NSA Violations lives, Clinic to Panopticon, rights imagine 135 physical (Stanford Asserting extrater- of obligations the that Ashgate, and Part Right Law 134 this mental II’,

A fortiori, School) type to 2009, of EJIL: Talk!,EJIL: Privacy’, the of health 133 con- “vir 152. and

131 -

sufficient such jects 140 not necessarily reach thethreshold of anoccupation (see Droege, ‘The Interplay’, supra fn108, 332). interpretation of theIHLnotion of ‘attack’, see thenext section. to 141 de Genève, http://archive-ouverte.unige.ch/unige:75120. 304, 2015, pp218, erned exist become should is state’s that 2016, organized is see of rules International Law (Summary), 139 BE2D518CF5DE54EAC1257F7D0036B518. https://ihl-databases.icrc.org/applic/ihl/ihl.nsf/Comment.xsp?action=openDocument&documentId= Conflict’, 138 https://ssrn.com/abstract=2887130. sity forIHLofIACtoapply. Most scholarsagreethathostilitiesdonothavetobeofaspecifiedlevelinten- a unilateralandhostileuseofarmedforcemeetingtheconditions’anIAC. eration ofastate’sarmedforcesontheterritoryanotherstate‘couldconstitute The majorityviewamong legal scholarsis that anyunconsented-tomilitaryop- acts capableoftriggeringanIAC. 137 tent-related requirementswouldbemetinlightofthereducedhumaninvolve- cause physicalharmisclear.Whatlesscertainwhetherthecausalandin- That actscarriedoutbymeansofanAWScanamounttouse‘armedforce’and by RadinandCoatsinarecentarticle. cidents notentailingtheuseofarmedforcewould,thus,besufficient. by anotherstate,incursionsintostate’sterritoryorairspace,borderin- or seriousdisruptionofcriticalinfrastructures.Thesurveillancemilitaryforces they havetocausephysicalharmintheformofdeath,injuryormaterialdamage, maintains thattotriggeranIACsuchactsneedresultinviolenteffects,is, no requirementregardingthedurationorrepetitionofactsviolence,but tween states(evenifthereisnoarmedresistance). ‘hostilities’ capableoftriggeringanIACmustentailrecourseto‘armedforce’be- be hostilities that cause the organized A. ‘if another Ibid, ICRC, S. acts that are D. In A. by Wounded soldiers). https://www.justsecurity.org/33222/triggers-thresholds-non-international-armed-conflict ‘the an trigger Radin Carron, ‘protected contrast, physical Haque, suspended’). 30 IHL to fall pp of armed Parties/armed Commentary violence between bring violence 210, Temple International andComparative Law Journal(2016) of prevalent within and and a ‘L’acte ‘Triggers IAC She NIAC’). group effects, and 212, J. intense persons Committee persons’ Coats, is needs the finds or can Sick déclencheur a With 218, interpretation on At is among question purview and groups’ but amount sufficiently (‘L’acte – the what in ‘Autonomous abroad to 238. even respect under which Thresholds Armed have First on International states See stage (ICRC of between déclencheur the beyond to within d’un Geneva reached the fails to 139 the also Forces an organized, that according Use a the Commentary Inarecentin-depthstudy,Carronunderlinesthat Geneva conflit Geneva to Weapon state of ‘attack’ Roscini, its the Convention: threshold create of sovereign do Non-International Law a in jurisdiction, Force, certain scope armé using so, d’un the Conventions. 137 Conventions to in Association, then Cyber Operations, the Systems for Thefollowingdiscussionconcentrateson Field which the conflit international’, an GC of Final Report ontheMeaning of Armed Conflict in for whatever IAC states intensity a Convention sense AWS this I first the (ICRC possibly, (2016), and IHL armé and study. – application use finds Armed and 2010, According that of of before Commentary the and reason, international’, IAC Art of supra AP Thèse by it itself Note (I) Threshold 140 armed that supra p is applies Conflict’, 49, for means the I Sheconfirmsthatthereis 2 only (e.g. fn does (stating that in of to it de the AP otherwise 138, fn must a force IHL once Carron, doctorat soldiers as Amelioration 133–150, I, GC of 41, situation not control of Just Security, once §421). supra of soon an Non-International p I), be that an by trigger NIAC, 136. AWS 2nd use IAC between or an prevailing as captured no fn exercised ‘[t]he For of available against Carron IAC events exists 140, of the D. edn, (see an occupation

a of 902, force

29 different exists. violence IAC. majority the p above) 2016, at that thereby by 217) September that or 141 peacetime by Université Condition However, least intended at (arguing persons another people a On Armed that group SSRN, §241,

state view, must does view gov two the 138 re- it -

The Applicable Law: IHL and IHRL Standards on the Use of Force 31 DEFENDING THE BOUNDARY 32 remote from a state agent’s decision to use force. precisely becauseitsapplicationsofforcecanbespatially,temporallyandcausally the causallinktostate’sintentattackanotherstatemaybetooremote. detonation ofamineatlaterpointintimemaynottriggeranIACeitherbecause violent effectsarenotproducedbythatact.Sheacknowledges,however,the Carron considersthatthelayingofminesdoesnotinitselftriggeranIACbecause deployed to engage in hostilities cannot trigger an IAC ‘on its own’ (even if a state plications offorceintheuseanAWSmeansthatisnotspecifically hostilities againstanotherstate,thelackofhumancontrolexercisedoverspecificap- be aproximatecausallinkbetweenanactofviolenceandstate’sintenttoconduct 142 exploded bythepresence,proximityorcontactofapersonvehicle:mines. for instance, because they are ‘victim-activated’, such as munitions designed to be involved whoseviolenteffectsarespatiallydistributedortemporallydeferred, Establishing theanimusbelligerendiisparticularlychallengingwhenweaponsare ment intheuseofforce. a state (a prerequisite for holding it responsible for violations of international law) Many commentatorshavepointedtochallengesinattributingtheactsofanAWS or ofultraviresacts perspective, whichissharedbytheICRC,situationsthatareresultofamistake in instructionstostateagents.Althoughdebatepersistsonthispoint,from ties against another state. This so-called and there has to be a direct causal link to thestate’s ‘intent’ to engage in hostili- carried outbystateagentsorotherpersonsauthorizedtoactonthestate’sbehalf, Weapons to CCW 147 connection withautomated weapon systems, see Grimal,‘Missile Defence Shields’, supra fn89,5. for a ‘hostilities’ 145 Carron, ‘L’acte déclencheur d’unconflit arméinternational’, supra fn140, pp353, 369. 144 ICRC Commentary GCI(2016), supra fn138,§241. 143 about thetarget detection andselection parameters of theAWS. 1257CE4004BCA51/file/Marauhn_MX_Laws_SpeakingNotes_2014.pdf. unoda-web.s3-accelerate.amazonaws.com/wp-content/uploads/assets/media/35FEA015C2466A57C is that T. 148 Law 146 more persons’ inArt2(1),APMBT, supra fn88. Theory, supra fn1, p 211). exploded civilian Chengeta, addressed who targets , 30September 2015, available atSSRN, http://dx.doi.org/10.2139/ssrn.2755211. ‘the Meeting Carron, Another Although For Art or human who 2(1), by Systems an what (DPH more Accountability Gap, Autonomous Weapon Systems andModesof Responsibility inInternational the ‘L’acte of introduction, in lays relevant CCW the Experts the the agent presence, Guidance, actively a on analogy violence déclencheur mine following AmPII, Responsibility directly aspect on 143 than in Lethal proximity see supra donottriggeranIAC. the supra is is for section. mines), not directed identifiable T. course 142 d’un Autonomous fn the Marauhn, fn perfect ForanIACtobetriggered,theuseofforcehas 106, and or 86. determination conflit sentry-AWS, For of at contact Accountability p an An (the (as as a 55). ‘An armé armed discussion anti-personnel the Weapons AWS target). At of Analysis efficient international’, a what especially could conflict person of animus belligerendi tends to be expressed Although whether for 144 Systems about point of behave cause Violations would and the mine 148 stationary an an that an of If we accept that there needs to Potential (LAWS), supra not attack less ‘armed be is death’ act discussed will defined considered of For deterministically fn of in ones, International Geneva, incapacitate, 140, violence is attack’ Impact a the the more as fn sense can here, victim ‘a 1329, 13-16 to under of be triggers detailed mine directly this of Lethal likened Law’, injure (Chamayou, May p Art and jus adbellum 239. raises designed an 49 2014, participate Presentation, analysis, could Autonomous or to In IAC AP kill questions contrast, mines https:// I relates search one 146 Drone exists to see

145 147 be or in in in

government tions’, ‘combat’ or‘warfare’ are not expressly defined underIHL. 152 Carron, ‘L’acte déclencheur d’unconflit arméinternational’, supra fn140, p364. 151 org/remote-controlled-killing-in-dallas/. force isintentionallyhostile. under attackfromanotherstatemayjustifiablypresumethattheuseofarmed same time,however,Carronpointsoutthatthestatewhosepeopleorobjectscome activities, whicharedesignedtosupportonepartytheconflict byharmingan- its deploymentbearsarealriskofescalation. armed conflicttomeansandmethodsofinjuringtheenemy. ‘Hostilities’ aresometimesdescribedasthe(collective)resort bypartiestoan hostilities paradigm. within whichtheuseofforcetakesplacelimitsapplication oftheconduct use anAWSasameansofwarfarebut,atthesametime,control overthecontext framework. Asweshallsee,theexerciseofhumancontrolisrequiredinorderto tive paradigmofhostilities,ratherthanbeingassessedwithinalawenforcement detention camp, a checkpoint or a military base, would be governed by the norma- the useofasentry-AWSduringanarmedconflicttosecureperimeteraround 149 technologies. enforcement. Inaddition,thereisgrowingconvergenceofmilitaryandpolicing roadblock, caninoneinstancebepartofhostilitiesandanotherlaw tinction betweenwar-fightingandpolicing.Certainactivities,suchasenforcinga operations butalsofulfilllawenforcementtaskshascalledintoquestionthedis- The increasingexpectationonarmedforcesthattheynotonlyconductcombat around whichinternationallegalstandardsontheuseofforcearearticulated. fight against‘terrorism’or‘violentextremism’,AWSchallengetraditionalnotions in thecyberdomainoruseofarmeddronesfor‘targetedkillings’abroad Like certainothertechnologiesandpracticesofviolence,suchashostileactivities the existenceofanarmedconflictistoberesolvedinfavourpeace. the broaderconsiderationthat,asanadjunctofrighttolife,anydoubtabout The conclusionthatanAWScannot‘accidentallytriggerawar’iscongruentwith such actsremaingovernedbyIHRLstandardsontheuseofforce. a neighboringstate’sborderguard,would,thus,nottriggeranIAC.Consequently, conflict). Asentry-AWSinstalledforbordercontrolpurposesthatfiresatmembersof would ‘Evaluating 150 International Humanitarian Law (2011) 32). C to C . The Belligerent Nexus: A DPH M. As onduct Hostilities a prioriberesponsibleforallactsofanAWScarriedoutinthecoursearmed E. noted Guidance, O’Connell, the is not Use earlier,

This raisesthequestionofwhether(thatis,inwhatcircumstances) free of supra ‘Remote-Controlled Force to such escalate fn During acts 106, can its p the 151 43. use violate jusadbellum. Arab EvenifanAWScannottriggerIAC‘onitsown’, The of Killing force Spring’, notion in WS in order Dallas’, of in , C ‘hostilities’ M. to Schmitt ontrol and EJIL: Talk!,EJIL: create It is also or and a NIAC related worth 19 L. Arimatsu July (A. recalling notions Bellal 152 2016, the Intent Theycomprise‘all 149 (eds), and http://www.ejiltalk. like that L. ‘military 14 Doswald-Beck, under 150 Yearbook of Atthe IHRL, opera- a

The Applicable Law: IHL and IHRL Standards on the Use of Force 33 DEFENDING THE BOUNDARY 34 160 cyber-weapons-and-autonomous-weapons-potential-overlap-interaction-and-vulnerabilities. the-weaponization-of-increasingly-autonomous-technologies-addressing-competing-narratives-phase-ii/ Conference, 159 Schmitt,Tallinn Manual,supra Commentary fn42, onRule 41, §2,p142. 158 HPCR,Commentary onAMWManual,supra p42. Rule 1(t), §4, fn42, 157 HPCR,AMWManual,supra Rule 1(t). fn42, 156 Art 22,1907 HagueRegulations refers to ‘means of injuringtheenemy’. 161 41, pp123–124; DPHGuidance, supra fn106, p58. possible regard or damage enemy personnel or objects’ are not included. that contributetomilitaryoperations,butare‘notdesignedorusedinjure,kill tial information’toaplatformactuallyengagingtarget.Incontrast,components an attacksuchassystemcomponentsthat‘providetargetingdataandotheressen- This includesobjectsuponwhichanattackingplatformdirectlyreliestocarryout of a‘meansinjuringtheenemy’,combat’orwarfare’? 155 Ibid,p269. 154 Melzer, Targeted KillinginInternational Law, supra fn92, p276. 153 against another’. of hostilitiesitsusehastobe‘designedsupportonepartyanarmedconflict ‘belligerent nexus’.Inotherwords,foraweaponsystemtobegovernedbythelaw Implicit inthisconceptionofa‘meanswarfare’is,first,therequirement as ‘weapons,weaponsystemsorplatformsemployedforthepurposesofattack’. ‘Means ofwarfare’havebeendescribedinthecontextairandmissilewarfare, other’. methods of injuring the enemy.’ that governthe‘choiceofandusebypartiestoanarmedconflictmeans the conceptofa‘meanswarfare’istiedtoIHLnotion‘attack’. ent force isusedbymeansofanAWSbecausethebelligerentnexuscontext-depend- used to conduct a cyber attack’. instrument, mechanism,equipment,orsoftwareused,designedintendedtobe Manual describes‘cybermeansofwarfare’as including any‘cyber device, materiel, control overtheuseofforce. sect andraisesomecommonchallengestothenormativeregulationofhuman of specialinteresttothedebateonAWSascyberandautonomousweaponsinter- and fn 41, p125;GaggioliandKolb, ‘A Rightto Life inArmed Conflict?’, supra fn109, 47) . 2002, Prosecutor vKunarac, Kovač andVoković, member 161 … See, The In Melzer, that §59). to andspecificapplicationsofforcemaynotbeunderhumancontrol.Second, 153 determining violation of e.g., term ‘the the The‘lawofhostilities’accordinglyconsiststhoserulesandprinciples the However, Targeted KillinginInternational Law, UNIDIR, ‘Cyber fact crime ‘means opposing of that Weapons is whether IHL, 160 none of committed 9 the Establishingthat(design-)intentmaybechallengingwhen warfare’ party; the October perpetrator of and International an these … Autonomous as is act the 2015, used, 159 part indicators is act is 158 sufficiently 154 of a may e.g., IT-96-23&IT-96-23/1-A, combatant; Insights from discussions on cyber security are http://www.unidir.org/programmes/emerging-security-issues/ So, when does the use of an AWS constitute use Criminal or in in Weapons: be is the Art on said supra related context its 36, Tribunal to … own fn AP Potential the serve 92, I. to of conclusive victim for Art p the an the 276; 51(4), the Overlap, Appeals armed perpetrator’s ultimate is Roscini, Former a (Roscini, non-combatant; AP 157 conflict Interaction I Chamber, goal Similarly,theTallinn refers Yugoslavia Cyber Operations, Cyber Operations, official of so to a Judgment, ‘means as and military duties’ … to (ICTY) Vulnerabilities’, the amount of campaign; victim (ICTY, combat’. supra has 12 supra June to 155 had The is 156 fn a a

of ‘hostilities’. For abriefoverview of this controversy, see Roscini, Cyber Operations, supra fn 41, p181. consider, not airborne 169 Arts51(1) and57(2), API(emphasisadded) 168 attacks’ (Rule 1(c)(3)). to defence rule19), of ‘military of operations’. See ihl.nsf/Notification.xsp?action=openDocument&documentId=0A9E03F0F2EE757CC1256402003FB6D2 Routledge, 164 Commentaryfn 42, onRule 30, §§7and15,110. 165 Pursuant to Article61(7)(a) and(b)of theRome Statute (Pre-Trial §§45–48. ChamberII),9June2014, use (proportionality, distinctionandprecautionsinattack)applytoattacksonly. However, accordingtoasignificantpartoflegalscholarship,therulesontargeting they ‘enjoygeneralprotectionagainstdangersarisingfrommilitaryoperations’. Under IHL, civilians and civilian objects are protected from unlawful attacks, and against anenemy,aswelloperationsindirectsupportofsuchoperations. use ofaweaponorplatforminvolvingtheactualpotentialforce tacks, militaryoperationscanalsoincludeotheractivitiesdirectlyconnectedtothe somewhat ambiguous,bothnotionsaresaidtoinclude‘attacks’.Inadditionat- Whereas therelationshipbetween‘hostilities’and‘militaryoperations’remains tilities, attacksandtargetsisnotwellestablished, though theprecisetemporalandgeographicrelationofmilitaryoperations,hos- 163 Art49,API. 162 in offenceordefence’. Attacks aredefinedunderIHLas‘actsofviolenceagainsttheadversary,whether 167 press/final-report-prosecutor-committee-established-review-nato-bombing-campaign-against-federal. the NATO BombingCampaign Against theFederal Republic of Yugoslavia, or that an attack ‘as a whole’ can encompass a series of incidents or engagements. failed’. tions thatactuallyresultinviolenteffects,andthosewhichwereintendedtobut what constitutes‘anattack’intheuseofanAWS? ervations 166 Operations’, Note, 5University of St. Thomas Journalof Law &PublicPolicy 1(2010) 208. injure, enemy potential Attacks’ directed also of According HPCR, Consider, E.g. R. See the so 163 C. against operations’ A. warning in International kill, by defences the as aircraft Else, 2015, Jachec-Nealie, Actsofviolencedirectedatcivilians(unlawfully)arealsoattacks. (ICRC Commentary contrast, ‘Air use at the to destroy, reservation for a to or take them’. ‘Proportionality of legal pp UK CIHL and the missile example, or force by 121–122;ICTY, include all to that missile HPCR target damage, Criminal control Database, They attacking feasible Arts on against combat the The Concept of Military Objectives inInternational Law andTargeting Practice, to Commentary AMW the attacks, include 51 or this such rules systems; 162 Court may measures and capture obligation in an enemy Rule operations’ Manual, effect Recentcommentariesclarifythatthisincludes‘opera- Final Report to theProsecutor by theCommittee Established to Review as the on enemy; attacks 57 be interceptions, (ICC), deployment, targeting 19, Law expected AP bombing; or made radar on to supra on https://ihl-databases.icrc.org/customary-ihl/eng/docs/v1_rul_ The Prosecutor v Bosco Ntaganda , neutralize I, as the and cancel of 28 (emphasis parties well stations by Armed AMW fn apply (ii) January to fighter several 42, or as launching, as cause operations to suspend targets, Manual well ‘refueling; Rule not Conflict: and conflict added) 1998, escort states disproportionate as to 1(e)(1) anti-aircraft (supra ‘attacks’, ‘activities guidance the ‘an 165 in to mean The https://ihl-databases.icrc.org/applic/ihl/ and jamming upon mostcommentatorsconsider attack’ direct support and exercise Proper fn fighter ‘air (6); 42, but ratification directly or support artillery if or of Schmitt, Tallinn Manual, §78, Rule of retrieval’, ‘Control it rather civilian Unit missile becomes sweeps enemy ICC-01/04-02/06, such 1(b)(1)–(c)(3), connected http://www.icty.org/en/ of of or to of operations, harm. during operations the Analysis missile radars; the and preceding AP apparent aforementioned broader I, Some involve the to e.g. sites; suppression for pp the Execution designed or Decision scholars that the bomber 164 Military 169 25–27) ‘actual notion use actual active supra Al- So, res- it 168 167 166 of is .

The Applicable Law: IHL and IHRL Standards on the Use of Force 35 DEFENDING THE BOUNDARY 36 Lethal uploads/2016/04/MHC-2016-FINAL.pdf. Richard 174 LieblichandBenvenisti, ‘The Obligationto Exercise Discretion’, supra fn81, p255. 173 all of thetargets thattheAWS mightinfact strike’ (originalemphasis). 172 a lot more people thanalandmine. bile an Recognition no roomfortheuseofAWS. degree ofhumancontroloverindividualapplicationsforcethatleaveslittleto tions andrequirementspertainingtoattacks,suchanapproachwouldimposea constitute ‘anattack’towhichtherulesontargetingapply.InlightofIHLrestric- endangered’ bytheAWS.Consequently,everysuchinstancewouldindividually selects (orevenonlydetects)atarget,thepersonorobjectconcernedis‘directly narrowly. AppliedtoAWS,thatcouldmeaneveryinstanceinwhichanAWS on how‘directlyendangered’isinterpreted,‘anattack’canbeunderstoodquite a personis‘directlyendangered’byminewasnotclarified,though.Depending to deploytheweapon’, some legalcommentatorsto‘shiftbackthepointofassessmentdecision The uncertainspatio-temporalboundariesof‘anattack’andatendencyamong launched? ber andcontextofspecificactsviolencemaynotbeknownwhen‘theattack’is span, howcancompliancewithtargetingrulesbeensured,eventhoughthenum- targeting parametersandoperatesindependentlyoverawidearealongtime- cant concernsaboutcompliancewithIHLrulesontargeting:iftheAWShasbroad violence committedwiththeAWS,butthiswidernotionof‘attack’raisessignifi- gered’. Inthiscase,thebelligerentnexuscanbepresumedforsubsequentactsof objects thatpotentiallyfallwithinitstargetparametersasbeing‘directlyendan- spatial, temporal,orconceptualboundariestoanattack’, tilities moregenerally).Ifthelawistofunction,however,there‘hasbesome technologies resultinacontinuousexpansionoftheconceptattack(andhos- 171 August 1949(ICRC Commentary APs),MartinusNijhoff Publishers,1987, §1881, p603. 170 AWS tocombatanotherpartyanarmedconflict, Another approachisto consider that ‘an attack’ starts withtheactivation of an attack wheneverapersonisdirectlyendangeredbyminelaid’. the draftersofArticle49APIinrelationtomines,isconsiderthat‘therean force, whendoesanattackbeginandend?Oneapproach,reportedlyadoptedby can be deferred in space and time. When such weapons are involved in the useof As mentionedearlier,theviolenteffectsofanAWS,notunlikethosemines, AWS components, Of ICRC, In Article Autonomous course, is Moyes this launching Commentary ontheAdditional Protocols of 8June1977 to theGeneva Conventions of 12 of 36, vein, Surrender’, such for a Key Elements of Meaningful Human Control, R. vast the Weapons an a Sparrow, restrictive attack. sensor Convention 91 International Law Studies 173 Systems Moreover, array ‘Twenty giverisetotheconcernthatdevelopmentsinweapon interpretation 171 on and (LAWS), Certain Seconds operating it seems may April Conventional natural to within well Comply: 2016, (2015) be to broad justified, p think Weapons 3, 725: Autonomous parameters http://www.article36.org/wp-content/ 172 Background of ‘If andtotreatallpersons this considering AWS (CCW) as are 174 launching Weapon could paper andindeed,some weapons Meeting that 170 potentially Atwhatpoint Systems an to an then AWS of comments attack Experts launching endanger with and against mo- the on by of requirements 176 direct military advantage’ (Art57(2)(a)(iii), API). 177 a military objective atthemost specific level (initsmost minimalform) inthecyber context). 179 Schmitt,Tallinn Manual,supra Commentary fn42, ons5,§2,p141. 178 damaging aseries of targets). Incontrast, see ICC, Elementsof Crimes , 2011, fn36,p19. place, manyhavewarnedagainsttreatingtheentireworldasa‘globalbattlefield’in though IHLdoesnotprovideforspatiallimitationsonwherehostilitiesmaytake that explicitrestrictionsmaybecalledfor. to counteractatrendtowardsassessingproportionalityintheaggregate,suggest 175 limitations areimplicitinIHL. in areasandovertimespansthatarenoteasilycoveredbyhumanoperators. Like certainothernew weapon technologies, AWS could enable theuseofforce tential legallimitsonwhereandwhenhostilities,includingattacks,maytakeplace. The requirement to exercise control points to a related legal debate; that about po- to begovernedbytheconductofhostilitiesparadigm. AWS toestablishtherequiredbelligerentnexusforsubsequentforceapplications an armedconflicthavetoexercisesufficientlyproximatehumancontroloverthe an attacker’scomputersystemtoatarget.Arguably,therefore,agentsofparty warfare’. that ‘anobjectmustbeinthecontrolofanattackingpartytocompriseameans which amilitaryobjectiveshouldbedefinedinthecybercontext, ‘sufficient control’ over a weapon (system) to employ it as if it were its own, attacks applyinrelationtoapartyanarmedconflictthat‘controls’oracquires an attack?Inthecontextofcyberoperationsithasbeenproposedthatruleson applications offorce.So,whenistheAWSusedasameanswarfaretoconduct an internationalborder,thebelligerentnexuscannotbepresumedforsubsequent installed to secure the perimeter around a power plant,detention facility or along conflict againstanother,asmaybethecasewithanautonomoussentrysystem Conversely, ifanAWSwasnotactivatedinordertosupportonepartyarmed Military pursuant as sensor intangibility, Testing theWaters , 180 is connected to thataboutthespatio-temporal boundariesof an‘attack’ (see below). (noting (servers, sions of hitherto unimagined duration’). system questions temporal a means Else, Ibid, H. For UNIDIR, array, or Objectives’, that Harrison fibre-optic to example, Commentary network ‘Proportionality about 179 Art and of ‘[a]utonomous are ‘code’ The Weaponization of Increasingly Autonomous Technologies in theMaritimeEnvironment: of TheInternet,forinstance,isnota‘meansofwarfare’evenifitconnects warfare 52(2), geographic ‘effective ‘the objects Dinnis, level’ the in and pp cables, precise 48 AP principle 3, on will ‘The cyber (Harrison in Israel Law Review I, 5, in contribution’, Rule and the in technologies etc.) http://www.unidir.ch/files/publications/pdfs/testing-the-waters-en-634.pdf proximity the level Nature all infrastructure 41, may, sense Law of this but at Dinnis, §3, military of thus, which of exposes exceptional of to p Objects: 175 Armed ‘definite 142. Art assess will ‘The be Ongoingdebatesabouttheappropriatelevelat the 1 necessity 52(2), making attacked. It (2015) make Nature ‘vast Conflict’, should military proportionality Targeting military situations AP possible amounts 50–54 up of has 177 be Harrison I. objective supra Objects’, its

Where advantage’ kept Networks a network (discussing restrictive qualify “lay of fn in when Dinniss that an mind 165, and supra should AWS and as and (Art infrastructure the 195–213 wait” that dimension a a convincingly fn the potential be databases, military makes military 52(2), 176, a (so-called defined Challenge weapon (proposing 46–48, AP use advantage objective from requirement to I), argues irrespective 176 of – system “long-loiter”) and code, 50). attack’ of which civilian andonways Defining a ‘concrete This by that requirement component, depends that derive and its networks to question 178 an 180 of is nature define Cyber raises and AWS’ used their Al- mis- and the on

The Applicable Law: IHL and IHRL Standards on the Use of Force 37 DEFENDING THE BOUNDARY 38 intensity Gaggioli andKolb, ‘A Rightto Life inArmed Conflict?’, supra fn109, 47. 188 meeting (Gaggioli,The Use of Force inArmed Conflicts, supra fn111, p59). or Implications’, 187 and humanitarian law’). During (if 185 GaggioliandKolb, ‘A Rightto Life inArmed Conflict?’, supra fn109, 47. 184 _armed_drones_-_final_pdf.pdf. 186 with combatant status underIHL,irrespective of thatperson’s location. Conflicts, 65–88, use Drones function 183 Linetal,Autonomous Military Robotics, supra fn72, p77. 182 Minefields to Minespace’, supra fn87, withfurtherreferences. 43-44, 10.1111/j.1475-4959.2011.00426.x. even onterritorynotcontrolledbythebelligerent. over theindividual,determiningifitismateriallyfeasibletoproceedanarrest’, and Kolbunderlineinthisrespectthat‘[t]hecontrolatstakeis…afactual necessity touseforceunderaconductofhostilitiesparadigmdiminishes. argued thatthemorecontrolapartyexercisesoversituation,military 181 and thedegreeofviolenceinvolvedishigh. under IHL,theStateisdeprivedofsufficientcontroloverpersons to enablearrest plicable aslexspecialisifpersonsaretargetedwholegitimatetargetsofattack are produced.GaggioliandKolbproposethattheconductofhostilitiesmodelisap- er itcanreasonablybeapplieddependsonthecontextwithinwhichviolenteffects It isworthrecallingthatthelawenforcementmodeldefaultparadigm.Wheth- an ‘everywhere’and‘foreverwar’, operated in‘regionsofheavyfighting’,‘killboxes’or‘engagementregions’. or aconductofhostilitiesparadigm, criterion todecidewhetherauseofforceisbeassessedwithinlawenforcement tus, functionorconductofthepersonagainstwhomforcemaybeused’ismain circumstances andarea’whereforceisused. tarian Law,takeintoconsiderationtheabilityofapartytoconflict‘tocontrol Guidance ontheNotionofDirectParticipationinHostilitiesUnderInternationalHumani- even ifthatpersonwouldbealegitimatetargetunderIHL? a person by means of a sentry-AWS in a location far from the ‘heart of the battlefield’ it bepermissible,forexample,apartytoanarmedconflictdirectforceagainst ‘zone not of See, See For For DPH force the the doi: and or of of a many supra also e.g., Guidance, discussion Arab only) 10.1093/jicj/mqs096; conduct violence the operations’ will 66 Melzer, D. experts Geographical fn depend Spring’, legal Gregory, IRRC 111, of or supra in Targeted Killing in International Law, p the criterion whether participating were, 893 on the 59). supra person ‘The fn the context (2014) From Scope 106, however, fn degree the Everywhere for J. 149, against p this Pejic, On determining situation of of 67–106, 80. in Armed 14 of the armed standpoint, a not ‘Extraterritorial Similarly, meeting control 181 (noting whom ‘spatial seen andsomesuggestthatAWSshouldonlyeverbe War’, is Conflict’, 185 https://www.icrc.org/en/download/file/7375/jelena_pejic- drones, taking others,includingtheICRCinitsInterpretive force what over that as organized 177 Bellal dynamics it decisive would place territory see may ‘the 184 The Geographical Journal 5 paradigm 186 Targeting Journal of International CriminalJustice and Whereassomeconsiderthat‘thesta- N. Fromthelatterstandpoint,itcanbe evaluation be supra inside be by of Lubell Doswald-Beck, by used’ or 188 legal post-modern the most governs over

fn by or and ICRC (Gaggioli, 92, outside to Means of a experts direct person N. p the on 183 the 297. Derejko, of this ‘Evaluating lawfulness a warfare’, an Armed use The Use of Force in Armed ‘conflict participating Control in topic attack 3 both of (2011) ‘A Drones: force Global in zone’, see human over against of the 2012, 238–250, 187 any Bolton, is Use 182 an in Battlefield? ‘battlefield’ Some Gaggioli the rights particular ‘the Would the area, a 1 of person (2013) status, ‘From Force Legal 2012 main doi: law the standards, andtoadaptoperationsaccordingly. area oranindividualenableandthusrequiretheapplicationoflawenforcement AWS needtobeinapositionrecognizewhencontrolovercircumstances,an fluidity ofcontemporaryarmedviolence,humanagentsinvolvedintheusean An approach endorsed by Arkin,Governing Lethal Behavior , supra fn81, p11. 190 Gaggioli,The Use of Force inArmed Conflicts, supra fn111, pp59–60. 189 sphere ofprotectiontheIHRL treatiesthestateisboundby. weapons rangewithinthejurisdictionofuserstate,and therebywithinthe capabilities ofcontrolstrongenoughtobringthosewithinthe AWS’sensorand surveillance andinstantweaponsdeliveryenabledbytheuse ofanAWSpresents uals. Ontheotherhand,acompellingargumentcanbemade thatthepersistent the violenceistooremoteand/orthattherenointenttotarget specificindivid- is uncertain,especiallyifoneadoptsthepositionthathuman involvementin establish ajurisdictionallinkfortheextraterritorialapplication ofIHRLtreaties by means of an AWS amounts to control over an area or individual sufficient to for statestoassumefactualcontrol.Specifically,whetherthe useofforceabroad person. Insuchsituations,theassertionoflegalcontrolcanbe astrongincentive undermining theobjectandpurposeoflegalrulesforprotection ofthehuman the assertionoflegalcontrolinsituationswhereevolving practices ofviolencerisk this expectationcannotextendtomattersbeyondthestate’sfactualcontrol,and ly their human agents–thatthey control the use of weapons, the acceptance that out thetensionsthatcanexistbetweenexpectationonstates–morespecifical- Through theprismofthreeongoinglegaldebates,discussionabovehasbrought rules governtheuseofforceinspecificcircumstances. of anAWS,humanagentsexercisethecontrolnecessarytodeterminewhatlegal minations, compliancewiththelawdemands,amongotherthings,thatinuse applicable law.FromthestandpointthatAWSdonotthemselvesmakelegaldeter- locus offorceapplicationimpactsintent-andcontrol-relateddeterminantsthe poral andcausalremotenessofhumaninterventionintheuseanAWSfrom vention intheuseofforcecanalsoaffectwhatlegalrulesapply.Thespatial,tem- lenges toensuringaccountabilityforitsconsequences.Themodeofhumaninter- functions’ comesatthepriceofreducedpredictabilityinuseforceandchal- gal rulesfortheprotectionofhumanperson.Increasing‘autonomyincritical man agentsinvolvedintheuseofforcebymeansanAWStocomplywithle- creasing automationinweaponsystemsraisesconcernsabouttheabilityofhu- The reconfigurationofthehuman–machinerelationshipthataccompaniesin- of forceprocedureincontentioussituations, on theconductofhostilities,asapracticalmatter,manyadvocateanescalation Although scholarsmaynotagreeonthelegalbasisforpotentialspatiallimitations the threat posed by the target, rather than its status or function. D. Preliminary Findings on the ApplicableL a 189 whereforceisuseddependingon 190 w In light of the

The Applicable Law: IHL and IHRL Standards on the Use of Force 39 DEFENDING THE BOUNDARY 40 an always Posed by Remote Attack’, 94 IRRC886 (2012) 585. from atic timely mannerwhenandwherethelawofhostilitiesisnolonger theappropriate of forceby means ofanAWS must besuch that humanagents can determinein a that are technically (still) ‘at war’. Consequently, human involvement in the use of asentry-AWS,evenduringanarmedconflictoralongborder separatingstates a conductofhostilitiesapproachistheappropriatemodelfor using forcebymeans there maynotbemanyscenariosinwhichitcanunquestioningly beassumedthat not betheprimaryone.Tojudgefromcontemporaryarmed violencesituations, erence fortheuseofforcebymeansanAWS,andmayinanumbersituations 191 AWS operatinginemptyspacesfaraway, Despite thefocusonIHLinpolicydiscussionsandcommentatorsenvisioning according toaconductofhostilitiesmodel. lowing alawenforcementlogicbecomespossible,anAWScannolongeroperate including personsthatforceisdirectedat,soadifferentiateduseoffol- wise, exercisescontroloverthecontextwithinwhichviolenteffectsareproduced, in spatio-temporal terms. Furthermore, if astate,by means of an AWS or other- when anattackislaunched,humanagentshavetoboundappropriately though thenumberandcontextofspecificactsviolencemaynotbeknown rules canbeappliedsoastoprovideeffectiveprotectionthevictimsofwar,even presumed forsubsequentapplicationsofforce.However,toensurethattargeting armed conflictwiththeintenttoconducthostilities,belligerentnexuscanbe of warfare.Conversely,ifanAWSisactivatedbyahumanagentpartyto to exercisesufficientlyproximatecontroloverthesystemuseitasameans use anAWStoconducthostilities,humanagentsofapartytheconflicthave plications offorceremaingovernedbythelawenforcementparadigm.Inorderto conduct ofhostilitiesinmind,thisbelligerentnexuscannotbepresumedandap- conflict byharminganother.IftheAWSwasnotactivatedahumanagentwith of force must be designed or intended by human agents to serve one party to the force bymeansofanAWS, there needs to bea‘belligerentnexus’,thatis,the use Likewise, duringanarmedconflict,forthelawofhostilitiestogovernuse IHRL standardsontheuseofforce. not ‘accidentally’triggeranIAC.Itsapplicationsofforcethusremaingovernedby deployed tosecureaninternationalboundaryintimesofpeace,forexample,can- another state,thestate’sanimusbelligerendicannotbepresumed.Asentry-AWS of anexplicitexpressionbyhumanstateagentsthewillto‘wagewar’against ger awar,‘onitsown’andtherebybringIHLofIACintooperation.Intheabsence Lack ofproximatehumaninvolvementmustalsomeanthatanAWScannottrig- AWS ‘the Schmitt warships completely would employment and in have areas Thurnher, devoid to of be of such ‘capable the of ‘“Out systems civilians high of of seas the geographic for or far Loop”’, civilian an from attack supra objects’. maritime restriction’ on 191 fn a IHLisnottheonlylegalframeofref- tank 19, See navigation 246, and formation also ‘temporal 250 W. routes’, (portraying Boothby, in a limitation remote whilst ‘Some as area acknowledging a priori since Legal of few the unproblem- Challenges areas desert that are or and Life Rappert, Controlling theWeapons of War, supra fn40, pp91–92, 102. 194 (R. Brooks, ‘Drones andtheInternational Rule of Law’, 28Ethics&International Affairs 1(2014) 98). combatant, aware that AWS ox.ac.uk/downloads/briefings/Robo-Wars.pdf). bodies The Regulation of Robotic Weapons, should berestricted to the[conduct of hostilities] paradigm’ Leveringhaus (A. andG.Giacca, Robo-Wars: 193 of that actions constitutes theappropriatecontextisitselfcontingentonsocialagreement. questions aboutthemeaningandappropriatenessofcertainactivitiesbecausewhat by-case basis.Itshouldbekeptinmind,however,that‘thecontext’cannotresolve gal controversies,thereisatendencytodeferjudgementandassessissuesoncase- 192 and whatthatlawdemands. reaching a widely shared agreement on what law applies in specific circumstances a questionofthesophisticationalgorithmsandsensors,butdepends,critically,on The likelihoodthatAWSusecomplieswiththeselegalstandardsis,thus,notsolely troversies concerningquestionsrelevanttothedeterminationofapplicablelaw. Finally, thediscussionabovedrawsattentiontomajorlegaluncertaintiesandcon- frame ofreferenceandcanadaptoperationsaccordingly. restrictive force, hostilities, in IHRL IHRL unequivocally to ‘If It militaries Armed of of even has there situations changes interact law is of and been not the if entails or Conflict?’, is the thinking in no no ‘paradigms’ two proposed in that all in paradigm fall agreement agreement various, that which frameworks respects into affect supra of human one deploying one that could complex changed’, the more fn on on or or in agents legal 109.) – what what adequately the the international circumstances 193 protective Oxford ways. autonomous qualification other Facedwiththeindeterminacyoflegalnormsandle- In other or constitutes constitutes must the ‘that (For paradigm’, paradigm view Martin exercise be than autonomous Aside human a encoded where of nuanced robots of an an IHL, Policy this a from imminent control armed applies, situation or rights and it author, ‘could into that Paper, is discussion, the weapons that unclear conflict, over law weapon ‘the or questionable threat, simply and the in 2014, – to the use situations which proposal restrict make what are no use see designs, the of programme 192 p only agreement autonomous could 14, of Gaggioli paradigm the law deployment force presupposition deployed to of http://www.oxfordmartin. appropriate is it continue limit parallel should no necessary and them on longer governs the robotic in who Kolb, application, be to to deployment with circumstances adjustments. kept a govern the counts that to guidepost’ ‘A the 194 the weapons Right conduct become in use entire more mind their as IHL to of of a

The Applicable Law: IHL and IHRL Standards on the Use of Force 41 DEFENDING THE BOUNDARY 42 Heyns, ‘HumanRightsandtheUse of Autonomous Weapons Systems’, supra fn22, 355. 197 UNdocA/HRC/23/47, supra fn13,§§92–93. 196 UNdocA/HRC/31/66, supra fn16,§67(f). 195 they institutecontrolmeasuresatinternaladministrativeboundaries andatinter- to certainlocations,suchasammunitiondepotsornuclearpower plants,andthat erations mayjustifyandevenrequirethattheauthoritieslimit thepublic’saccess goods withinthemisasovereignprerogativeofstates.Security andsafetyconsid- To controltheirinternationalborderandregulatethemovement ofpeopleand the level oftechnical competence at which they operate.’ negative, nootherconsiderationcanjustifythedeploymentof[AWS],matter force’. Thisquestion,heargued,‘isanoverridingconsideration:iftheansweris because itentailsnon-humanentitiesmakingthedeterminationtouselethal report totheHumanRightsCouncil,whetherAWSuse‘isinprincipleacceptable, The SpecialRapporteuronextrajudicialkillingshadalreadyquestioned,ina2013 control shouldbeprohibited’. human meaningful no require that systems weapons ‘[a]utonomous that recommended Heyns, Christof executions, arbitrary or summary of association, Maina Kiai, and the Special Rapporteur on extrajudicial, and assembly peaceful of freedom to rights the on RapporteurSpecial the 2016, February in Council Rights Human the to report joint a In including theconductofhostilities. examines theextenttowhichtheyareextenuatedinsituationsgovernedbyIHL, remainder ofthisstudyinvestigatessomethese‘principledissues’furtherand means ofanAWS]islethalornot,andwhetheritusedinwarpolicing’. that ‘thesameprincipledissues’ariseindependentof‘whethertheforceused[by for Sentry-A A Requirements and . Automated WS Kill Zones: Preparing the Ground ? on the Use of A 195 5. Human Rights C onstraints 196 Heynsfurtherargued

WS 197 The

198 being shotbyEastGermanborderguards. border by triggering an anti-personnel mine or an‘automatic-fire system’, or after flee totheFederalRepublicofGermanylosttheirlivesattemptingcrossthat German DemocraticRepublic(GDR)between1961and1989.Manypeopletrying international humanrightsbodieswasthatinstitutedbytheauthoritiesof A bordercontrolregimethatwasfoundtoviolateinternationallawbyseveral national borders. border. and thedangerofenteringareawasknowntopeopleattemptingcross indicated bywarningsigns’,‘involuntaryaccesswaspreventedhighfences’, derlined thatthemineswereonlyused‘inmilitaryexclusionzones’,‘clearly did notprohibittheinstallationofminesalonganinternationalborder.Heun- Nations HumanRightsCommittee(HRCttee)that,atthetime,internationallaw mer headoftheGDR’sBorderTroopspointedoutinproceedingsbeforeUnited resemblance tothoseadvancedinfavourofautonomoussentry-systems,thefor- in particular,andevenmoreso,ceasefirelines,canbedangerousplaces. or whofindthemselvesinneartheareaforotherreasons.Internationalborders range ofhumanrightsandthedignitypersonsattemptingtocrossboundary es, detainorremoveapersonfromanareacanadverselyaffecttheenjoymentof defend aperimeterinordertodetect,intercept,identify,andsomecircumstanc- create ‘zonesoflawlessness’inviolationinternationallaw. ception regardinghumanrightsprotections,legitimizingpracticesthateffectively cases, nationalregulationscharacterizeboundaryareasaszonesofexclusionorex- prising other ures goods’ p and transit 199 org/Documents/Issues/Migration/OHCHR_Recommended_Principles_Guidelines.pdf). Principles andGuidelinesonHumanRightsatInternational Borders, Press us/mexico-investigate-border-killings; 202 The Guardian, asp; 201 OHCHR,Recommended Principles,supra fn198,ppii,2. 200 jan/23/india-bangladesh-border-shoot-to-kill-policy. (2003), 7.5. §§5.3, Chamber, guns. official 8, HRW, extraterritorially along http://www.osce.org/cpc/85263?download=true). ‘A ECtHR, IACommHR, HRCttee, Release In political zones documents (OSCE, several boundary reality, 202 Judgment, ‘US/Mexico: the He argued thatthemineswere ‘a preventive military measure against a Streletz, Kessler andKrenz vGermany, and no Applied Issues inInternational LandBoundaryDelimitation/Demarcation Practices, Klaus Dieter Baumgarten vGermany, entities, lines these politically 23 93/12, ‘IACHR is embassies, contributed and/or essentially January were (Office 22 198 Investigate as 24 Atthesametime,measurestakenbystateagentstoguardor March Condemns defined well fence-mounted, July zones, of and 2011, to as 2012, a the 2001, line the Border at boundaries in whose UN the checkpoints so-called public https://www.oas.org/en/iachr/media_center/PReleases/2012/093. https://www.theguardian.com/commentisfree/libertycentral/2011/ of B. §13. High definition, Recent Killings’, Adams, primary directional References perception Commissioner separating ‘no-man’s Death ‘India’s 11 and function Comm while App June 201 SM-70 border of that to nos Usingargumentsthat bear astriking States 2010, a Mexican shoot-to-kill no land’ their ‘Selbstschussanlagen’ border is the for 34044/96, 960/2000, fragmentation the posts https://www.hrw.org/news/2010/06/11/ between territory border can Human regulation National is at exercise usually [undated], policy train was UN Rights 35532/97 or border by maritime doc secured mines, a of stations, on border U.S. 200 more movement (OHCHR), Recommended CCPR/C/78/D/960/2000 the posts, p or Border and triggered 4, by complex ‘Todesautomaten’ Bangladesh zones sea- governance http://www.ohchr. automated 44801/98, on Patrol of or their 199 from by entity people airports, Insome trip-wire. territory Agents’, border’, that sentry meas-

Grand 2011, com- and of in in

Human Rights Requirements and Constraints on the Use of AWS 43 DEFENDING THE BOUNDARY 44 Velez-Green, ‘The Foreign Policy Essay’, supra fn206. 212 Arkin,Governing Lethal Behavior, supra fn81, p93. 211 same vein, Arkin,Governing Lethal Behavior, supra fn81, p93. along Prevent sides’, because ‘itisuninhabited’,‘scrupulouslyguardedbythousandsofsoldiersonboth narratives as‘theideallocation’forweaponsliketheSGR-A1orSuperaEgisII ishment andnon-discrimination, Whereas theCommitteelimiteditselftoansweringquestionsofretroactivepun- dignity andlibertyoftheperson. infringed fundamentalhumanrights,includingrespectforandprotectionofthe ders to“annihilateborderviolators…andprotecttheatallcosts”’flagrantly matic andindiscriminateeffect,thecategoricalnatureofborderguards’or- recourse toanti-personnelminesandautomatic-firesystems,inviewoftheirauto- automatically anenemy’. heat-signature within itssensorrangeas a target. ‘When you cross theline,you’re a sentrysystemidentifiesanymovingobjectwithhuman-body-shapedinfrared sentry%E2%80%94-killer-robot-prevent-war 203 mously securedboundaries. Controlled, emptyspacesareacentralcomponentofmanyimaginariesautono- possible attackbyNATO’andwerenotdeployedwith‘theintenttokill’people. not ‘militarythreats’. tonomous systemsinsituationswherethe‘vastmajorityofborderviolators’are der areasare‘equallycontrolledenvironments’andcautionagainstdeployingau- some supportersofautonomouslysecuredboundariesconcedethatnotallbor- mote-controlled machineguns,groundsensors,anddronesalongitsborderwith sonably presumedtobecombatants’, individuals thatcanphysicallyenter’theweaponsystem’stargetingrange‘arerea- ‘areas ofexceptionalconditions’ Essentially thesameargumentsusedtoadvertiseautonomoussentrysystemsin 210 Ibid.Similarly, Linetal,Autonomous Military Robotics, supra fn72, p77. 209 Velez-Green, ‘The Foreign Policy Essay’, supra fn206. 208 Parkin, ‘KillerRobots’, supra fn50. 207 environment can reasonably beestablished’). (citing spectrum.ieee.org/robotics/military-robots/a-robotic-sentry-for-koreas-demilitarized-zone of 206 205 Klaus Dieter Baumgarten , supra fn202, 11. §§9.3, 204 Chamber, Judgment,22March 2001. the

J. Streletz etal,supra fn201, §73. Ibid, See, borders, Myung GDR 207 Kumagai, War’, and‘soheavilyfortifiedthattherearenociviliansinit’. e.g., §7.5. in Ho Lawfare, Streletz etal ‘where A. Similar Yoo, ‘A Velez-Green, Robotic a civilians 1 principal arguments March 212 (supra Sentry Forexample,Israelhasreportedlynetworkedtogetherre- ‘The do 2015, 210 research

For not fn 206 were Foreign 201) https://www.lawfareblog.com/foreign-policy-essay-south-korean- TheKoreanDMZ,inparticular,ispresentedsuch Korea’s travel 211 engineer 205 advanced 204 and arealsodeployedindifferentsettings,evenif Policy theECtHR,inarelatedcase,considered‘that or (envisaging Demilitarized 209 in can theargumentgoes,itisunproblematicif ECtHR, at Essay: (equally be Samsung’s prevented K.-H. W. vGermany, The the Zone’, unsuccessfully) South deployment Optics IEEE Spectrum, from Korean & traveling Digital Sentry of by App AWS Imaging former 1 – – no 208 March where A for Because‘any 37201/97, “Killer political Division). defensive 2007, a controlled Robot” leaders http:// Grand In use the 203 to

Streletz etal,supra fn201, §§96–97, 102. 218 of proportionality’. North 215 guns, witharange of 800meters (‘Armour: Israeli Robots Roam theEarth’, supra fn213). mated reportedly 216 North_Gaza1_july_2015.pdf. ered to February automated 217 2009, §§50, 51. Loaded’, with open fireatanybodywhoentersthatzone. zones’ thatpeopleareprohibitedfromentering.Soldiersallegedlyallowedto the IsraelDefenseForcesclassifiedsubstantialareasnearborderfenceas‘no-go embedded. nature ofthestatepracticeforuseforcewithinwhichtheirdeploymentwas of theweapons(ostensibly)deployedinisolation.Itobjectedto‘categorical’ GDR cases,theECtHRdidnotassess‘automatic’and‘indiscriminate’effects these ‘no-goareas’areneitherclearlymarkednorfenced. most theentirepopulationfromleavingGDRcouldnotbenecessarytoprotect mately 213 ning andcontroloftheactions’theirregulationinabstractterms. force application and ‘all the surrounding circumstances, including … the plan- legality of the use of force is assessed includes both the specific circumstances of legitimate lawenforcementobjective.UnderIHRL,thecontextwithinwhich ployment ofsuchasystemwouldhavetobeabsolutelynecessaryinpursuit legal preceptsontheuseofforceforlawenforcementpurposes.Forone,de- Securing azoneorboundarywithsentry-AWSseemsdifficulttoreconcile Gaza to‘create1500-meterdeep“automatedkillzones”’. did notservetoquellariotorinsurrection. against unlawfulviolence,wasnotpursuedinordertomakealawfularrestand secure alegitimatelawenforcementaimasthepracticedidnotprotectanyone were innosensetheresultofauseforcewhichwas“absolutelynecessary”’to garded theneedtopreservehumanlife’,andfoundthat‘thedeathsoffugitives 214 11 February 2011, https://www.strategypage.com/htmw/htarm/articles/20110211.aspx?comments=Y). B. The Duty toIndiviuate the Use of F the Live that Streletz etal, B’Tselem, McCann etal, Shachtman, In Gaza, Palestinian kill 1.5 contrast Wired Fire’, the 2006, zones’ kilometres equipped its December 217 border security , ‘Suspicion IMEMCnews, 12 TheCourtconsideredthattheborder-policingregime‘clearlydisre- to http://www.btselem.org/firearms/20060227_shooting_around_gaza_fence. along ‘Robo-Snipers’, Centre May supra supra the with guards’ inside along 2008, inner the 2014’, fn fn that for 12.7mm Gaza-Israeli 93, 201, the 11 the German orders Human https://www.wired.com/2008/12/israeli-auto-ki/. Israel May Map, §150; border Lebanese §73. supra machine 2015, has ‘left UN Rights, ECtHR, border In fence’ fn border Classified Klaus Dieter Baumgarten (supra OCHA, no http://imemc.org/article/71548/). 46; border guns, room Andreou vTurkey, persons fortifications, (‘PCHR-Gaza: N. are June Shachtman, with (‘Armour: Areas for on have weighing 2015, a Palestinian 214 range near 218 OntheGazaside,boundariesof been A‘generalmeasure’preventingal- which Israeli Israeli http://www.ochaopt.org/sites/default/files/ Gaza ‘Israeli of App the 2,000 attacked were Buffer Robots territory. Perimeter use no “Auto orce under IHRL 45653/99, 213 on meters, of 215 Zone fn Roam AccordingtoB’Tselem, ‘anywhere firearms the Kill See, 202, Towers Fence Policies Zone” GDR’s or Israel the e.g. §7.2), Judgment, with as Earth’, StrategyPage, against ‘Access along Towers territory, has upto «Killing Typically 7.62mm Germany reportedly the [sic] the and 216 Locked Zones»’, 27 According the fence Inthe Enforced principle machine approxi- October Closure: consid- ‘auto- also and are 27

Human Rights Requirements and Constraints on the Use of AWS 45 DEFENDING THE BOUNDARY 46 and hadtherebyviolatedErkanErol’srighttolife. failed totakeallsecuritymeasuresnecessaryremovetheriskofinjuryordeath, it hasoroughttohaveknowledge.IntheCourt’sassessment,authoritieshad lives ofthosewithinitsjurisdictionagainstanyrealandimmediateriskwhich Instead, itreferredtothestate’sdutytakeappropriatestepssafeguard and thepracticewasstillfoundtobeinbreachofobligationrespecthuman and architecturalmeasuresweretakentopreventpeoplefromenteringthearea, ‘exclusion zones’wereclearlycommunicatedtothepopulationandregulatory automated killingattheGDR’sborderwasgeographicallybounded,limitsof 219 become ‘potentiallyexplosivetraps,hauntedbythepossibilityofkilling’. necessary toremovetheriskofinjuryordeath,lest‘[a]llmomentsandallplaces’ Spatial limitationsonautomatedkillingareclearlyamongthesecuritymeasures the state’ssecurity. Erkan Erol,aboywholosthislegafterfollowingflockofsheepintothezone. Court acceptedthatthestatehadnotdeliberatelysoughttoendangerlifeof government’s argumentthatthemineswerelaidtoprotectsecurityforces.The in a‘militarysecurityzone’aroundtheirstation,theECtHRdidnotchallenge dealing withtheemplacementbyTurkishsecurityforcesofanti-personnelmines Automated killzonesarenot,however,completelyexcludedunderIHRL.Inacase rather thantokillisexploredfurtherbelow.) of whetherthesituationwouldbedifferentifanAWSwereusedtoincapacitate, mated killingwouldbegrosslydisproportionatetotheaimpursued.(Thequestion sued could hardly justify the institution of an automated kill zone and/or auto- facility, warehouse,palace,pipeline,powerplantorevenanarmoury,theaimpur- deployed toprevent illegal immigrationorunauthorized access to an airfield,port ings leave exposing speaks Bolton, ‘From Minefieldsto Minespace’, supra fn87, 44. 222 ing itfreely’ andto ‘comprehensively warn’ thevillagersof thelocation of theminesandrisksinvolved). of §§80–90 dangerous and emergence. suitable ‘No-go whether line 45388/99, forbidden_zones?level=1). 221 ECtHR, Paşa andErkan Erol vTurkey, Appno51358/99, Judgment, 12December 2006, §32. 220 June 2008,(ademonstrator §48 voluntarily crossed theceasefire lineto climbaflagpole). afin their deactivate to d’empêcher took Ibid, All one’s greet to existence Zones for human the place to (where the §§30–31, to Judgment, own See, farming mortal state his proportionality access Near the in rights country’ in la Turkish the the the created mines, particular, pénétration danger 37–38 Gaza Russian impacts ECtHR (up context 27 bodies 219 counterpart); to (Klaus Dieter Baumgarten, or October the Strip (originally Consider, considered Arguably,therefore,ifanautonomoussentrysystemwere 1500 failing everyone authorities ECtHR, of the of dealing risk de the Perimeter a legality meters to system this, 2009, civils measure. Albekov etalvRussia , formulated life, in with who that to ECtHR, denied contrast, innocents but from §63 mark of which Fence’, they the wanted the weighs In (a Solomou etalvTurkey, having the and GDR had light in border effectively Cypriot ECtHR, à French border 1 ‘seal to supra border a cet all January of laid positive realize the this, off security endroit’). border mines Kallis andAndroulla Panayi vTurkey, as fence) fn App the denied more regime the a 202, 2011, obligation their requirement area’ no near 221 guard fact regime comprise heavily This §9.4; 68216/01, to had right http://www.btselem.org/gaza_strip/ so a that the village voluntarily positive as App regard Streletz etal, put under if to areas population to about state to no ‘prevent freedom in but Judgment, ‘prendre Art place to duty 36832/97, in because agents 35% breached the the 2 of exists anybody … by supra of Gaza fact toutes of the the contributed 9 Israel movement. they the that October ECHR Judgment, irrespective right Strip fn the les land were from 201, (B’Tselem, ‘[t]he ceasefire 222 to freely deemed mesures App that aware 2008, enter locate Yet, §63), to This kill- 220 24 no its to of is -

Kakoulli , supra fn 226, §116; Kallis andAndroulla Panayi, supra §62. fn219, 228 peacekeeping/missions/unficyp/background.shtml). July since Judgment, crossed 227 Judgment (Merits,Reparations andCost), SeriesCno251, 24 October 2012, §85. Wanted correct 226 south-korean-defector-wanted-for-assault/. along theUNmonitored‘bufferzone’inCyprus entering thearea.Inpractice,thisexpectationwillprovedifficulttofulfill fall withinthesystem’stargetparameters,butwhomaynotbelegallykilled,from such aweaponsystemis,therefore,thattheyeffectivelypreventeveryonewhomay 224 Streletz etal,supra fn201, §73; K.-H.W., supra fn203, §67. 223 rights. 2005, of anindividualillegallypresentinexclusionzonetoobeyawarning, area toarealandimmediaterisklife. sure-activated mine),thestateknowinglyexposeseveryonesusceptibleofentering and anythingexertingapressureofsetvalueaboveinthecasepres- lethal effectscanbetriggeredbythepresenceofanyoneenteringarea(anyone a weaponisusedwhoseparametersofvalidtargetaresobroadthatpotentially area, especiallyasit was locatedonthevillage’sgrazing land. Thissuggeststhat if cient’ topreventtheentryofinnocentcivilians,includingchildren,intomined about thedanger.TheCourtneverthelessdeemedthesemeasures‘clearlyinsuffi- ing signswereinstalledandtheinhabitantsofnearbyvillageinformed of forcemayresultinthepersonevadingcapture. threat ofdeathorseriousinjury.Thisconditionholdsevenifrefrainingfromtheuse sonable forthestateagentusingforcetobelievethatpersonposesanimminent justified a high-securitydetentionfacility).Tobelegal,theuseoflethalforcemustalso suspected or convicted of a violent crime approaching a police station or fleeing a certaincategoryofpeoplefromillegallycrossingboundary(suchaspersons it isnotenoughtostipulatethatpotentiallylethalforcemaybeusedprevent As therighttolifeisinherentineverypersonandnotonly‘innocentcivilians’, capable ofreliablydistinguishingbetween‘lawfultargets’and‘innocentcivilians’. omously secured,wouldbeIHRL-compliantifthedeployedweaponsystemwere raises thequestionofwhetherautomatedkillzones,thatis,thoseareauton- June 225 Instead, theCourt cited atreaty among theapplicable law thatdidnot exist atthe timeof theevents. 2016; 2015, 1974 The The McCann etal, Although §108; international 223 for the jurisprudential buffer 20 Similarly,theminedareaatissueinPasaandErkanErolwasfenced,warn- in theconcretecircumstancesprevailingattime.Itmustbeobjectivelyrea- ‘UNFICYP is Assault’, http://www.bbc.com/news/world-asia-33130382; DMZ, Inter-American between December rare, zone for supra Asian Correspondent, Background’, legal instance. North and 20 2004, value fn standards meters ceasefire Court 93, Korean of §66; See §200; of this and UN Human ‘North ECtHR, line on defectors case Department 7 ECtHR, the kilometers separating 28 is Korea Kakoulli vTurkey, Rights use somewhat October Makaratzis vGreece, and, of Soldier (IACtHR), of 224 mines wide Greek on Peacekeeping Theexpectationonstateagentsusing 2009, diminished one Walks and and Cypriote Nadege Dorzema etalvDominican Republic, 227 App https://asiancorrespondent.com/2009/10/ occasion, contains A. indicatethatneitherthefailure the 226 over no C. Casesdealingwithincidents marking by App from Operations, 38595/97, Archive, DMZ the minefields a no Turkish Court’s South and 50385/99, and ‘South Judgment, Defects’, BBC News, http://www.un.org/en/ fencing Cypriote Korean failure (UNSC Korean Grand of to Res 22 civilian, communities mine identify November “Defector” 2300, Chamber, 225 228 fields. and nor have the 26 15

Human Rights Requirements and Constraints on the Use of AWS 47 DEFENDING THE BOUNDARY 48 so 236 LieblichandBenvenisti, ‘The Obligationto Exercise Discretion’, supra fn81, p271. 235 HRCttee, General Comment no6:Article6(Rightto Life) , 30April1982,§3. 234 LieblichandBenvenisti, ‘The Obligationto Exercise Discretion’, supra fn81, p271. 233 Kakoulli, supra fn226, §114 (emphasisadded); Principle5(b),BPUFF. 232 such afar-reaching decision unavoidable’). limit thecircumstancesinwhichapersonmaybedeprivedoftheirlife’. termined processcannotbereconciledwiththestate’sdutyto‘strictlycontroland ly harmedindividualisnotconsideredinrealtimebut,rather,factoredintoaprede- adjustments inacomplexworld’, also removesakeyjustificationfortherecoursetodeadlyforce. mous sentrysystemremovesthreatstolifeandlimbofhumansecurityagents,it The technologyinvolvedplaysaroleinthisregard:whentheuseofanautono- and strictlyproportionateineveryindividualcase. releases theauthoritiesfromassessingwhetherlethalforceisabsolutelynecessary of deprivationlifeorbodilyharm’. parameters andtoorganisetheiractionscarefullywithaviewminimisingrisk force, human agents are nevertheless expected to retain ‘the ability to assess all tions wheretheprotectionoflifecould,asalastresort,justifyrecoursetolethal circumstances’. by consideringtheeffectsofaspecificactonindividuals,inlightprevailing herent intheuseofAWScontradictsdutyto‘give“duerespect”individuals and Benvenisticompellinglyarguethatthe‘pre-binding’oftargetingdecisionsin- of anAWSforlawenforcementpurposes. individuate theuseofforceunderIHRLseverelylimitsscopeforlawful cation offorceisthusinherentinthedutytopreservehumanlife.Theneed to remainconstantlyandactively(personally)engagedineveryindividualappli- 230 Kakoulli, supra fn226, §115. 229 camp thatwouldtargetanyprisonerattemptingtoescape.Theuseofsuchasystem an armedconflict.Consider,forexample,AWSinstalledaroundaprisonerofwar This conclusionholdsalsofortheuseofforcelawenforcementpurposesduring their (suspected)possessionofaweapon problems release, 231 enemy based ontheiruniform’). ing narrow all who (are made to) carry aweapon orhave similarfeatures ormannerisms (e.g. siblingsorparents). a Weapons in theparticular circumstances, tion, human in of that to Heyns, Left advance, the Kallis andAndroulla Panayi, release spatio-temporal such DoDAAM sniper information Systems’, open presented ‘Human as deadly on is could in the engineers the the supra 235 by Rights possibility be in force ArequirementforhumanagentsinvolvedintheuseofanAWS basis hostage computer boundaries good too fn against and 22, are of slow time’. hypotheticals, scenario the of 358: the hoping algorithms supra to a system a targeting Use and ‘an react, In hostage-taker such to fn of AWS envisaged under 233 develop can 219, Autonomous in a puttinginplaceasystemwherethepotential- that a could system while 232 scenario, be complex constant, §60. Giventhe‘inherentneedtomakeconstant guaranteed determine in systems conceivably who there that 236 229 See Heyns, orwearingofan‘enemy’uniform situation Weapons the is active also autonomously is exposed ‘able system no ‘Human when Parkin, to be human true target to where Systems’, programmed, force for computationally functions and Rights ‘Killer a the supervision, the determines split will pressing hostage-taker, Robots’, supra human and be second, autonomously released the based fn emergency 231 mind the so supra Use identify 22, a Eveninsitua- situation as on moment 370 of ‘is cannot to but 234 facial fn Autonomous that ensure within the 50 (one Lieblich not rendering in recogni- (report they process of type others of which force very that the 230 do of -

‘Twenty he about as dedicated escape’ escalation offorceprocedureisgoodpracticeinthisandsimilarsituations. their assumptionthatIHLis,indeed,theprimarylegalframeofreference. lary ofproponentsautonomoussentrysystemsandautomatedkillzonesbetrays IHL allowsfor‘categorical’(generic,corporate)killinginthissenseandthevocabu- must ‘keepaclosewatchonthesituation’toavoidsuchseriousdevelopments. ‘weapons ofwar’necessary,thesamecommentarynotesthatDetainingPower riots withinacamptoescalatesituationthatcouldultimatelymakerecourse 237 a categoryofpeoplewhomaybemadetheobjectattack,suchas‘combatants’. death, butonthebasisoftheirlegalstatus,thatis,(imputed)membershipin permissible totargetpeoplenotbecausetheirconductposesanimminentthreatof In contrasttolawenforcement,underaconductofhostilitiesparadigm,itcanbe treme measure’ that should be reserved for extraordinary circumstances. would beunlawfulunderIHRLandIHLbecauseitapplycategoricallyan‘ex- to allow for the meaningful application of the rules on targeting. If an AWS, whose of anattack.Asnotedpreviously,wouldneedtobesufficientlybounded force only at (human-body-shaped) objects that are legal targets the questionofwhether,insuchasituation,anAWSwouldbecapabledirecting area ofactivehostilities.ThedebateonthelegalityAWSunderIHLfocuses an autonomoussentrysystemtosecuretheperimeteraroundamilitarybasein One situationpresumablygovernedbythelawonhostilitieswouldbeuseof ‘death line’,‘[f]iremaynotbeopenedautomatically’. material conditionshavebeenmet,suchaswhenaprisonercrossesanestablished prisoners of war, even when there is justification for opening fire and all the required ing to an authoritative commentary on the Geneva Convention for the protection of cessity 241 for functionalprotection, asinthe case of themedical mission. computational 263, 240 Ibid,p248 (emphasisadded). 239 ICRC, 1960, 2ndreprint, 2006, p247. jects —andshoot atthehostiles’). 238 stances’ (emphasisadded). sentry December Melzer, C . The S the use considering Art See, J. To whether robot designed protection of in Targeted KillinginInternational Law, S. as Seconds 42, what its weapons Pictet to e.g., ‘an 2012), restrictive cope for CategoricalKilling under IHL 1949 civilian extreme technologies a extent K. (ed), that person to http://www.hoover.org/research/law-and-ethics-robot-soldiers, of Anderson Geneva against Comply’, purposes’ combatants AWS Commentary: IIIGeneva Convention Relative to theTreatment of Prisoners of War, for IHL sense. measure, is perimeter a with limits Convention prisoners civilian and to Limitations supra (Art analyze ‘adjustable which M. status-based hors decombat, 52(3), (Art Waxman, fn protection, of Relative 172); shall and war, 50(1), can AP doubt differentiate always especially protective supra also I)) ‘Law AP or to able (Schmitt thresholds’ spatially the I) be necessitating and fn be or to inferred Treatment 92, about preceded against detect presumptions Ethics enemy and pp unbounded the would from 397–399, Thurnher, shapes for those 238 an of by threats civilian Robot Consideringthepotentialfor other Prisoners opportunity warnings be who to and an killing for ‘“Out character from central Soldiers’, operate adequate are motions, an of appropriate of escaping friendly is argument War for the in the circumstances institutions subject (considering in of 176 surrender solution); and Loop”’, (GC case an or or Policy Review object III) to combined on innocuous to of attempting 237 241 debate. qualifies supra of military the doubt, Accord- or

(Sparrow, ‘normally IHL, ‘[a] lethal circum- 239 respect fn such An with See e.g. ob- ne- 240 19, ‘[t] to (1

Human Rights Requirements and Constraints on the Use of AWS 49 DEFENDING THE BOUNDARY 50 could of whether 244 supra fn241). 261). imate given were Seconds action machine’ contextual Policy 247 Yearbook of International Law (2012) 205–230. 243 time, offers adefinite military advantage.’ 246 Art57(1) and(2)(b),API. 245 to surrender, heargues, requires keeping AWS leash’). ona‘tight and 1956 those long astheriskof ‘wrongful targeting of non-combatants’ would not do‘more harm thangood’). tem’, fewer wrongfulcasualtiesintheaggregate,comparedtoa‘humancontrolledsys- sibility ofAWSusetendstobetiedanAWS’hypotheticalcapabilityproduce 242 vantage anticipated’. harm whichwouldbeexcessiveinrelationtotheconcreteanddirectmilitaryad- objective isnotamilitaryone’orif‘theattackmaybeexpectedtocausecivilian including bycancellingorsuspendinganattack‘ifitbecomesapparentthatthe care’ to spare thecivilian population by taking‘all feasible precautions’ in attack, implementation ofIHLrulesonattack,notably,theobligationtotake‘constant to detect,selectandengagetargetswithouthumanintervention)isapermissible It is contentious, however, whether ‘autonomous targeting’ (the use ofanAWS tage’ oftheirdestructioncanbepresumedinthesecircumstances. ‘effective contribution’toenemymilitaryactionandthe‘definiteadvan- of enemytanks) can arguably be treated as lawful targets of attack because their their ‘nature’militaryobjectives(e.g.objectswiththeinfraredsignatureandshape for alimitedperiodoftime,objectsthatenteritssensorrangeandwhichareby sensor andweaponsrangeisappropriatelyrestricted,operatesindependentlyonly military commander’, the proportionalityofanattackaretobeassessedingoodfaithbya‘reasonable presumptions become, ly, orthebroaderparametersofavalidtarget,moreproblematicthese wider thespatialandtemporalscopewithinwhichasystemtargetsindependent- target parameters. military preemptively See rejected ICRC to Art start Note Schmitt The ‘Feasibility objects human 247 Essay’, and the circumstances prevailing attherelevant timeandplace’ to a anditissometimesarguedthathumaninvolvementinthedecision-mak- also 52(2), might precarious Draft necessity. and whose with particular that Comply’, supra involvement. in G. which and requires Rules ‘fixed not AP the proposals Noll, is terminate total Thurnher, an I fn 1970s, be In by provides supra ‘Analogy transition conduct for lists issue 206 spite 244 or saved their the the partial 245 of to in See, (acknowledging fn 246 ability the of of Limitation lawful 243 Eventhoughthefeasibilityofprecautionarymeasuresand favour nature, ‘“Out list ‘as that at amounts 172 inconductofhostilities-centereddiscourses,thepermis- this, reasonableness’ from e.g., engagement’, especially if human-body-shaped objects fall within its War: destruction, categories the to ‘[i]n (stating of proponents targets’ of DPH location, being operator interpret Proportionality, the a so of to definitional Guidance, the far Loop”’, direct targetable that (Anderson that of capture as Dangers purpose but would and legitimate of (Schmitt the objects ‘tragically’, participation AWS supra suggesting identify supra approach nature or likely Equality to incurred and or suggest neutralization, are fn and being use objects fn Waxman, be human 19, of concerned, ‘the 106, Thurnher, that that make and unable the in by 261. protected that initial hostilities, pp of the a demands the signals intentions. (emphasis in an sentry-AWS See attack ‘Law 41–42 Civilian to order Law non-combatants in effective military ‘“Out foresee also the against and used of (as (noting a ‘to due circumstances of Population Targeting’, contextual Velez-Green, Retaining added); Ethics envisaged program objectives to the would contribution the consideration attack 242 indicate that Loop”’, for wrongful However,the Sparrow, be engaged the Robot 43 demands distinction’ ‘in in analysis acceptable in, are Time ‘The determining ruling Netherlands surrender supra opportunity to e.g., limited targeting Soldiers’, must ‘Twenty military of Foreign in by at fn Art prox War) light one the the 19, be as to is 7, - Noll, ‘Analogy atWar’, supra fn246, 223. 251 autonomous functions’, including‘monitoring theoperation of theweapon system’). precautions Law of War Manual, 250 thus, thespace andtimeof independent functioningare sufficientlybounded andunderhumancontrol. does 249 Use of Autonomous Weapons Systems’, supra fn22,371. Suchman gorithms Weapons be (that wouldnotamounttoanarbitrarydeprivationoflife), positiveobligations It bearsrepeatingthatevenifthereisscopeforcategorical targeting underIHL or atleastfrequent,periodic,humancontroloverindividualattacks. hostilities wouldneedtoexerciseactiveandconstant,inthesenseofcontinuous man servicemembers’. and, wherenecessary,adjustmentofAWSinputs,outputs,andinterfacewithhu- on anAWS’independentoperation,coupledwith‘frequent,periodicassessment zation ofAWStargetingdecisions,but would demandspatio-temporallimitations reading, adutytoexercise‘dynamicdiligence’wouldnotcallforexanteauthori- gulies derivesaconceptof‘dynamicdiligence’fromtheruleonprecautions.Inhis constant altered inrealtimeifcircumstancesrequireisunlawfullightofthedutytotake sions tobemadebyanAWSbasedonpre-programmedalgorithmsthatcannot of makingadjustments’isleftopen.Intheirview,allowingfinaltargetingdeci- uation, IHL notions of reasonableness and feasibility demand that ‘the possibility cannot bedeterminedinadvancewhatis‘reasonable’or‘feasible’anygivensit- targeting decisionsaboutspecificattacks.LieblichandBenvenistiarguethat,asit Others considerthattherulesofIHLprecluderemovalhumanagentsfrom targeting rulesaretobeproceduralized. fication despiteprecautionarymeasures,andbecauseIHLissilentabouthowkey resulting fromanattackonalawfulmilitaryobjectiveorastheresultofmisidenti- portion ofwrongfullykilledpersonsistolerated,whetherasincidentalcasualties 248 reasonableness’). precisely meter doubt’ (a concept that, they concede, is framed in terms of ‘human maximum collateraldamagethreshold’andcontemplate‘algorithmsthatcan… for example,envisagetheuseofalgorithmsthatallowanAWStoadjustits‘base cation whereviolenceisadministeredcanstillbeadequate.SchmittandThurnher, ing processthatistemporallyandgeographicallyremotefromthemomentlo- making sense, adjustments andappeartoassumecapabilitiesofsensing,aswell ments anddeterminationsremainnebulousinmanyregards,theydemandtimely Clarified’, not Schmitt Lieblich Margulies, mean and, and and care andtherebyexercisecontinuoushumandiscretion. ‘may 90 International Law Studies and Weber, and International therefore, that ‘Making Thurnher, be 251 Benvenisti, the suggestingthathumanagentsusinganAWSfortheconductof ‘Human-Machine 248 more supra use IHLlendsitselftosuchanorientationbecauseacertainpro- according Autonomous Humanitarian ‘“Out significant’ of fn 250 ‘The a 9, AlthoughthepreciseproceduraldemandsofIHLassess- cruise of s Obligation to the 6.5.9.3, a Autonomies’, Weapons missile Loop”’, when plan Law: (2014), p established a to is 330 supra Advantages, person illegal, Exercise Accountable’, 322, (acknowledging supra fn 19, 336 as uses by Discretion’, fn long 256–257, Open (assuming humans). 73, ‘weapon supra as pp Technical the 85, 263. that supra fn context For that systems 92; 85, See the a Heyns, Questions machines fn pp critique also obligation of 81, 19, with its M. 249 p ‘Human 22. use 270. Sassòli, Similarly,Mar- of more act and See (the this to This according Legal Rights sophisticated take also ‘Autonomous premise, attack), orientation Issues US feasible and to DoD, and, see the al- to

Human Rights Requirements and Constraints on the Use of AWS 51 DEFENDING THE BOUNDARY 52 Remote being precautionary siders (supra 255 Rosén, ‘Extremely Stealthy andIncredibly Close’, supra fn130, 124–125. 254 Tech where itwas ‘unclear whetherthecasualties [were] farmers orgunmen’). http://www.ynetnews.com/articles/0,7340,L-3856218,00.html heightened expectationsonstatesintermsoftheirpositiveobligations. bot, areadvertisedfortheir‘interception’ weapons brandedas‘non-lethal’.TheMDARSandtheSCOUT,anotherpatrolro- of engagement. ed toconstantlyvideo-recordtheSentryTechareaofcoverageenablerecords amounts ofdata.IntheGazaborderarea,forexample,controlcentresarereport- on technologiesofferunprecedentedsurveillancecapabilities,andproducelarge where theviolencetakesplace.Itisrelevanttoassessmentthatmodernweap- jurisdiction orexerciseofauthority,powercontroloverindividualsthearea of astate’spositiveobligationtosafeguardlifewillvaryinrelationthe 252 cious activities’ military combatappearstobetheircapability‘[d]etect…anintruderorsuspi- A keysellingpointforsentrysystemsthatarenotprimarilyorsolelyconceived under IHRLcontinuetoapply. the conductofitsoperations. lack ofawarenessaretheresultstate’sownfailuretotakeconstantcarein to recognizethatatargetwasnotlawfulonewhenthesecircumstancesandthe effective advancewarningofanattackthatmayaffectthecivilianpopulation,or of humanagentsfromtheselectionspecifictargetstojustifyitsfailuregive stance, astateusingansentry-AWScouldnotinvoketheremovalorremoteness capabilities.aspx). June evaluative cording that 63). the system 257 ChunandPapanikolopoulos, ‘Robot Surveillance andSecurity’, supra fn38,p1606. 256 the attacker, ‘even greater caution isrequired’). supra 253 situations thatare widelyregarded asNIACs (albeitnot by thestates involved). Palestinians D. ‘ attack intruder See, the Non-Lethal’ Autonomous Interception ‘Scout Although Hughes, A fn 2004, from that is fn mounted similar Attack’, of 170, to Court e.g., 220) precautionary take ‘depending the detected the have refused §2221, http://www.af.mil/News/ArticleDisplay/tabid/223/Article/136631/robotic-warriors-display- A. ‘IDF duties found the argument and Sentry place, autonomous supra Waked, on been Deploys ECtHR 256 253 in p Scout’ to and impractical’ violations and Albekov etal andapprehendatarget.Suchsystemstendtobeequippedwith Thedeploymentofsuchtechnologiesshould,arguably,entail fn 686 Tech on killed’ obey ‘Palestinians: rules 191, confronted is can the pointing Sentry (referring ambiguous (T. area aspect commands be 584–585 pattern by D. at of made the of and Erazo, the Tech the out (supra coverage, of 255 an to system mission thus of 1 respective about that on the under (finding 252 the intruder dead, ‘Robotic life issued Gaza Whatoperationalstepsarecalledforinterms ‘militarily fn decision-making need if in (J. whether direct planning 221) IHL. 4 the the it Border’, by Cook, trying injured for Warriors states’ unsatisfactory See, as Israeli relevant the observation an 257 non-feasible’. part it ‘Israel’s to stage). controller, e.g., attacker and‘on-the-spotdetainmentofiden- from assesses supra positive gain army Display of area, Boothby, process IDF the unauthorized Video Consider fn (reporting has is to to the fire obligation 52. it conduct he not Capabilities’, ‘observe’ At argue may so use Game renders was In in ‘Some possible the far spite Gaza’, also be of ‘that disabled same of an ‘refuse[d] mines Killing access possible the to Legal ICRC hostilities, of the incident Ynetnews, due protect the U.S. Air Force News, context the time, performance in Commentary Technology’, to Challenges to with absence persistent Paşa andErkan Erol to the the to however, life involving comply it within a say flightline. remoteness 1 is pepper in March 254 of noteworthy how respect Posed Forin- video-re- which a with of supra on he human Sentry 2010, these many spray After AP con- the 25 by an fn of of I, 261 technorobot/). Support deliver §§56–57 htm). degree andintensity,notoneofnatureorsubstance’. restriction onlibertyofmovementandadeprivationis‘merelyone security’ or‘publicorder’,amongotherpermissiblepurposes. can bejustifiediftheseare‘providedbylaw’and‘necessarytoprotectnational terference withlibertyofmovement’, involves more severe restriction of motion within a narrower space than mere in- on thebasisthattheyposeasecuritythreat,burdenisstate‘toshow persons notinrelationtotheprosecutionfororpreventionofaspecificcrime,but temporarily restricted.Itisalsoworthnotingthat,totheextentthatstatesdetain persons thatitdetains,comparedtowhosefreedomofmovementhas ters becauseastateassumesdifferentandmoreextensiveresponsibilitiesvis-à-vis right tofreedomofmovement. for exampletoascertaintheiridentity,wouldconstituteaninterferencewith cluding, critically,itseffectsonthetargetandbystanders.‘Intercepting’people, depends on its legalbasis, finality, necessity, proportionality andother factors, in- Whether the use of a sentry-AWS would conform to IHRL in a concrete situation be subjecttolesserlegalrestraintsthanapotentiallylethalone. can beaskedwhethertheuseofasentry-AWStointerceptandincapacitatewould means toeliminatethethreat’. them back until manned security forces arrive’, but it can also ‘use various forceful is describedasbeingcapableof‘suppress[ing]suspiciouselements’and‘hold[ing] no determination onArt5). 259 General DynamicsRobotic Systems, ‘MDARS’, supra fn62. 258 tified intruders’, nial’ rivation ofliberty. though, detainingapersonforeven‘veryshortduration’canamounttodep- 260 gyreview.com/s/413544/modeling-sneaky-robots/. 265 HRCtte, General Comment no35:Article9(Liberty andsecurity of person), 16December §5. 2014, 264 Art9,ICCPR; Art6,AfCHPR; Art7, AmCHR; Art5,ECHR. 263 Art12(3),ICCPR. 262 Convention onHumanRights(AmCHR);Art2,Protocol 4to theECHR. control. vertised centrated fence-crosser’ [the] 2009, and development’ K. Art ECtHR, The ‘Enguard! http://aviationweek.com/awin/robot-sentry-patrols-borders). over Grifantini, According Robots’, for ‘urban 12, (considering load RiotBot use 700 ICCPR; Gillan andQuinton vThe United Kingdom, (B. of warfare’.

in Introducing armytechnology.com, rounds Sweetman, ‘Modeling PAVA to is a of Art 258 wide equipped TechnoRobot, the that ortheirabilityto‘catchanescapedprisoner’. 263 (Capsaicin 12, per (TechnoRobot, variety robot WhereastheHRCtteeconsidersthat‘deprivationofliberty the African Sneaky minute. the ‘Robot ‘element with include Guardium of II) Robots’, Charter the Sentry situations, ammunition (TechnoRobot, powder, http://www.army-technology.com/contractors/unmanned_vehicles/ 261 260 manufacturer of ‘riot ‘Riotbot’s Temporaryrestrictionsonfreedomofmovement Thehyperboleofadvertisersaside,thequestion coercion’ Patrols on MIT Technology Review, UGV’, control’, an Human blurring 264 active that supra Borders’, Applications’, fortheECtHR,differencebetweena ‘Rapidly is ‘civil of ‘combines ‘indicative and and the App fn RiotBot, order’, Peoples’ 59. Defense Technology International, line potent Deployable no Allegedly, between http://www.technorobot.eu/en/riotbot. 4158/05, of a ‘jails ‘scenarios frangible 20 ingredient a An Rights deprivation May and autonomous Remotely 265 perimeter it Judgment, 2009, prisons’, (AfCHPR); that Thedistinctionmat- has plastic of pepper 259 have ‘already https://www.technolo of Operated 262 TheGuardium as liberty’, sphere security system Importantly, 12 Art been well spray’ January detained 22, as with but Less-Lethal studied can and 1 American ‘area and October making be a crowd 2010, con- one can dead- for

Human Rights Requirements and Constraints on the Use of AWS 53 DEFENDING THE BOUNDARY 54 notion on App weapon American Institute of Aeronautics andAstronautics, 2ndedn, 2013, pp 283–284). 274 Arts52-56, API. 273 destruction of applicants’ houses by airstrikes onavillage). 272 p33. 2014, in jects arespecificallyprotectedagainstdirectattackunderIHL, limited totheuse(anddevelopment)of‘lethal’AWS magnetic energy.IntheframeworkofCCW,discussionsonAWSareformally means, includingthediffusionofchemicalsubstancesordirectionelectro- or bycreatingblastoverpressurethroughadetonation,forexample)other ably harmthroughkineticenergy(causingaprojectiletohitorpenetratetarget gally. Suchdecisionscannotbebasedon‘amandatoryruleforabroadcategory’. security’ canjustifycontinueddetentionofapersonwhoenteredthecountryille- of absconding,adangercrimesagainstothersorriskactsnational 269 E.g., ECtHR, Nakayev vRussia , Appno29846/05, June2011, Judgment,21 §58. 268 Ibid,§18(emphasisadded). 267 Ibid,§§15(emphasisadded). 266 severe mental trauma can amount to inhuman or degrading treatment. damage arealsohumanitarianandhumanrightsconcerns.Seriousphysicalinjuryor ing fromAWSuse,physicalharmshortofdeath,severementalsufferingandmaterial Whereas afocuson‘lethal’AWSdrawsattentiontotheriskofdeathdirectlyresult- can neverthelessfallwithintheambitofrighttolife. Furthermore, useofforcethatdoesnotresultinorisintendedtocausedeath the conflict (ormoreaccurately,theiruseasameansofwarfare). vulnerabilities ofthoseaffectedbyit. weapon areneversolelyafunctionofitsdesign,butalsodependonuseandthe tion ofweaponsinto‘lethal’and‘non-’or‘lesslethal’obscuresthattheeffectsa including death,forexampletopeopleinthevicinityofanobjectattack. ject (or of an ‘anti-materiel’ AWS) can have indirect adverse effects on human health, direct effectsonpeople(orof‘anti-personnel’AWS),theuseanAWSagainstob- 270 Indiscriminate Effects, UNdocCCW/MSP/2015/9, 27 January2016, §§5,35. (serious stances. 1 sporadic 271 use of weapons for law enforcement purposes duringanarmed conflict. adopted S. the no individual posessuchathreat’. Casey-Maslen Weapons In Final Preamble As Use of any 23502/06, See, against injury recognized acts ‘lethal Report, on of case, e.g., Certain of 21 from termed and a violence, area’, in December A. specific Meeting military (ed), Judgment, Art the Dymond-Bass in Conventional for ECtHR, ‘non-lethal’ 1, use Weapons UnderInternational HumanRightsLaw, CCW. and target instance, of parlance, 2001 of the other Abdullah Yaşa etalvTurkey, tear-gas 12 Originally, (M. High expands November and Weapons are is R. acts ‘kill’ Contracting used Driels, capable N. grenades or of Corney, the its ‘lethal’ a in 266 Which Weaponeering: Conventional Weapon System Effectiveness , similar scope 2013 scope weaponeering Likewise,onlythe‘individualizedlikelihood of 271 ‘The during Parties causing does (witnessing May of nature’ to Use the not NIACs. be demonstrations) to CCW and of necessarily Deemed the are App “Less-lethal” to The have was Convention the excluded express no use 269 limited killing in to 44827/08, imply inthecontextofanarmed fact be of the 268 from Excessively weapons Weapons and of caused on to AnAWScanconceiv- death Cambridge effectiveness close IACs. Prohibitions 273 in the Judgment, 270 ECtHR, and in addition to andinadditionto in death CCW’s An relatives in Thecategoriza- in a Injurious ‘riots, Law medical amendment 272 University Benzer v Turkey, in ambit, Civilian ob- of or Enforcement’, some 274 16 isolated and Restrictions a

sense. or July particular as wanton to circum- Press, to is 2013 Have and The 267 the Art

injury no sulted ly evaluate’itsdeploymentandto‘carefullycontrol’use. and underIHLstatesarepermittedtotakequiteintrusive‘measuresofcontrol’,includ- under therighttolibertyofmovementandsecurityperson, In timesofwarorotherpublicemergency,statescanderogatefromtheirobligations intended to‘intercept’ratherthan‘eliminate’. irrespective ofwhetheritisequippedwithweaponsbrandedas‘non-lethal’and type ofindividuatedhumancontrolintheuseanautonomoussentrysystem, on individualcircumstances,compliancewithIHRLrequiresessentiallythesame the samesecuritymeasureinvolvinganAWScanvaryconsiderablydepending the limitedargumentadvancedhereisthatbecauselegaldutiesflowingfrom effects andmannerofimplementationthemeasure.Withoutgoingintodetail, situation islawfulwilldependonarangeoffactors,includingthetype,duration, of injuryanddamage, equipment thatallowsforadifferentiateduseofforcecanhelpminimizetherisk or theprohibitionofcruel,inhumandegradingtreatment.Theavailability 275 a deprivationofliberty. stances, constituteameasureof‘coerciveandrestrictivenature’,amountingto thus, interfere with the right to freedom of movement, and in particular circum- The useofasentry-AWStointerceptandpotentiallyincapacitatepeoplecould, lethal force’orasforcethatisnotusuallyfatal. acterized asuseofa‘non-lethalincapacitatingweapon’,‘lethalforce’,‘potentially particular situation,butcaselawindicatesthatitisnotdecisivewhetherchar- used affectswhethertheuseofforceisdeemedproportionateandnecessaryina ‘supposed Chamber, explosive to §§56–63, 279 paragraph. 18299/03 containment October 278 Principle3,BPUFF. 277 Principle 2,BPUFF. 276 caused the‘detention’). In nevertheless restrict liberty administrations’ pedestrians of instranet.CmdBlobGet&InstranetImage=2833408&SecMode=1&DocId=2278178&Usage=2). this an Europe, 50275/08, For unjustified ECtHR, In … in case, if freedom or 2013, the ECtHR, example, the weapons Judgment, and the to indeed Note on the ‘detention’ and Austin etalvThe United Kingdom, ‘susceptible death kill’), Judgment, §§39, Court 27311/03, the of intervention Scavuzzo-Hager etalvSwitzerland, to deprivation Verbale, crowd-control movement in of it enforce streets, as of June has grappled killing 15 41 ‘lethal’ a does demonstrator (finding March recognized Judgment, to 22 2015, 5 roads curfews 276 someone’ lead June of of July or not and butthisdoesnotreleasethestatefromdutyto‘careful- 275 with liberty); 2012, a the ‘potentially techniques that and to exceed Itcouldevenfallwithintheambitofrighttolife the human 2014, 2015, and Ukraine death’. the 20 that right terrain airport §§60, (Abdullah Yaşa etal,§§42–43). both ECtHR, ‘to December §46, https://wcd.coe.int/com.instranet.InstraServlet?command=com. differentiation ‘the agent riot-control to temporarily Whereas notified lethal’ such as private areas’ 64–69 time border the Gahramanov vAzerbaijan, ‘lethal permitted as App Court use strictly 2011, (Permanent its ‘kettling’ life’, (noting control the agents force’ App of derogation nos restrict between described including §232: ECtHR force necessary the 278 39692/09, no and holding that like could, Whethersuchuseinaparticular timely (ECtHR, or 41773/98, considering Representation tends as tear prohibit it force by from the in ‘potentially cannot to granting a correction gas to particular use Finogenov etalvRussia, passenger 40713/09 comply In Art that describe App ECtHR, bear the Judgment, of 2 277 be that of is tear-gas ‘powers movement no Thetypeofweapon with excluded Protocol not a of lethal of circumstances, bombs Ataykaya vTurkey, 26291/06, ‘risk and is the Ukraine a ‘in relevant not database 7 to of force’ use 41008/09, grenades itself 4 and a February military causing that of to deprivation of to vehicles ‘temporarily in fatal’ Decision, formalities’. missiles firearms the the error the give App and that serious Council use 2006, Grand but same App that civil and rise nos are 279 re- 15 or of of is

Human Rights Requirements and Constraints on the Use of AWS 55 DEFENDING THE BOUNDARY 56 September Administrative 284 (internment Art21 of prisoners of war), GCIII. 283 Commentary, thiswas compensated by procedural safeguards against arbitrariness). – rejected mention against sis Civilian Persons inTime of War , to automatically transfers 191 viduals againstarbitrarinesswouldseemtoseverelylimitthescopeofsuchameasure. an IACbut,inpractice,proceduralsafeguardsunderbothIHRLandIHLthatprotectindi- opens up the theoretical possibility of using an AWS to intercept and detain persons in vention (civiliansinIACs). that thisisalsothecaseinrelationtopersonsprotectedunderFourthGenevaCon- ternment decisionsmustbeindividuatedinNIACs,andthereisastrongpresumption must bemadeonanindividualbasis,inregularjudicialoradministrativeproceedings.In- speaking, onlybejustified,exceptionally,byimperativesecurityreasonsanddecisions 280 an AWSservesasaproceduralsafeguardtoupholdhumandignityandrights. right toaneffectiveremedy.Fromthisperspective,humaninterventionintheuseof ed againstandnottobesubjectedcruel,inhumanordegradingtreatmentthe threaten humandignity,aswelltherighttoprivacy,notbediscriminat- section of the study elucidates aspects of algorithm-based target construction that how andwhypersonscanlawfullybemadethetargetofsecuritymeasures.Thislast Yet, IHRL also places demands on decision-making processes, including in terms of IHRL) with aview to assessingthelegalityof use offorceintermsitsoutcome. ly fromtheneedtoadequatelysituatelegalevaluations(toindividuatethemunder Requirements andconstraintsontheuseofAWShavethusfarbeenderivedmain- ing assignedresidenceandinternment. 282 HRCtte, General Comment no35,supra §64. fn264, 281 1949 Geneva Convention Relative to theProtection of CivilianPersons inTime of War (GCIV). IHL. under IHRLtotheextentthattheyareauthorizedandregulatedbycomplywith Geneva Convention(combatantsfallingintothepowerofenemyinIACs). persons whobyvirtueoftheir‘status’areprotectedasprisonerswarundertheThird E Rights and which . Algorithmic T the added)). (noting 281 E.g. See, See requirements certain Even during an armed conflict, though, recourse to such measures can, generally Evenduringanarmedconflict,though,recoursetosuchmeasurescan,generally would by of and Arts e.g., ECtHR, 2014, a that the In requirement evacuations justifying 35 Detention contrast, categories N. drafters, force ‘the §106. (right Melzer, Hassan v The United Kingdom, Human Dignity of mere a each government For to internment J. ‘on in in without to International Humanitarian Law: AComprehensive Introduction, S. fact leave Armed a individual occupied the Pictet take arget C discussion, that ICRC, the 282 ground decisions always Conflict (ed), In contrast, IHL allows for the ‘categorical internment’ of Incontrast,IHLallowsforthe‘categoricalinternment’of a without territory); to person 1960, territory); case act see that Commentary IVGeneva Convention Relative to theProtection of finding and onstruction: A on without and 2nd J. completely is there Pejic, 41 internment an Other 280 78 reserving reprint, it (assigned enemy Such measures are not, in principle, arbitrary Suchmeasuresarenot,inprinciple,arbitrary (assigned might possible ‘Procedural delay App Situations defeating 2006, national be no internment individually to residence to situations residence prevent 29750/09, consider Art of Principles Violence’, cannot the 41(1), for hostile and was idea Threat to Human or – individual a the p Grand be and interment internment); threat discussed 256 of 87 regarded most acts, Safeguards tailoring IRRC (reporting Chamber, of cases’. serious and in 858 invasion at as occupied security to 49 length (2005) According a for cases’ that take security (deportations, Judgment, ICRC, Internment for and an measures measures territory), 375–392. 283 (empha- example 2016, explicit This This threat finally to the 284 16 p /

processing’ CoERMPublicCommonSearchServices/DisplayDCTMContent?documentId=09000016806a616c, Art rm.coe.int/CoERMPublicCommonSearchServices/DisplayDCTMContent?documentId=09000016806b6ec2). time, nological the ered protection principles. enabling persistentsurveillance,and,possibly,attacksontargetswithinawide given accesstosurveillance infrastructures and data held anywhere in the world, of being targeted during their presence in that zone, an AWS could conceivably be patterns withinadelimitedzonewouldexposepeopletosurveillanceandtherisk automatic processingofpersonaldata social environmentsfrequentedbythem.Suchusewould,thus,likelyinvolvethe everyday life, places ofresidence, movements, activities,socialrelationshipsand ing certaincriteriaandtrackingthementailssurveillanceofpeople’shabits 285 rorism/violent extremism. secret, surveillancesystemsputinplacerecentyearsthenameofcounter-ter- Imaginaries ofautonomoustargetingtakeshapeagainstthebackdropintrusive, 1. Surveillance of tion, intervention. statistical, dataanalysisorAItechniquescanbeapplied,withminimumhuman tion technologiesaregeneratingeverlarger amounts of digitized data to which 19 Automatic an 286 investigatory-powers-act-a7426461.html. 287 DisplayDCTMContent?documentId=09000016806840b9. distributed Modernized Committee the Profiling Mechanism: SomeIdeas for theFuture Work of theConsultative Committee, tion the combination 288 the carrying outof logical and/or arithmetical operations onsuchdata’. coe.int/Documents/Handbook_data_protection_ENG.pdf to the limitation 28 November the identified January Personal 2(b), Processing Council data of storage, ’identifiable’ S. effort Art For J.-M. Convention Carlo, data Council collection. developments 2(a), example, if Processing and as or 1981, Dinant of of CETS on ‘The it of 2016, or preservation, Data, ‘any other Europe, Handbookon European Data Protection Law, 1981 the entails of different age, are identifiable 286 Snooper’s of §30). 108, Personal if for operation , http://www.independent.co.uk/voices/snoopers-charter-theresa-may-online-privacy- Convention Europe UsinganAWScapableofdetectingindividualsorobjectsmatch- brought data his resources’ See Council C. sex, 11 the of supra subsequent Lazaro, It or such January Art Personal types mining occupation, also Protection her Charter alteration, Data Draft 288 of individual’. together fn 5(4)(b), or as includes (Draft identification Whereasanautonomoussentrysystemusedtoanalyse Europe 287, of Y. for set growing (Draft for Modernised 285 Poullet, data, 2008, Data Passed of the processing §§46–47; targeting of Developmentsinsurveillancepracticesandinforma- Explanatory Draft geolocation, via retrieval, operations (CoE) situations Modernized Individuals (CETS The Protection such into processing computer p N. Modernized definition would Convention 5, as Law Convention Lefever 108) risks European disclosure, of physical, 287 https://rm.coe.int/CoERMPublicCommonSearchServices/ which where Report this family data with defines of require withthepotentialtoundermine CETS links infringing power, Week and Individuals also is CETS regard that it . to Union for status, for 108), for physiological, performed is making A – ‘personal ‘unreasonable covers Draft Say possible ‘may the purposes the Rouvroy, modifies 108 the as to Agency Goodbye etc. Protection Protection Modernized change Automatic of and with available, principle ‘sets Whereas September data’ on to 2014, of Application of Convention 108 to Draft the for genetic, Regard of identify personal to processing’ what time, Fundamental data as Your purposes of of of erasure, Explanatory Processing pp CETS ‘any an Individuals Individuals purpose effort constitutes Privacy’, to which an economic, 68-70, 2016, individual data, information 108, Automatic individual or (Explanatory originally or are https://rm.coe.int/ such §§16, http://www.echr. destruction specification of The Independent , Rights resources’, Report with with geographically Personal or ‘unreasonable is as Consultative defines through

a not 18, Processing key data relating the Regard (FRA) Regard combina- justifying to https:// consid- Report collec of, tech- Data, Draft ‘data and and the or to to to -

Human Rights Requirements and Constraints on the Use of AWS 57 DEFENDING THE BOUNDARY 58 unlimited discretion to subject individuals to secret surveillance orasystem of secret files’). of Surveillance Studies, 290 [of anautomated surveillance-military killingmachine]taking root indomestic contexts). technologies and 4 the 295 or even destroying democracy ontheground of defending it’). limited See org/es/texto-biblioteca/eng/b022-technologies-of-exception-urban-warfare-and-us-military-technoscience. 294 Draft Explanatory Report to Draft Modernized CETS108, supra fn287, §9. 293 Preamble, Draft Modernized CETS108, supra fn287. 292 Art 17, ICCPR; Art11 AmCHR;Art8,ECHR. 291 Military USA/CO/4, of critical Concluding Observations ontheFourth Periodic Report of theUnited States of America, to technology’ for -%20Budget%20Entries%20-%20FY2005%20(Approved).pdf; www.darpa.mil/attachments/(2G12)%20Global%20Nav%20-%20About%20Us%20-%20Budget%20 DoD, http://archive.darpa.mil/DARPATech2005/presentations/diro/leheny.pdf; safeguard humandignity. So-called ‘strategic’,automatedorlarge-scaleinterceptionthateffectivelyresults by law.Secretsurveillancebearsaparticularriskofabuseandarbitrariness. competent topermit,carryoutandsuperviseit,thekindofremedyprovided and durationofthemeasure,groundsrequiredfororderingit,authorities safeguards willproveeffectiveinaparticularcasedependonthenature,scope 289 human rights. lection ofpersonaldata,includingextraterritorially,canhaveontheenjoyment the negativeimpactthatblanketinterceptionofcommunicationsandmasscol- In recentyears,leadinghumanrightsactorshaveexpressedgrowingconcernabout area orinageographicallyunboundedmanner(the‘hunter-killerscenario’). with therighttorespectforprivacy, or otherpublicauthoritiesrelatingtoanindividual’slifeconstituteinterference be putinplaceorderforindividuals‘nottotreatedasmereobjects’. When processingpersonaldata,humandignityrequiresthatadequatesafeguards interference withtherighttoprivacybutstrictconditionsapply. acts, ormorebroadly,thesafeguardingofnationalsecuritydefencecanjustify so farastheyarestrictlynecessaryforsafeguardingthedemocraticinstitutions. in thenameofnationalsecurity,powerssecretsurveillancearetolerableonly December weaponize Extraterritoriality’, Robot United also Concurring Department of Defense Fiscal Year (FY)2005 BudgetEstimates , Ibid, ECtHR, UNGA Consider discussion discretion Technoscience’, D. Soldiers’, §42. 23 Wilson, Nations to Res Klass etalvGermany, 2015, to such April identify Opinion R. migrate See 68/167, to 290 of Leheny, ‘Military systems §§231–232; 2014, High supra subject also such Thesystematic collection andstoringofdatabysecurityservices CCCB specific Routledge, of Conference into ECtHR, Commissioner 21 §22. imaginaries, fn Judge 10–11 ‘DARPA’s Surveillance’, and persons January civilian 241 Several persons Roman Zakharov v Russia, then ECtHR, 292 November Wildhaber (envisaging Thedetectionofseriouscrimeorpreventionterrorist App 2012, within Lecture application 2014; Urban perhaps complaints see Rotaru v Romania, for suspected no in p their 5029/71, S. The Right to Privacy in the Digital Age, Human K. Operations 274 2005, at (joined Graham, Ball, ‘[t]iny go 291 the jurisdiction should (cautioning the afundamentalhumanrightthatservesto about of K. available Rights, Symposium by surveillance Judgment next D. terrorist ‘Technologies six alert Haggerty Programs’, mass step other App UN Grand to at that us acts, secret doc surveillance to (Plenary), publicspace.org, no to ‘Arxipelago judges) Anderson robots ‘the and allow Chamber, the 47143/06, A/HRC/27/37, and DARPATech, of surveillance’ February D. long-standing possibility Exception: them noting equipped Lyon (noting 6 ‘Combat are and September of Judgment, to (eds), Grand Exceptions that pending 2004, Waxman, act http://www.publicspace. 9–11 that of 30 would Urban Report with Zones ‘[i]t autonomously’). technophiliac Routledge Handbook Chamber, June tendency ‘States August pp 1978, 4 is before facial risk UN Warfare 170–171, May not ‘Law that of 2014; - Sovereignties doc ‘undermining do the §49 a 2005, recognition 2000, 293 Judgment, See’, the of and large 294 not CCPR HRCttee, Office What (‘an military dreams 289 and Even http:// ECtHR. Ethics enjoy in For p step §47 un- /C/ 295 US US 38 of a

made of State for theHomeDepartment vTom Watson and Others, 297 C-293/12 andC-594/12, Grand Chamber, ECLI:EU:C:2014:238. Judgment,8April2014, Profiling Mechanism, Regard 300 SchmittandThurnher, ‘“Out of theLoop”’, supra fn19,268. 299 Ibid,§§106–110. 298 Chamber, December Judgment,21 2016, ECLI:EU:C:2016:970, §103 (emphasisadded). , v Minister for Communications, Marine and Natural ResourcesIreland and Others and Others 301 fra_uploads/1133-Guide-ethnic-profiling_EN.pdf. Preventing Discriminatory EthnicProfiling: AGuide, surveillance Rec(2010)13), §§1(d)and(e). terrorism tered to themorpredictingtheirattitudebehaviour. teristics that are not observable with a viewto taking individual decisions relating viduals on the basis of some observable characteristics so as to infer other charac- The miningofmassivedatasetsoffersnumerouspossibilitiesforcategorizingindi- terrorism orparticipationinhostilities,butwhoseidentitiesneednotbeknown. group ofpersonslikelytoposeasecuritythreat. tiating andlimitingsuchmeasurestoaparticulartimeperiod,geographicalareaor a relativelyabstractlevel,exante,andingeneralizedmanner,ratherthandifferen- arguably, furtherthetrendtowardjustifyingnecessityofsecuritymeasureson lematic fromahumanrightsangle.Also,relyingonautonomoustargetingwould, of securitymeasuresbuildsonpracticesthatarealreadyidentifiedasdeeplyprob- in detail,sufficeittounderlineherethatthealgorithmicconstructionoftargets cutting-edge 296 life analysis”ofthetargetarea’. It hasbeensuggestedthatAWS‘maybeprogrammedinpartbasedon“patternof 2. S necessary for the purposes of that fight’. indiscriminate retention of all traffic and location data should be considered to be be, it‘cannotinitselfjustifythatnationallegislationprovidingforthegeneraland objective ofgeneralinterest’thefightagainstterrorismororganizedcrimemay European Union(EU)CourtofJusticemadeitclearthathoweverfundamental‘an data bytelecommunicationprovidersandaccessoftheauthoritiestothisdata, In arecentcaseconcerningnationallegislationrequiringtheretentionofpersonal in ‘theunlimitedsurveillanceofalargenumbercitizens’isnotacceptable. tive viduals (orgroups)whopossesscertainattributesthatarebelievedtobearposi- 73. large nications’ interception’ The orting People CoE number statistical Court ECtHR, executive are The to Court Committee at and applied Automatic aim of bay techniques Szabó andVissy vHungary, is technologies’ considers of ‘automated Justice ‘a is power … citizens’. matter ‘to correlations toparticularkindsofconduct,suchasinvolvementin automatically if the Processing of aid supra of intruding Ministers and terrorist it the of a In and ‘a in decision fn serious prerogatives’. the natural European pre-empting systemic 286, and of Court’s threat Recommendation into Personal concern’ pp more consequence’ modifying citizens’ were 3, Union, data view, 299 App 11); See effectively’ terrorist Insuchascenario,anAWSwoulddetectindi- Data collection’. in paradoxically no ‘it FRA, Towards More Effective Policing, Understanding and private also Tele2 Sverige ABvPost- ochtelestyrelsen andSecretary that a 37138/14, would in course of 297 CM/Rec(2010)13 attacks, Court the it 2010, ‘present-day (Dinant Without analysing this important case spheres paves Context defy But of of substituted Judgment, p including Justice for action the the 298 9, 300 et by the of https://fra.europa.eu/sites/default/files/ Thisconstitutes‘profiling’. al, Case purpose way virtue Profiling, terrorism Court, of Application of Convention 108 to the while on ‘the for nos the 12 for the of January ‘the massive of ‘so-called European ensuring by C203/15 uncontrolled Protection 23 government that a unlimited November perceived 2016, governments monitoring that and strategic, Union, of §§56, surveillance C698/15, yet Individuals choices threat efforts 2010 Digital Rights far-reaching of large-scale 67–69, Case (CoE resort of commu- to already 296 Grand unfet 301 keep with of CM/ nos 71, to

a -

Human Rights Requirements and Constraints on the Use of AWS 59 DEFENDING THE BOUNDARY 60 306 Violence from Afar’, 239–254. supra fn44, 307 Xenophobia and Related Intolerance, UNdocA/HRC/29/46, 20 April2015, §39. wrongly, asbeingdangerous. to their(imputed)membershipinacategoryperceivedbytheauthorities,rightlyor bears aparticularlyhighriskofnegativelyaffectinghumandignityandexpos- Subjecting peopletolawenforcementmeasuresonthisbasisisunlawful. 305 Ibid. 304 Dinantetal,Application of Convention 108 to theProfiling Mechanism, supra fn286,p6. 303 SuchmanandWeber, ‘Human-Machine Autonomies’, supra fn73, p91. 302 ethnic origin,politicalopinions,religiousorotherbeliefs,healthsexuallife, genetic orbiometricdata,personaldatarevealinginformationaboutracial Moreover, profilinginvolving‘specialcategoriesofdata’(sensitivedata)suchas conduct, as anylawandorderdeterminationshouldbebasedonanindividual’spersonal ing onarangeoffreedomsandrights,includingeconomicsocialrights.Where- plex, opaquealgorithmsthisrisksadverselyaffectinghumandignityandencroach- When publicauthoritiesbasedecisionsontheprocessingofpersonaldatabycom- adequately, as“patternsoflife”’iscoursequestionable. position that‘relevantcircumstancescanberenderedalgorithmically,andstill correlations and inferences alwaysentailsacertain error rate.The very presup- puted) membershipinthatcategory.Thecategorizationofindividualsbasedon sis ofinformationpertainingtothemnotasindividuals,butbecausetheir(im- Profiling placespeopleincategories,usuallywithouttheirknowledge,ontheba- based onthepreviousbehaviourofotherindividualswhomtheydonotknow.’ their futurebehaviourthatisnothingtodowiththemandnomorethanaforecast measures involvingarmedforce. facilitates thetranslationofbodiesintotargetsforsecuritymeasures,including for decision-making’. the profilegeneratedbyautomatedprocesseswhichareliabletobeusedasabasis be criticizedforitsdehumanizingqualityinthat‘ittendstoreducetheperson vation on or Art6,Draft Modernized CETS108, supra fn287. 310 organization cannot form asufficientbasis for asuspicionjustifying thearrest of anindividual). ed 309 Dinantetal,Application of Convention 108 to theProfiling Mechanism, supra fn286,p34. 308 sports events’ (Ibid,§80, emphasisadded)). was subjection a category having in entries impermissible a E.g. On ECtHR, Report of ‘Surveillance this liberty ECtHR, propensity 306 of on of Shimovolos vRussia, a aspect, algorithmicprocessesputindividualsundercategoricalsuspiciondue the individuals of human for Ostendorf vGermany, the applicant as preventive Database’ in to it rights Special relation unlawful amounted who 303 in activist Thisis‘oneofthemostacutedangersprofiling’ set purposes a Rapporteur are to police acts. up App armed perceived to 307 to by ‘a no App law ‘Individualsthereforefacetheriskofaprediction The database Russian lawful, policy 305 30194/09, drones, enforcement Court no on by 15598/08, Contemporary not of authorities the on underlined see, general least persons authorities, Judgment, in measures because particular, Judgment, prevention’ to that prepared facilitate Forms 21 rightly the mere on June Wall 7 police the f Racism, of March discovery directed to ‘membership’ or 2011, 302 and basis use wrongly, Theapproachcanalso ‘had Monahan, 2013, §§54–56 violence that against not of Racial §66 his ‘potential

as based in name being ‘Surveillance in (finding (finding an a Discrimination, the human their 309 individual was extremists’ dangerous context 304 findings that a includ- depri- asit rights and the 310 308 or of

313 of-sentencing#.mku7Zs9Ym. The MarshallProject, predictive-assessments; M.Barry-Jester, A. B.Casselman andD. Goldstein, ‘The New Science of Sentencing’, ment The Guardian, igin veillance with Kingdom, Ruin (‘profiling Xenophobia lives/; computer-predicts-crime-but-is-it-racist; The Verge, Comment no 18: Non-Discrimination , pressing 8 origin, gender,sexorreligion. rights andfreedoms,solely(ormainly)ongroundsoftheirrace,nationalorethnic to surveillance,identity checks ortheuseof force, and thus interfere with their risk assessmentsinthecriminaljusticesystem. dictive policingandthecompilationof‘no-flylists’byexecutiveauthorities, of algorithm-baseddecisionsabound,fromcreditscoringbyprivateactors,topre- Fundamental that 312 processing 311 harshly criticizedinthecounter-terrorismcontext. al originstereotypesinordertoidentifypotentialthreatsandvulnerabilitieshasbeen Targeting securitymeasuresbasedonbroad-brushracial,ethnic,religiousandnation- ing individualstodiscrimination. tionate andnecessarytoachievealegitimateobjective. compared toanotherinamannerthatcannotobjectivelybejustifiedaspropor- in practice,theymustnotmainlyormorenegativelyimpactoneparticulargroup tional factorsthatgive‘reasonablegrounds’fortargetingaparticularperson,and, 2.4, on discrimination the cessing 315 2016/680 of theEuropean Parliament andof theCouncil of 27 April2016). latif-et-al-v-holder-et-al-aclu-challenge-government-no-fly-list. www.aclu.org/cases/latif-et-al-v-lynch-et-al-aclu-challenge-government-no-fly-list?redirect=cases/ Holder, and 314 UN docA/HRC/15/53, 12July2010. legal and the Police CoE ‘Latif, or results Preamble, Lives’, See the States bases, M. agencies); Report S. Ongoing 10 of religion Stroud, concern Committee 20 Ackermann, programs, e.g., principle App sensitive based Sector, in et is 19 Wired must in Arts and enshrined, respect al. October of UN Freedoms February in discrimination no CoE Report Serious ‘The 10 are v. Related law 2 about on the , put doc 17 24960/15 19 Lynch, of and data CM/Rec(2010)13, complaining August particularly of stereotypical September Minority and in of November ‘No-Fly 8 A/HRC/29/46, non-discrimination’); Special of Ministers, racial 26, 2011, place Implications e.g., 2014, April requires foreign Intolerance While in the et ICCPR; fact al. in 2015, (pending, Special List and 2015, effective is Report: Rapporteur http://www.theverge.com/2014/2/19/5419854/the-minority-report-this- https://www.aclu.org/files/assets/aclu_letter_to_ag_re_rm_102011_0.pdf Article – prohibited in 1987; Countering 2014, nationals); additional likely of ACLU Recommendation Uses ethnic assumptions Arts any https://www.theguardian.com/us-news/2015/aug/10/us-no-fly-list- https://www.themarshallproject.org/2015/08/04/the-new-science- a 10 of Rapporteur supra on supra Chicago’s 1, violation 313 Art https://www.wired.com/2014/11/algorithms-great-can-also-ruin- safeguards field to “Predictive Islamophobia, November lodged 1 Challenge CETS the profiling L. and Tobelawful,suchmeasuresmustrelyonaddi- commit 3.11, 311 fn on fn safeguards Dormehl, (Art ECtHR, 10 HumanRightsOrganizations etalvThe United regulated American Manifestations Terrorism, 108. 24, 306 Accountsofunfairandstigmatizingimpacts 301. the on CoE that of New 6, on being AmCHR; to 1989, See against 20 (looking The Art crime Assessments” Draft Contemporary CM/Rec(2010)13, Promotion no persons Government Police ‘Algorithms for May and also Civil 14 and right R(87)15 institutionalized UN §12). Modernized may the (non-discrimination) discrimination Art protected 2015, UN profiling at of Liberties Computer to doc Enjoyment 312 of 315 ethnic The Defamation 2, doc equality lead and Regulating Itisillegaltosubjectpersons a Itisdoubtfulthattheseverein- AfCHPR; concerning A/HRC/4/26, Instead Are No principle Forms certain A/HRC/29/46, through to and Protection by Union, supra 314 CETS Fly Great Predicts and practices public through of of Racism, in racial List’, of Art of “race”, 108, of purpose all the fn non-discrimination and automated Hard Letter the Religions, 14, non-discrimination 301; Rights 20 authorities profiling Crimes, in 29 Use supra of risk All, NSA’s that supra ECHR; national conjunction August Evidence, Racial Art to January Human and of profiling But by Attorney are fn Personal 11, but secret and fn decision HRCtte, by their effect. Discrimination, They 287; 2015, (among 306, Directive incompatible or law is in US Rights 2007, it Followers, software). ethnic mass Particular with Can prohibits Principle §22 The enforce- Racist?’, Admits’, General General https:// making Data in other data Also pro- (EU) §34 Arts (ex sur and or in - - -

Human Rights Requirements and Constraints on the Use of AWS 61 DEFENDING THE BOUNDARY 62 ate riskofextrajudicialkilling. of aparticularagegroupandreligionincertaingeographicareastodisproportion- pinning ‘signature strikes’ are gendered and racialized in a manner that exposes men tinction alsoappliestopersonsdirectlyparticipatinginhostilities. Geneva Conventions, applicable in certainNIACs,the prohibition onadversedis- definitions ofpersonsorobjectsthatmaylegallybemadetheobjectattack, challenged onthegroundsthatcategoriesuseddonotmapexactlyontolegal on thebasisofanumberobservable,behaviouralorother‘signatures’,hasbeen drone strikes purportedly connected to an armed conflict, where targets are selected 316 ing arenotlimitedtoAWS.However,algorithm-baseddecisionsreflectstructural As forthehumanrightschallengeslinkedtosurveillance,risksposedbyprofil- or similarcriteriaisprohibited. in armed conflict. Adverse distinction based on race, sex, religion, national origin hostilities frameworkbuttheprincipleofnon-discriminationcontinuestoapply As notedpreviously,thereisscopefor‘categoricaltargeting’withinaconductof justified asnecessaryandproportionatetosafeguarddemocraticinstitutions. terference withtherightsofeveryoneaffectedbyalgorithmictargetingcaneverbe behaviour justifyingattack. for being based on insufficient evidence that the targets exhibited characteristics or sure the tension. of from complainant’s concerned only bejustified by a‘concrete threat’ to nationalsecurity orto aperson’s life, limborfreedom. The case ing on wake of theterrorist attacks of 11 September 2001 withaview to terrorist identify ‘sleepers’ inGermany quirement secrecy stigmatization facts Federal 318 Common Art3,GCI–IV;75, API;Art2(1)II. 317 http://www.bverfg.de/e/rs20060404_1bvr051802.html). or 320 the ‘inabilityto contextualize’ asasource of disproportionate civilianharm(37). Law’, 321 Journal of International CriminalJustice 1 (2013) 1-22. org/images/documents/Publications/sex-and-drone-strikes.pdf. 319 (eds), Foreign Fighters UnderInternational Law andBeyond, T. Asser M.C. Press, 2016, p256. Casualty Analysis, Habitual a the sensitive fast person. could in S. In human K. 27 R. See, basis a Krähenmann, Benson, 2006, of Constitutional To processing Acheson, Pacific McGeorge Global Business &Development Law Journal1 concrete not of the e.g., the be Residence, The of dignity personal a compatible be rights. the measure and automated such concrete K. ‘“Kill Court justified J. R. case German Reaching increased of criteria Heller, and Moyes ‘The ‘em Other State considered large, data), Court, indicate and threat the and with Obligation by processing Federal ‘“One as of aggravating and its Critical complex the reference right risk Beschluss Sort Transit age, the does wide with T. linking Hell that of 320 fundamental it to Constitutional educational Nash, Will not sufficient Out becoming Under Ithasalsobeenpointedoutthatthecategoriesunder- scope and the 321 datasets, of of the to 317 demand des of Later!”: massive factors a and Sex andDrone Strikes: Gender and Identity inTargeting and State a free Remarkably,underAdditionalProtocolIItothe automated, Killing International datasets, general (affecting Ersten Article probability development of the enrolment, contributed right included Signature an Court amounts Machine”: Destination’, subject terrorist imminent Senats, 36, to the between computerized held informational Law October potential that the Drone faith, of of to 4 Signature or that of of data or unjustified types a April in security the one’s 200,000 the concrete country present A. a Strikes 2014, severity secret, for by de Foreign 2006, of nature personality, self-determination, Guttry, the Strikes adverse threat personal (2014) http://www.reachingcriticalwill. and of threat, threat and suspicion automated German 1 birth of Fighter’s of International BvR F. and or 300,000 the the Capone consequences, 31–32. will but a and data such 518/02 measure, 318 situation International interference Federal or materialize it State nationality Thelegalityof data-mining demands collected investigation, Benson a and people). measure a - Humanitarian of right Police C. of allowing Rn. Nationality 316 Paulussen increased identifies including (German with 319 that Law’, and (1-184), (includ- derived The in could and mea- sex the the the the for re- 11 adequately encoded ex ante). potentially Weber, 328 LieblichandBenvenisti, ‘The Obligationto Exercise Discretion’, supra fn81, p276-277. 327 Swiatek, ‘Intending to Err’,supra fn324, 247. 326 Stoddart, ‘A Surveillance of Care’, supra fn323,p375. 325 Information Technology 4(2012) 241 (emphasisadded). arXiv:1611.04135v2 329 Lethal Decision-Making’, 94IRRC 886(2012) 695. ‘On and 324 Lyon (eds), Routledge Handbookof Surveillance Studies, Routledge, 2012, p375. 323 technology, purportedly ‘free of any biases’). 322 tive human intervention’ Others haveeffectivelycritiquedautonomoustargetingforprecluding‘delibera- results’, ‘[n]o algorithmicsystemofanyusefulcomplexitywilldeliverperfectlyaccurate Whereas conduct of hostilities-oriented narratives aboutAWS readily acceptthat 3. Calculated affecting humandignityandequalityinpresentpractice. tential toreinforceessentialiststereotypesthatarealreadyrecognizedasadversely discrimination. but whenmachinesaremadetolearnfromdatasets,theyreifyexistingpatternsof biases insociety.AWSdonotthemselvesmakelegaljudgementsordiscriminate tion ofthesystem’. requires thedeliberatetargetingofnoncombatantsasastatisticallynecessaryfunc- tolerance oferrorneedstobedefined.Putdifferently,‘probabilisticmatching… tral act, acceptance, into accounttheinaccuracyofunguidedartillerywhenorderingshelling,and are required‘toreflexivelyconsidertheimplicationsoftheiractions,andtoapply imately delegatedtoanautomatedprocess’becausehumanagentswhouseforce explains, ‘theauthoritytodecideinitiatetheuseoflethalforcecannotbelegit- and anoutcome’. which isadeliberateactthatsitssomewhereincausalchainbetweenvolition ly foreseen because they result from the meticulous programming of the device, Failing www.nytimes.com/2016/06/26/opinion/sunday/artificial-intelligences-white-guy-problem.html?_r=1 Banning X. E. K. M. Lieblich Heyns, to Zhang, in Stoddart, Crawford, S. 323 325 recognize ‘Human-Machine Swiatek, relevant thesetendtoobfuscatethatmakeautonomoustargetingwork,some andonsomelevel,‘wrongfullytargeted’peopleare‘morethanmere- ‘Human Autonomous and Automated Inference on Criminality using Face Images, ex ante,ofapercentagefalsepositivesinautonomoustargeting. ‘A ‘Artificial Benvenisti, Blindness ‘Intending 322 Surveillance circumstances’, [cs.CV] these Rights 326 Autonomoustargetingthreatenstoperpetuateandhasthepo- 324 Thereisadifferencebetweenhumanagentshavingtotake structural Calibratingsensortechnologyis,thus,notanethicallyneu- Weapon Intelligence’s Autonomies’, (proposing and ‘The to 328 of Err: the Obligation and the exercise of discretion. Care: where Systems: The biases Use the Evaluating Ethical White of supra relevance bears automated Autonomous to Human Guy fn Challenge Exercise a 73, Surveillance Problem’, significant is p Rights, continuously, prediction 92 Discretion’, Weapons of (describing Lethal, The New York Times, Automation, risk Ethically’, of of Systems’, Autonomous supra socially criminality regression the last 329 in fn world and K. revised Even in war, as Asaro constructed supra 81. Ball, the Similarly, as Systems’, using as 25 K. fn 21 illustrated ‘an Dehumanization D. June 22, November face open Haggerty and 370; Suchman 14 2016, recognition horizon Ethics and cannot by P. http:// and Asaro, X. 2016, 327 and Wu be of of D. .

Human Rights Requirements and Constraints on the Use of AWS 63 DEFENDING THE BOUNDARY 64 335 penalty assuchconforms withIHRL. 336 Headnotes, §(ix). p 337 Academy 12.126 (Dwight Fletcher) and12.146 (Anthony Rose), supra fn333,§203. 338 no CCT/3/94, Judgment, 6June1995,§26. 334 Tobago, Decision (Comm no845/1998),UNdocCCPR/C/74/D/845/1998 (2002), §7.3. as (Carl Baker), 12.126 (Dwight Fletcher) and12.146 (Anthony Rose) vJamaica, 333 police hadnot solved other problems around thecity’(emphasisadded)). nated bythestate’, the convictedpersonofalldignityandtreatshimorherasanobjecttobeelimi- members of afaceless, undifferentiated mass’, dignity oftheindividual’. to cruel,inhumananddegradingpunishment. objects’, Garlicki, Ser.L/V/II.106 330 higher court’. of criminalresponsibility,andcannotbe‘thesubjectaneffectivereviewbya eration of eachindividual case’, including mitigating factors and different levels the deathpenaltyforparticularcategoriesofoffences,precludes‘reasonedconsid- In thecontextofjudicialexecutions,forexample,mandatoryapplication consequences areirreversible. cumstances isunlawfulinmanysituations.Itparticularlyproblematicwhenthe compassion and judgement in an explicit appeal to their humanity’. tation astotheirscopeorduration, individuals to measures of ‘an automatic nature’ withoutan explanation or limi- of thelaw’, ical, automaticapplicationofthedeathpenaltyas‘blindadherencetoletter 37331/97, drawal 332 authorities hadascertained theindividualwould not suffer unduehardship). Chamayou, sibility is 331 limitation to armed violence: thecritical conscience of itsagents’). 36 Oxford ‘Faith, and Journal of International Law) (LJIL 3. reduced a (automatic The result misjudgment US ECtHR, Constitutional IACommHR, Cases 12.023 (DesmondMcKenzie), (Andrew 12.044 Downer yAlphonso Tracey), 12.107 (CarlBaker), Asaro, Supreme E.g. A. Identity, of of Circus Supreme §10 focus Priddy disobedience of passport); Gillan andQuinton, Judgment, 332 to of Drone Theory, ‘On International Battista vItaly, (arguing being andfailingtocarryoutafreshreviewtakingaccountofindividualcir- time-limited doc 335 at on imposition Court and Banning around thattreatspeople‘notasuniquelyindividualhumanbeings,but and Court, 333 the Cases 12.023 (DesmondMcKenzie), (Andrew 12.044 Downer yAlphonso Tracey), 12.107 3 are synonymous Court the rev Thishasbeenfoundtoproducearbitraryresultsandamount ECtHR, 4 mandatory of M. that part June comes Woodson vNorthCarolina, Killing at 337 India, Autonomous 2p.m. of Mattirolo, of Humanitarian supra designations 918 ‘kettling’ andthatisirreconcilablewith‘theessentialrespectforthe Stamose vBulgaria , of App the a 2002, travel supra ‘at Mithu etalvState of Punjabetal, of the were (1999), Republic fn with 338 no application the the 1, (1997) §70 law,

43978/09, was ban). ‘The fn p treated cost the Innocent: Weapon 218 §§194–196. 265, (finding of Law however ‘applied most of Mandatory of In 159–160 certain (‘in South simultaneously contrast, Joint and of like objects 331 lobotomized redefining Systems’, the Judgment, International App no Human

Judgment, indiscriminately’ much areas Africa, treating allpeople(indiscriminately)‘like Dissenting disproportionate (noting death See no Death e.g., 29713/05, of it supra also T. Makwanyane and M.Mchunu v The State, Rights, and “ethical” penalty 336 ECtHR, is Amsterdam 2 334 that discipline suppressing 2 December Penalty thatis‘degradingbecauseitstrips dressed HRCttee, Opinion Lawyers were July Courtshaverejectedthecategor- fn ‘[d]iscretion Final 328, Landvreugd vThe Netherlands, is 1976, as Judgment, as Judgment, force not restriction conforming Under ‘all or Draft, in 689, of and Mr Rawle Kennedy vTrinidad and as 2014, 428 meant docility’. the the Judges people to ‘emergency 700; Nuclear February principal remain US and International voice 27 §§47–48 7 to on 280 See April Report who Tulkens, November “evaluation”, Ruling mechanically suggest freedom of Weapons’, (1976), there also 2013, source happened areas’ 1983, universal 330 out (automatic no M. Spielmann that as Subjecting Law’, Syllabus, of Koskenniemi, the unpublished, 41/00, 2012, 2 because of long movement even SC 10 to the infralegal very reason’); to Geneva rules, App R Leiden §§33– as death with- be OEA/ error §(d). 690, Case pos- and the the no at it for of democratic derlying of Convention 108 to theProfiling Mechanism, supra fn286,p34. refer and that 342 HRCttee, General Comment no16:Article17(Rightto Privacy) , 8April1988,§10. 341 enization upondifference’). Opinion surveillance Modernized on that particular category. out anopportunitytochallengesuchadecision,includingone’sassignment icantly affectingapersonbasedsolelyonanautomatedprocessingofdatawith- in theEuropeancontextrecognizerightnottobesubjectadecisionsignif- es involvedintheuseofanAWSisaffronttohumandignity. procedural level.Inthisreading,thecalculatedblindnesstoindividualcircumstanc- nores theobjectifyinganddehumanizingpotentialofautonomoustargetingona gramming ofalgorithmsasanadequateimplementationlegalobligationsig- automated decisionisakeyelementofdueprocess. to beheard‘atameaningfultimeandinmanner’challengean 340 Sassòli, ‘Autonomous Weapons andInternational Humanitarian Law’, supra fn248, 332. 339 controls thatdataandtorequestrectificationorelimination. sonal dataisstoredinautomaticfiles,andforwhatpurposes,toknowwho have therighttoascertain‘inanintelligibleform’,whether,andifso,whatper- volvement areessential.Ithaslongbeenrecognizedthateveryindividualshould To policetheproblemsraisedbyautomateddecisions,humanoversightandin- 4. Process Matters Even ifextrajudicialkillingisnotabout‘renderingjustice’, explanation ofthedecisionreachedortochallengedecision. guards includetherighttoobtainhumanintervention,inparticular,receivean unless authorizedbylawandonlyifsuitablesafeguardsareprovided.Thesesafe- presumption thatsubjectingpersonstosuchautomateddecisionsisprohibited, for police and criminal justice purposes, which takes effect in 2018, establishes a supra technologies 343 no-fly list asunconstitutional). al 344 2016/680, supra fn314. security the an the the Monahan, ‘identification’ explicitly Preamble, On US Art fn automated fundamental protection Directive and 289, data the District 8(1)(a), issues society’ CETS that Order, tension and processing p to ‘Surveillance 274; §38 accord Court does (Preamble, processing human 108 the Draft of predicated pursuant 24 Chamayou, rights national and between creation nor not for June Modernized with 343 where intervention, Art §73(a), apply the and AnEUDirectiveconcerningtheprocessingofpersonaldata and of 2014 11, §14). to the security the of District on data the Art freedoms Drone Theory, Directive to Violence precepts blindness Draft an dominant (striking CETS national 9(1)(a), results and CETS analysis or and of Explanatory 108 the defence (EU) Oregon, and 108, from of of both Draft to down security right ‘techno-fantasy’ has such categorical 2016/680, of supra critical constitutes supra Afar’, the Modernized behaviour a when ‘to the Latif etalvHolder, processing broader Report activities right fn obtain, differentiations, fn supra existing 1, this suspicion 287; p supra not a 42 to scope ‘is patterns fn on necessary CETS of or are to Draft (explaining Preamble, provided redress fn 44, request, achieving to be applied 314 of 108. agencies and 240 Modernized subjected 342 application is (emphasis see Recentregulatoryefforts procedure social and not See for knowledge (describing §38 to in omniscience Wilson, individual by also proportionate him or relation Case sorting to and 339 units law, 341 CETS Dinant or but a added). treatingthepro- 340 no 344 Theopportunity against ‘Military decision Art her’ respects of dealing to drones neither 3:10-cv-00750-BR, that 108 but the ‘signature 11, trough et can Note, (supra al, ‘force inclusion generic); Directive reasoning measure Surveillance’, be based with the as Art Application all-seeing restricted however, forms essence homog- 8, fn nation- strikes’ solely Draft 287) on Wall (EU) in un- of a a

Human Rights Requirements and Constraints on the Use of AWS 65 DEFENDING THE BOUNDARY 66 used algorithmictechniques’, ‘a right to explanation’ could ‘require a complete overhaul of standard and widely features relatetopredictionsandcategorizations. agents mustbeinapositiontoprovideconcise,intelligibleaccountofhowinput security conditions,includinginacontextofarmedconflict’. This proceduralobligationundertherighttolifecontinuesapply‘indifficult mous systemsmakedecisions’. may notnecessarilyalwaysbeabletounderstandhow(andpossiblywhy)autono- As mentionedearlier,however,‘itisbecomingincreasinglyclearthathumanbeings der IHRLandtherighttoaneffectiveremedy. This isamajorworryfromtheperspectiveofstates’proceduralobligationsun- rithms arenotbuiltwithaconcernforcausalreasoninginmind. 347 Draft Explanatory Report to Draft Modernized CETS 108, supra fn287, §75. 346 Art8(1)(c),Draft Modernized CETS108, supra fn287. 345 derlying decisions based on the processing of one’s data, European lawalsorecognizesthepressingimportanceofknowingreasoningun- use offorce,thereshouldbesomeformeffective,independentinvestigation. use offorcebytheauthorities.Whenindividualshavebeenkilledasaresult tive in practice if there existed no procedure for reviewing the lawfulness of the plained, thelegalprohibitionofarbitrarykillingbystateagentswouldbeineffec- volving the use of algorithms for automated-decision making including profiling’. to the identification andpunishmentofthose responsible. nation ofwhethertheforceusedwasornotjustifiedincircumstances,and an investigation,amongotheraspects,hastobecapableofleadingadetermi- 352 February 1998,§91. countable_for_their_algorithms.html 2016, 351 McCannetal,supra fn 93, §161; Al-Skeini , supra fn100, §163. 350 Explicitin,e.g., Art2(3),ICCPR; Art13,ECHR; Art25,AmCHR. 349 GoodmanandFlaxman,‘European UnionRegulations onAlgorithmicDecision-Making’, supra fn75, p1. 348 AWS’s calculation’). Act’). February 1; digital.law.washington.edu/dspace-law/bitstream/handle/1773.1/1318/89WLR0001.pdf?sequence= 25 89 prisingly decisions (asserting the 353 cants ‘any realistic possibility …to challengetheconclusions’ of theauthorities’ account (emphasisadded)). questions to N. distribute July Washington Law Review conduct Diakopulous, In Anderson See Jaloud, Al-Skeini, this http://www.slate.com/articles/technology/future_tense/2016/02/how_to_hold_governments_ac 2013, few 2005, by also about the vein, AWS of attempts supra convicts §848 need ECtHR, et hostilities, supra §§221–223 D. compliance al, should ‘How Keats fn to ‘Adapting (finding Khodorkovskiy andLebedev vRussia, have 123, among fn to to Citron be find 100, Hold see §166. recourse (finding interpretable with a an explanation (2014) the prisons Margulies, and §164. violation Governments Consider, Law IHL F. Pasquale, that to (proposing 1–33 See principles, of and 348 347 correct of Armed the ‘Making ascommonsupervisedmachinelearningalgo- AccordingtoGoodmanandFlaxman,recognizing and also Art to (detailing e.g. for investigation Accountable describe transparent, ‘The errors, 8 ECtHR, ECtHR, Isayeva v Russia, Conflict these the ‘a due Autonomous Scored state introduction to ‘technological and serious Kaya v Turkey, to the the should Society: Autonomous for for was and 349 method App failure interpretability, the and AstheECtHRhasrepeatedlyex- not Weapons that 353 be nos of Algorithms Due credible Inhumanrightsproceedings, effective, to able a if or due 11082/06 ‘Freedom a ‘explain Process 345 Weapon App nomination algorithm App to process’ Accountable’, ‘in particular in cases in- allegations’, 352 present no and no They and Accordingly, human for how’ and of 22729/93, Systems’, 57950/00, noting 351 Automated noting Information used). requirements) Use’, a is 13772/05, a Tobeeffective, clear mistaken plan denying supra Slate, that that supra With account was Judgment, Judgment, it Predictions’, ‘nomination 11 fn Processing Judgment, ‘made respect fn and drawn the February 85, 19, https:// of raises appli- p 394. sur the 350 346 24 up 23 19 to -

Judgment, explanation 355 Summary orArbitrary Executions, 13September 2013, §98. UNdocA/68/382, can targets that surveillancewasappliedtothem. surveillance’ withoutthatpersonhavingtodemonstratetheexistenceofarisk an effectiveremedytoanyonewho‘suspectsthatheorshewassubjectedsecret of ‘anyperson’canbeintercepted;and,two,thedomesticsystemdoesnotafford targeted’ orbecausethelegislationinstitutesasystemwherecommunications can ‘possiblybeaffected’,‘eitherbecauseheorshebelongstoagroupofpersons 354 if twoconditionsaremet: challengeable andoutsideofjudicialsupervision. Concerns havebeenraisedthatautonomoustargetingrisksbeingeffectivelyun- assist victimsandsocietyatlargeintheirquestforthetruth. dures toallowanindependentassessmentofthelegalityuseforceand the onusisonstatetoprovidesufficientdetailsitsdecision-makingproce- particular circumstancesviolatestherighttolife. resulting deathsaprioriincapableofdeterminingwhetherforcewasjustifiedin follow that using force by means of atechnology that rendersaninvestigation into mere existenceofsecretsurveillancemeasuresorlegislationpermittingsuch individual canclaimtobeavictimofviolationtheECHRoccasionedby those raisedbysecretsurveillance.Inthatconnection,theECtHRhasheldan target parametersandsensorweaponsrangetotheopportunitychallenge ployment ofanAWSinareawouldentitlepeoplepotentiallyfallingwithinits Independent result UN was of puted Judgment, Judgment, class of people who riskbeingdirectly affected by thelegislation’). 16066/90, if mediate 49458/06, measures an 358 Roman Zakharov, supra fn295,§163; Klass etal,supra §34. fn294, 357 HRW andIHRC, MindtheGap,supra fn114, pp27–29. 356 Treaties’, supra fn95,pp214–233. ignated they men in-depth doc draw accompanied If This Roman Zakharov , in or are a and A/HRC/29/CRP.4, ‘security who risk a government’s cannot of inferences concerns, violation ‘required 29 18 24 Decision, 16068/90, in precautions discussion implementation, to disappeared Commission September July January a the human be risk by 1988, life established. of either areas’ notably, 15 in widespread the failure of 2008, of favour 16069/90, May rights incorporated Series supra a 24 2009, the of to right ‘in of man, 2012, June Inquiry if modify information to Amsterdam, life-threatening §§84, duty proceeding of individuals C 357 fn to submit See, §§184–186, including no the arrests 2015, one,thescopeoflegislationissuchthatperson §60 life, 295, to 16070/90, 4, their 92 Established e.g., applicant in account, including §§127-146). information (concerning (failure §171. §§216-218; such and ECtHR, Varnava etal v Turkey, about conduct the ‘run prevents failure 191, killings’); Court circumstances criteria. See the see 16071/90, (e.g. 358 to targeting in Pursuant to 194 risk or Itisarguablebyanalogythattheveryde- take also, or cases Krähenmann, a recognized A the launch IACtHR, Report risk court failure otherwise of (failure See, ECtHR, application reasonable e.g., being being where decisions, 16072/90 to an from e.g., of to where Case of Velásquez Rodríguez vHonduras, 355 ECtHR, Human to 356 investigation Osmanoglu vTurkey, directly victim account the prosecuted, to Report the reaching account Thechallengesarenotunlike ‘Positive provide of Special measures the involvement including Colon vThe Netherlands, status Rights and ‘preventive affected’ for of App conduct for factual Obligations the a 16073/90, Rapporteur the into or in satisfactory nos Council the to 354 the if the Detailed fate his by prevent they Itwouldseemto whereabouts 16064/90, of of conclusions, search’ absence criteria the disappearance). of App state military Resolution are an in Grand measure, on Findings and no a Human members orders individual agents Extrajudicial, for real of 48804/99, 16065/90, operations convincing individual Chamber, the selecting and and App S-21/1, in that Rights of is court des- fate of dis- can the im- For no is, a

Human Rights Requirements and Constraints on the Use of AWS 67 Human Rights Requirements and Constraints on the Use of AWS 68 a lower rank interms of theirresponsibility). Judge of justice, 363 Ibid,§75; K.-H.W., supra fn203, §§56-57. 362 Streletz etal,supra fn 201, §22). rights, ‘Radbruch 361 tiveness of thesupervisionarrangements’); Szabó andVissy, supra fn296,§88. 360 exercise of itsjurisdiction’). Explanatory by ognized by the community of nations, and which were reflected in domestic law. and worthofhumanbeingswerealreadyatthetimegeneralprincipleslawrec- respect forandprotectionoffundamentalhumanrightsfaithinthedignity in acriminalcourt.Allreviewbodiesrejectedthisargumentonthegroundsthat been impossibleforthemtoforeseethattheywouldonedaybecalledaccount remotely, inautomatedkilling.AllapplicantstheGDRcasesclaimedthatithad that theythemselveshadputinplace. the securityregimetheyputinplace.Norcouldrelyonlawsandregulations obligations enteredintobytheirstateoroftherepeatedinternationalcriticism Political leaders, the ECtHR considered, could not be ignorant of the international 359 has adutytotakeallmeasuresnecessaryeffectivelyprevent anyonepotentially a realandimmediaterisktolife.Tosafeguardlife,thestate deployingtheAWS objective wouldexposeanyonefallingwithintheparametersofavalidtargetto The useofforcebymeansanAWS,inpursuitalegitimatelawenforcement the protection ofthehumanperson,andacerbatesexistingones. for force of use the new of regulation poses international the systems to weapon challenges in autonomy increasing of advent The tive law andjustice present debateonAWS:therelationshipandtensionthatcanexistbetweenposi- Finally, the GDR casesdiscussed above touch upon another issue at the heart of the the legalityofitsdeployment,includingextraterritorially. unlawful, arbitrary,disproportionateordiscriminatoryinterference. meaningless, andthatthesafeguardsinplaceprovideeffectiveprotectionagainst of rightsservealegitimatepurposeanddonotrendertheessence on thestateusingAWStodemonstratethatanylimitationsenjoyment a a lower public In Streletz etal, E.g. E.g. Pellonpää, including the a decision Roman Zakharov, rank Formula’, Roman Zakharov, principle sector Report cannot the Joined entity of supra to right held of 24 Draft ‘show legal 6. C October by 361 falls fn that to 201, Judge and the accountability ofhumanagents involved, however supra Modernized life’ certainty total, directly supra in §103; extraordinary (K.-H. W., 1996, fn Zupančič blind oncluding Remarks fn 295, within may Klaus Dieter Baumgarten, the 295, obedience CETS §284 supra have German (differentiating the §§286–301; 108, cases, (‘it 363 jurisdiction to fn is to yield supra Federal 203, for where orders the precedence §75). fn Rotaru, between Constitutional of 287, Government positive which the Cf, supra § however, Party, supra flagrantly 24 high-ranking to law fn (‘Any that 202, as is to 359 fn Court, intolerably Partly illustrate it of data Theburdenwouldbe 295, infringe §4.2. is objective the relying processing officials Dissenting §69. Even result the recognized inconsistent 360 on See justice security practical and of the carried the also, Opinion agents so-called (cited human Party’s agents effec Draft with out 362 of of in -

AfCommHPR, General Comment no3,supra sE,§31. (emphasis added). fn84, 365 and IHRC, ShakingtheFoundations, supra fn22,pp9–14. including intermsofhowandwhy personsmaybetargetedorkilled.Thealgorith- with legalrequirements,IHRLarticulatesdemandsondecision-making processes, In additiontoadutyensurethattheoutcomeofsecurity measurecomports comes factuallypossibleand,thus,legallymandated. also safeguardtheopportunitytoshiftalawenforcement model whenthisbe- individual attack.HumancontroloverAWSduringtheconduct ofhostilitiesmust in thesenseofcontinuousoratleastfrequent,periodic,human controloverevery make adjustmentsinatimelymanner.Arguably,thiscallsfor activeandconstant, man agents to retain control sufficient to recognize changing circumstances and to all feasibleprecautionsinattack,fromwhichcanbederiveda requirementonhu- tion of legal rules by human agents. Thisincludes,notably,the obligation totake must remainsufficientlyboundedinspatio-temporaltermstoallowtheapplica- ening thecontextofevaluationtothatanattack(asawhole).However,attacks in lightofIHL,thereissomescopeforcategoricaltargeting,whichallowsbroad- In relationtohostilities,whereIHRLstandardsontheuseofforceareinterpreted 364 general public’, capable ofensuring respect fortherightsofanyparticularindividual,aswell as the remain an individualbasis.Forallpracticalpurposes,therefore,humanstateagents‘must there isapresumptionthatdecisionsondetentionmust,inprinciple,bemade limited scopeforamorecategoricalapproachtodetentioninIACs,evenunderIHL as ‘non-lethal’andintendedto‘intercept’ratherthan‘eliminate’.Althoughthereis mous sentrysystem,irrespectiveofwhetheritisequippedwithweaponsbranded essentially thesametypeofindividuatedhumancontrolinuseanautono- son, in the circumstances of everyparticularcase.Compliance with IHRLrequires terfering withtherightstofreedomofmovementandsecuritylibertyper- ly fromtheneedtoevaluatelegalityofsecuritymeasures,includingthosein- means ofanAWSfollowsnotonlyfromthedutytosafeguardlife,butmorebroad- The requirementtoplacestrictspatio-temporallimitationsontheuseofforceby autonomous targetingisextremelylimitedunderIHRL. of forceapplication.Duetotheneedindividuateuseforce,scopefor harm, humanagentsmustbecontinuouslyandactivelyengagedineveryinstance used only as a lastresort whilst minimizing therisk of deprivation of life or bodily fied asabsolutelynecessary.Tocomplywiththerequirementthatlethalforcebe the locationwhereforceisadministered,recoursetolethalcannotbejusti- to lifeorriskofseriousinjury,includingduetheremovalstateagentsfrom strictly proportionateinaconcretesituation.Whenthereisnoimminentthreat force caninabstracttermsbejustified,itmustalsoabsolutelynecessaryand from enteringthesystem’ssensorandweaponrange.Evenwhenrecoursetolethal falling withinthesystem’stargetparameters,butwhomaynotbelegallykilled, Heyns, personally incontroloftheactualdeliveryorreleaseforce,amanner ‘Human 365 Rights bothduringandoutsideofarmedconflict. and the Use of Autonomous Weapons Systems’, 364 supra fn 22, 362–366; HRW

DEFENDING THE BOUNDARY 69 DEFENDING THE BOUNDARY 70 dignity, that proactive to (‘If effective compliance this man lifeanddignity,cannotbeacceptable. mote astate’scapacitytofulfillitshumanrightsobligations and tosafeguardhu- technologies (ofviolence)thatinhibitordiminish,ratherthanfacilitatepro- 368 Kerr andSzilagyi,‘Evitable Conflicts, Inevitable Technologies?’, supra fn40, 31. 367 Chamayou, Drone Theory , supra fn1, p211. 366 of irresponsibility, and technologiestotheexceptionalextraordinary.Itcountersfabrication assume vis-à-vistheirownpopulations,andhelpsconfinemilitarizedrationalities outcomes alone.Itremindsusoftheresponsibilitieswe‘normally’expectstatesto cesses involvedinautonomoustargeting,ratherthanbeingconcernedwithits attention totheobjectifying,dehumanizingandpotentiallydiscriminatorypro- ly contestedandhavebeenseverelycriticizedbyhumanrightsbodies.Itdraws to besituatedagainstthebackdropofpracticesautomatedkillingthataredeep- ing AWS.AnIHRL-orientedapproachallowsautonomousweapontechnologies study challengestheappropriatenessofexistingIHLassolemeansregulat- By focusingonIHRLrequirementsandconstraintstheuseofanAWS,this to theruleoflaw,intimespeaceaswellwar. ity fortheuseofforceandmaintainingpublicconfidenceinstates’adherence es. This is essentialto ensuring the availability of an effective remedy, accountabil- explain thereasoningunderlyingalgorithmicdecisionsinconcretecircumstanc- main involvedinalgorithmictargetingprocessesamannerthatenablesthemto manizing. Tosafeguardhumandignityandrights,agentsmustre- by meansofanAWS,andtheabsencedeliberativehumaninterventionisdehu- in itseffects.Theautomaticityandobjectificationinherenttargetconstruction uals undercategoricalsuspicion,isblindtocriticaldifferencesandstigmatizing remedy. Targetingbasedon‘patternsoflife’analyses,forinstance,placesindivid- subjected tocruel,inhumanordegradingtreatmentandtherightaneffective dignity, therighttoprivacy,notbediscriminatedagainstand of AWSislikelytosustainandevenpromotesuchpractices,threateninghuman large-scale interceptionofpersonaldataandalgorithm-basedprofiling.Theuse problematic fromahumanrightsperspective,includingsecretmasssurveillance, mic constructionoftargetsdrawsonpracticesthatarealreadyconsidereddeeply value-neutral newtechnologies, design ofIHLisconsistentwithpromoting,ratherthanrestricting’purportedly use envisaged ifthey strengthen theprotection of therightto life of those affected’). whether a is during it state In Asaro, that is such this justified steps uses we hostilities humans vein, use ‘Determinism, end weapons to of under Heyns, ensure up force retain with with of 366 human systems ‘Human should new that IHL organizations andfavoursaprecautionaryorientation:whereas‘thecore Machine sufficient and new weapons rights not Rights that IHRL); technologies be Agency, levels prima law permissible.’); and technologies AfCommHPR, and 367 is of facie fromanIHRLperspective,theintroductionof the thus and systems control are Use Responsibility’, limit clearly optimized of UN that such rights over 368 General Comment no 3, Autonomous on doc are as the the such A/65/321, increasingly remote in state release terms supra as those Weapons … controlled of fn If supra of their there 82, designed listed force fn 292 Systems’, supra capacity is above, 12, not aircraft doubt, (‘The §48 for to fn the 84, irresponsibility.’); implicate to supra (recommending consequence for should promote more onus s example, F, fn §35 to only 22, human show (‘The 374 be as of 674; 370 2011, p4. itability ispurposefullyconstructedbyhumanagents. Increasing autonomyinweaponsystemsisneitherautomaticnorinevitable.Inev- of forcecanhelpsafeguardhumandignityandrights. formal, legalrequirementfortheexerciseofmeaningfulhumancontrolinuse system ratherthantopeople. and a political act when human agents attribute agency to a technological device or 369 managed’. The lawisa‘crucialmeansbywhichtheeconomyofviolencecalculatedand ‘means that international law itself is undergoing a transformation’. with aradicallydifferentmeaning’,for,accommodatingpracticeinlegalterms stablished normsandrulesofinternationallawarepreservedformally,butfilled ishing existinglaw’scapacitytoserveasaguidepost.Thereisalsotheriskthat‘[e] uncertainties abouttheapplicabilityandmeaningofexistingnorms,thusdimin- which thelegalregulationofforceisarticulated.Thisgeneratescontroversiesand challenge thecategoriesanddisrupthuman–machineconfigurationsaround ing andautomatedkillingbutnewtechnologiesevolvingsecuritypractices the equipmentor,perhapsmostsignificantly,whoaresubjectedtoits operation. of institutionsthatdeploythetechnology, who areinvolvedinitsdesign,use juridical Autonomies’, formed 371 that “forces” new practices’). 372 power to generate inourmindswhatmay later beperceived of asnecessary’). ‘Evitable to technique Stoddart, ‘A Surveillance of Care’, supra fn323,p375. 373 technique –and anything can’). resolve Kerr S. Chamayou, Koskenniemi, E. by Krasmann, discourse Weizman, Conflicts, and allows a any collective 369 supra Szilagyi, political Legalnormsalreadyregulateandlimitalgorithmicdecisionmak- Drone Theory, the ‘perpetuates Inevitable ‘Targeted The Least of AllPossible Evils: Humanitarian Violence from Arendt to Gaza, fn ‘Faith, 73, abdication omission ‘Evitable conflict p Identity, 91 Killing Technologies?’, (on or the Conflicts, without of supra the new and illusion and personal 372 conflation practice its Thisreturnsresponsibilitytousasrepresentatives fn the becoming Inevitable Law: of 1, Killing supra responsibility p. the … On of 211. achieved existence the a fn of political Technologies?’, Mutually See 40, descriptive the 25 also through Innocent’, for of itself. (‘what Constitutive a anything Suchman privileged and Its the supra technology 371 supra the bureaucratic introduction Itisanethicalquestion that and promissory); Relationship’, fn (legal) fn 40, can Weber, 330, makes 28 rationality be attachment of 160 (‘law 370 supported a Kerr ‘Human-Machine possible 25 new An explicit, (the can LJIL and that technology dominant be 3 Szilagyi, to has by (2012) is Verso, trans- legal able 373 this the

Concluding Remarks 71 The Geneva Academy provides post-graduate education, conducts academic legal research and policy studies, and organizes training courses and expert meetings. We concentrate on branches of international law that relate to situations of armed conflict, protracted violence, and protection of human rights.

Established in 2007 by the Faculty of Law of the University of Geneva and the Graduate Institute of International and Development Studies, the Geneva Academy has acquired a global reputation for excellent teaching and research, and it attracts students of high quality to its master’s and training programmes. Our more than 660 graduates are em- ployed around the world, promoting and protecting International humanitarian law (IHL) and human rights (HR) in governments, NGOs, international organizations and academic institutions. The Geneva Academy thus contributes to the dissemination of legal knowledge in these crucial sectors.

Our scientific research focuses on clarifying IHL, strengthening human rights protection, and developing the areas of complementarity between IHL and international human rights law. In these areas, the Geneva Academy makes a specific contribution to policy development and debate, in government and among scholars and practitioners.

The Geneva Academy is a cosmopolitan community located in the heart of Geneva, an international city and humanitarian hub. Through close interaction with international organizations, NGOs, experts, and governments, we actively participate in global discussions of IHL, HR, international criminal law, and transitional justice.

978-29701003-3-1

Villa Moynier This work is licensed for use under a Creative Rue de Lausanne 120B Commons Attribution-Non-Commercial-Share CP 1063 - 1211 Geneva 1 - Switzerland Alike 4.0 International License Phone: +41 (22) 908 44 83 (CC BY-NC-ND 4.0). Email: [email protected] www.geneva-academy.ch