Cite as Miller, 11 J. NAT’L SECURITY L. & POL’Y __ (forthcoming 2021) Command Responsibility: A Model for Defining Meaningful Human Control By Matthew T. Miller* In the relatively near future, the United States and other countries are likely to develop varying levels of artificial intelligence (AI) and integrate it into autonomous weapons.1 There are significant voices, spearheaded by The Campaign to Ban Killer Robots, advocating for a preemptive ban on these weapons.2 The opponents of lethal autonomous weapon systems (LAWS) argue that it is unethical to allow a machine to decide when to kill and that AI will never be able to adhere to International Humanitarian Law (IHL) obligations.3 Although this opposition campaign has not yet achieved its goal of a ban, it has prompted considerable debate over the legality of developing and using LAWS. One of the concepts that has arisen in this debate is a legal requirement for meaningful human control (MHC) over LAWS.4 The idea of MHC has gained traction within discussions at the United Nations Convention on Certain Conventional Weapons (CCW), but the concept has its detractors.5 One of those detractors is the United States, whose delegation to the CCW Group of Governmental Experts continues to warn that MHC is an ambiguous term that “obscures rather than clarifies the genuine challenges” related to LAWS.6 Instead of human control, the U.S. argues that the key issue is ensuring “machines help effectuate the intention of commanders and the operators of weapon systems.”7 The U.S. Department of Defense showed its focus on intent, rather than control, by adopted the policy that “autonomous and semi-autonomous weapon systems shall * Major Matthew Miller is a Judge Advocate in the U.S. Army and currently serves as the Chief of the Operational Law Branch in the National Security Law Division of the Army’s Office of The Judge Advocate General. Major Miller holds a Master of Laws (LL.M) in National Security Law from the Georgetown Law Center and an LL.M. in Military Law from The Judge Advocate General’s Legal Center and School. The views expressed in the paper are the author’s alone and do not necessarily reflect those of the author’s employer. © 2021, Matthew T. Miller. 1 Melissa K. Chan, China and the U.S. are Fighting a Major Battle Over Killer Robots and the Future of AI, TIME, Sep. 13, 2019, https://perma.cc/62ZU-4FUZ. 2 See CAMPAIGN TO STOP KILLER ROBOTS, https://perma.cc/9RGG-A6ZU (providing an overview of the campaign and its goals). 3 HUMAN RIGHTS WATCH, HEED THE CALL: A MORAL AND LEGAL IMPERATIVE TO BAN KILLER ROBOTS 21 (2018), https://perma.cc/9WDZ-X655. 4 See Hayley Evans, Lethal Autonomous Weapons Systems at the First and Second U.N. GGE Meetings, LAWFARE (Apr. 9, 2018, 9:00 AM), https://perma.cc/9ARQ-3EHA (discussing numerous states’ references to meaningful human control). 5 See Karl Chang, U.S. Mission to Int’l Orgs. in Geneva, Consideration of the Human Element in the Use of Lethal Force, Address Before the Convention on Certain Conventional Weapons Group of Governmental Experts on Emerging Technologies in the Area of LAWS (Mar. 26, 2019) (discussing his skepticism over the ability to determine the level of human control that is necessary to comply with International Humanitarian Law). 6 Id. 7 Id. Cite as Miller, 11 J. NAT’L SECURITY L. & POL’Y __ (forthcoming 2021) be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.”8 The difference between meaningful human control and appropriate levels of human judgment may seem trivial to some, but it demonstrates the ambiguity of MHC. Using an ambiguous term can be useful to gain political and diplomatic consensus,9 but it has little value when attempting to apply the term as a legal obligation.10 The United States and others may interpret MHC to require measures that effectuate command intent and maintain human judgement over the use of force, while States that are more hesitant about LAWS may interpret MHC to require direct human control of every possible action by the weapon.11 The purpose of this paper is to provide a solution to this ambiguity and offer a workable definition of MHC. The overall thesis is that MHC should be defined as the control necessary to facilitate responsible command. Commanders do not have direct control over each engagement. Rather, command responsibility is based upon a leader’s broader control of military operations and responsibility for her forces’ adherence to IHL.12 Therefore, MHC should require that a LAWS be designed to ensure commanders can: 1) understand the capabilities and limitations of the LAWS and convey this information to their forces; 2) limit, at a minimum, the time and space in which the LAWS will operate; and 3) effectively investigate the causes of a LAWS taking unexpected action.13 Defining MHC through this lens of command responsibility will provide states with a clearer standard that is grounded in a well-developed IHL concept. To explain how the command responsibility model can be applied to MHC, the paper will begin by defining LAWS and providing an overview of the ways in which humans can interact with autonomous systems. This first section will also describe how a common method for understanding human-machine interaction is to look at where humans are located in the system’s decision loop: providing direct input “in the loop”; providing supervision “on the loop”; or being “out of the loop” and unable to provide input.14 Section II will outline the fundamental IHL 8 U.S. DEP’T OF DEF., DIR. 3000.09, AUTONOMY IN WEAPON SYSTEMS 2 (Nov. 21, 2012) [hereinafter DIR. 3000.09]. 9 See, e.g., Rebecca Crootof, A Meaningful Floor for Meaningful Human Control, 30 TEMPLE INT’L & COMP. L.J. 53, 54 (2015), available at https://sites.temple.edu/ticlj/files/2017/02/30.1.Crootof-TICLJ.pdf. 10 See Merel Ekelhof, Autonomous Weapons: Operationalizing Meaningful Human Control, INT’L COMM. OF THE RED CROSS (Aug. 21, 2018), https://perma.cc/2G2F-PA2P (explaining that abstract concepts about human supervision provide little value if they do not address the reality of military application). 11 Crootof, supra note 9, at 54. 12 See U.S. DEP’T OF DEF., DOD LAW OF WAR MANUAL § 18.4 (Dec. 2016) [hereinafter LAW OF WAR MANUAL]. 13 Id.; see Protocol Additional to the Geneva Conventions of 12 August 1949, and Relating to the Protection of Victims of International Armed Conflicts, art. 87, June 8, 1977, 1125 U.N.T.S. 3 [hereinafter Additional Protocol I] (discussing commanders’ responsibility to ensure their subordinates are aware of their legal obligations and take necessary steps to prevent violations); see generally LAW OF WAR MANUAL, supra note 12, § 19.20.1 (discussing how the United States has not ratified Additional Protocol I, but supports many of its provisions because they comply with longstanding U.S. practice or are based upon customary law principles). 14 PAUL SCHARRE, ARMY OF NONE: AUTONOMOUS WEAPONS AND THE FUTURE OF WAR 28-30 (2019); PAUL SCHARRE & MICHAEL C. HOROWITZ, CENTER FOR A NEW AM. SEC., WORKING PAPER: AN INTRODUCTION TO AUTONOMY IN WEAPON SYSTEMS 6 (2015), https://perma.cc/T4GP-PEBS. 2 Cite as Miller, 11 J. NAT’L SECURITY L. & POL’Y __ (forthcoming 2021) principles that are most relevant to LAWS: military necessity; distinction; proportionality; precautions in the attack; and command responsibility. After the explanation of the key concepts in autonomy and IHL, section III will merge these concepts to demonstrate how MHC can be applied to the design and use of LAWS through the lens of command responsibility. This section will use vignettes to analyze how the level of human control necessary to facilitate responsible command will vary, depending on the capabilities of the LAWS and the circumstances in which it will be used. Section IV will conclude with a discussion on how the command responsibility framework can address concerns that the use of LAWS will prevent accountability for IHL violations. Specifically, this section will argue that a commander’s obligations to train her forces and investigate and remediate potential IHL violations will allow for accountability even if a LAWS performs an unforeseen action. I. AUTONOMOUS SYSTEMS AND HUMAN INTERACTION IN THE DECISION LOOP The first step in discussing MHC is to provide a working definition of a LAWS. There remains some debate over this topic and, even after five years of work, the CCW Group of Governmental Experts has yet to agree on a definition.15 Opponents of LAWS define autonomy as a machine that acts on its “own deliberations, beyond the instructions and parameters its producers, programmers, and users provided to the machine.”16 This definition implies that it is impossible to apply human control to LAWS, because its actions cannot be contained by its programmers or operators. The U.S. Department of Defense defines autonomous weapons as those that, “once activated, can select and engage targets without further intervention by a human operator.”17 The International Committee of the Red Cross (ICRC) similarly defines fully autonomous weapons as those that “can select (search for, detect, identify, track or select) and attack (use force against, neutralize, damage or destroy) targets without human intervention.”18 Unlike the definition offered by LAWS opponents, the U.S. and ICRC definitions do not remove the possibility that humans may retain some ability to control a LAWS’ actions. Therefore, these two later definitions provide a more effective starting point for the analysis of MHC.