40 The Security Times February 2018 February 2018 The Security Times 41

SECURITY BRIEFS SECURITY BRIEFS The new weapons of mass ? Building a lethal autonomous weapon is easier than building a self-driving car. A new treaty is necessary

BY RONALD ARKIN pathos and hype while focusing on Smart autonomous weapons sys- BY STUART RUSSELL All of the component technologies – flight Compliance with international humanitar- the real technical, legal, ethical and tems may enhance the survival of control, swarming, navigation, indoor and ian law, even if achievable, is not sufficient to moral implications. noncombatants. Human Rights outdoor exploration and mapping, obstacle justify proceeding with an arms race involving et me unequivocally state: Numerous factors point to auton- Watch considers the use of pre- eginning in 2014, the High Contract- avoidance, detecting and tracking humans, tac- lethal autonomous weapons. President Obama: The status quo with respect omous robots soon being able to cision-guided munitions in urban ing Parties of the Convention on Cer- tical planning, coordinated attack – have been “I recognize that the potential development Lto innocent civilian casual- outperform humans on the battle- settings to be a moral imperative. Btain Conventional Weapons (CCW) demonstrated. Building a lethal autonomous of lethal autonomous weapons raises questions ties is utterly and wholly unaccept- field from an ethical perspective: In effect, there may be mobile have held meetings at the United Nations weapon, perhaps in the form of a multi-rotor that compliance with existing legal norms – if able. I am not in favor of Lethal • They are able to act conser- precision-guided munitions that in Geneva to discuss possible limitations on micro-unmanned aerial vehicle, is easier than that can be achieved – may not by itself resolve, Autonomous Weapon Systems vatively, as they do not need to result in a similar moral impera- the development and deployment of lethal building a self-driving car, since the latter is and that we will need to grapple with more (LAWS) nor of lethal weapons protect themselves in cases of low tive for their use. Such weapons autonomous weapons systems (AWS). In held to a far higher performance standard and fundamental moral questions about whether of any sort. I would hope that certainty of target identification. have the possibility of deciding November 2017, the CCW convened a must operate without error in a vast range and to what extent computer algorithms should LAWS would never when to fire and – more formal Group of Governmental Experts of complex situations. This is not “science be able to take a human life.” need to be used, as I importantly – when not (GGE), chaired by India’s Ambassador to fiction.” Autonomous weapons do not have One of the “fundamental moral questions” am against killing in to fire. They should be the UN Amandeep Singh Gill, with a man- to be humanoid, conscious and . And the is the effect of autonomous weapons systems all its manifold forms. Robot designed with overrides date to “assess questions related to emerging capabilities are not “decades away” as claimed on the security of member states and their But if humanity per- to ensure meaningful technologies in the area of lethal autono- by some countries. peoples. On this matter, the message of the sists in entering into human control. More- mous weapons systems.” This article reflects UN Special Rapporteur Christof Heyns, AI community, as expressed in the letters warfare, which is an imperative over, they can employ views shared by a great many in the artificial Human Rights Watch, the International Com- mentioned above, has been clear: Because unfortunate underlying fundamentally different intelligence community. These views were mittee of the Red Cross and other experts have they do not require individual human super- assumption, we must tactics while assum- expressed in an open letter on July 28, 2015, expressed concerns about the ability of autono- vision, autonomous weapons are potentially protect the innocent The moral obligation of using ing far more risk than signed by over 3,700 AI researchers, and in mous weapons to comply with provisions scalable weapons of mass destruction; an noncombatants in the human warfighters in a letter to the Obama administration written of international humanitarian law regarding essentially unlimited number of such weapons battlespace far better AI to reduce atrocities terms of protecting non- on April 4, 2016, by 41 leading American AI military necessity, proportionality and dis- can be launched by a small number of people. than we currently do. combatants and assess- researchers, including almost all of the living crimination between combatants and civilians. This is an inescapable logical consequence Technology can and ing hostility and hostile presidents of AAAI, the main professional Discrimination is probably feasible in most situ- of autonomy. As a result, we expect that should be used toward that end. • The eventual development and intent. In essence, these systems society for artificial intelligence. The British ations, even if not perfectly accurate. However, autonomous weapons will reduce human Is it not our responsibility as sci- use of a broad range of sensors can more effectively operate on a AI community sent a similar letter to then determining proportionality and necessity is security at the individual, local, national and entists to look for effective ways will render robots better equipped philosophy of “First do no harm” Prime Minister David Cameron. most likely not feasible for current AI systems international levels. to reduce man’s inhumanity to than humans for battlefield obser- rather than “Shoot first and ask The UN defines autonomous weapons as and would have to be established in advance It is estimated, for example, that roughly one its fellow man through technol- vations. questions later.” having the capacity to “locate, select and elimi- with reasonable certainty by a human opera- million lethal weapons can be carried in a single ogy? Research in ethical military • They can be designed without Building such systems is not nate human targets without human interven- tor for all attacks the weapons may undertake container truck or cargo aircraft, perhaps with robotics can and should be applied emotions that would otherwise a short-term goal, but rather tion.” Some have proposed alternative defini- during a mission. This requirement would only 2 or 3 human operators rather than 2 or 3 toward achieving this goal. cloud their judgment or result in part of a medium- to long-term tions – for example, the UK Ministry of Defence therefore limit the scope of missions that could million. Such weapons would be able to hunt I have studied ethology – animal anger and frustration with ongoing agenda addressing many challeng- says that autonomous weapons systems must legally be initiated. for and eliminate humans in towns and cities, The considerations of the preceding para- BY KIM MIN-SEOK operations and have experience to replace the Songgolmae. On behavior in their natural environ- battlefield events. ing research questions. However, “understand higher-level intent and direction” Another important component of interna- even inside buildings. They would be cheap, graph apply principally to weapons designed in developing norms and rules of the division level, an indigenous ment – as a basis for robotics for my • They avoid the human psycho- exploiting bounded morality within and “are not yet in existence and are not likely tional humanitarian law is the Martens Clause, for ground warfare and anti-personnel opera- engagement for their use. The US version of the existing KUS-9 entire career, ranging from frogs, logical problem of “scenario fulfill- a narrow mission context helps to to be for many years, if at all.” according to which “the human person remains STUART RUSSELL tions, and are less relevant for naval and aerial onventional warfare tactics, military has actively used drones in drone must be developed and the insects, dogs, birds, wolves and ment,” which contributed to the achieve better performance with Much of the discussion at the UN has been under the protection of the principles of human- is a computer science professor combat. It is still the case, however, that “to traditionally maintained Afghanistan and Iraq in efforts to RemoEyes replaced. The Agency human companions. Nowhere has downing of an Iranian airliner by respect to preserving noncomba- stymied by claims that autonomy is a mysteri- ity and the dictates of public conscience.” In this and the Smith-Zadeh Professor in entrust a significant portion of a nation’s through rifles and tanks, eradicate terrorist forces including for Defense Development (ADD) Engineering an the University of C it been more depressing than to the USS Vincennes in 1988. tant life, and thus warrants robust ous, indefinable property. In the view of the regard, Germany has stated that it “will not California, Berkeley. defense capability in any sphere to autonomous artillery, and fighter jets, is now al-Qaeda and the Taliban. Israel will soon complete development study human behavior in the battle- • They can integrate more infor- research on humanitarian grounds. AI community, the notion of autonomy is accept that the decision over life and death is systems is to court instability and risk strategic centered on the rapid innovation was the first country to use a drone of a MALE (medium-altitude field. The commonplace occurrence mation from more sources far faster Other researchers have begun essentially unproblematic in the context of taken solely by an autonomous system” while surprise.” Autonomous weapons in conflict and advancement of IT, artificial militarily and is still plays a leading long-endurance unmanned aerial of slaughtering civilians in conflict than a human possibly could in related work on at least four conti- lethal weapons, which is quite distinct from Japan “has no plan to develop robots with effective, unattributable and easily proliferated with other autonomous weapons must adapt intelligence (AI), avionics and cyber role in the development of weapon vehicles), which is similar to the over millennia gives rise to my pes- real-time before responding with nents. Nonetheless, many daunting the philosophical context of human autonomy. humans out of the loop, which may be capable once the major powers initiate mass produc- their behavior quickly, or their predictability technology in developed countries. zed drones. During a prolonged US MQ-9 Reaper and, aside from simism in reforming human behav- lethal force. questions regarding lethality and The autonomy of lethal weapons is no more of committing murder.” BAE Systems, the tion and the weapons become available on the will lead to defeat. This adaptability is neces- The battlefield is expanding to period of joint military exercises basic reconnaissance capabilities, ior yet provides optimism for the • When working in a team of autonomy remain unresolved. Dis- mysterious than the autonomy of a chess world’s second-largest defense contractor, has international arms market. For the victor they sary but makes autonomous weapons intrinsi- five dimensions – ground, sea, air, with the US, the South Korean could carry air-to-ground mis- prospects of robots being able to combined human soldiers and cussions regarding regulation must program that decides where to move its pieces asserted that it has no intention of developing would have advantages over nuclear weapons cally unpredictable and thus difficult to control. space and cyber. How- siles like the AGM-114 exceed human moral performance autonomous systems, they have be based on reason, not on fear. and which enemy pieces to eliminate. The key autonomous weapons, stating that the removal or carpet bombing: they leave property intact Moreover, the strategic balance between robot- ever, the South Korean Hellfire. in similar circumstances. the potential to independently and Until these questions are resolved, is that the specific targets are not identified and of the human from the loop is “fundamentally and can be applied selectively to eliminate only armed countries can change overnight due to military faces a triple Miniature drones I have the utmost respect for our objectively monitor ethical behav- a moratorium is more appropriate approved – either in advance or at the time of wrong.” those who might threaten an occupying force. software updates or cybersecurity penetration. handicap. Thus, with- on par with America’s young men and women in the bat- ior in the battlefield by all parties than a ban. Only then can a careful, detection – according to human judgment, but At present, the broader public has little Finally, whereas the use of nuclear weapons Indeed, a nation’s autonomous weapons might out technical innova- Droning on WASP UAV are in the tlespace, but they are placed into and to report any infractions that graded introduction of the technol- are instead selected by an algorithm based on awareness of the state of technology and the represents a cataclysmic threshold we have – be turned against its own civilian population. tion, it will be difficult works. They will be situations where no human has may be observed. ogy into the battlespace be ensured. sensory input the algorithm receives after the near-term possibilities, but this will presumably often by sheer luck – avoided crossing since With no possibility of attribution to an external to maintain its combat deployed to the new ever been designed to function. LAWS should not be considered The status quo is unacceptable mission is initiated by a human. change if the killing of humans by autonomous 1945, there is no such threshold with scalable adversary or individual, one can imagine that power. The South Korean military “decapitation” special This is exacerbated by the tempo an end-all military solution. To with respect to noncombatant The feasibility of autonomous weapons is robots becomes commonplace. At that point, autonomous weapons. Attacks could escalate the nation’s government would be less popular The first of Korea’s is catching up forces unit established at which modern warfare is con- the contrary, their use should be deaths. It may be possible to save also not in question, at least for a broad class of the dictates of public conscience will be very smoothly from 100 casualties to 1,000 to after such an event. Finally, the possibility of handicaps is its sur- last year by the South ducted. Given this pace and resul- limited to specific circumstances. noncombatant lives through the use missions that might currently be contemplated. clear, but it may be too late to follow them. 10,000 to 100,000. an accidental war – a military “flash crash” rounding conditions. with a new drone army Korean Army. These tant stress, expecting widespread Current thinking recommends: of this technology, and these efforts involving spiraling and unpredictable high- The Korean Peninsula micro drones provide compliance with international • Specialized missions where should not be prematurely termi- speed interactions among competing algorithms has achieved stabili- intelligence on targets humanitarian law seems unreason- bounded morality applies, e.g. nated by a preemptive ban. AI can – cannot be discounted. zation following the Cold War military has observed the US army’s to the air force and ground troops. able and perhaps unattainable by room clearing, counter-sniper be used to save innocent lives where It seems likely that pursuing an arms race in era, but there are indications of deployment of drones and learned Moreover, the Northrop Grum- flesh-and-blood warfighters. operations or perimeter protection humans may and do fail. Nowhere lethal autonomous weapons would result in a another potential clash between how to use them in combat. It man RQ-4B Global Hawk will be I believe judicious design and in the DMZ. is this more evident than drastic and probably irreversible reduction in land powers such as China and also received technological support introduced in 2018 to conduct sur- the use of LAWS can lead to the • High-intensity inter-state war- on the battlefield. international, national, communal and personal Russia and sea powers like the from Israel when it first developed veillance over the vast territory of potential saving of noncombatant fare, not counter-insurgencies, to security. The only viable alternative is a treaty United States and Japan. its own weaponized drones. Even North Korea. lives. If properly developed and minimize likelihood of civilian that limits the development, deployment Secondly, the threat posed by now, many South Korean compa- In addition to the UAVs, the deployed, it can and should be used casualties. and use of such weapons and prevents the North Korea has increased. The nies in the defense industry develop South Korean Army currently plans towards achieving that end, and • Deployment in concert with large-scale manufacturing that would North boasts an immense conven- military drones through technical to make use of a dog-horse robot not simply about winning wars. soldiers, not as their replace- result in wide dissemination of these tional force of 1.2 million soldiers cooperation with Israel. developed by the defense ministry We must position this humanitar- ment. Human presence scalable weapons. and is becoming an actual nuclear The South Korean Army currently with some $43.34 million invested ian technology at the point where in the battlefield This argument parallels that used state. operates the RQ-101, or Song- from 2006 to 2012. This robot is DARPA war crimes, carelessness and fatal should be main- by leading biologists to convince Thirdly, South Korea faces a steep golmae (wingspan: 6.4 meters), capable of autonomous navigation human error occur and lead to tained. US Presidents Lyndon Johnson “demographic cliff,” and a decrease developed through its homegrown along a set route, short-distance noncombatant deaths. Unmanned and Richard Nixon to renounce in its number of troops is inevitable. technology in 2000, along with surveillance and reconnaissance, as systems will never be able to be America’s biological weapons Thus, the South Korean military is Israeli-made Searcher IIs, while well as mine detection. The army perfectly ethical in the battlefield, program. This in turn led to expected to shrink from its current some Army corps have deployed established a dronebot military but I am convinced that they can the drafting by the United level of 620,000 troops to 500,000 Israel’s Heron drones. While the research center in January 2018. ultimately perform more ethically Kingdom of the Biological by 2022. While the decrease in division level commands the KUS-9 A pilot-scale drone combat unit than human soldiers. Weapons Convention and its manpower has little influence on drone (2014), the regiment level will enter service this year. Drones I am not averse to a ban should subsequent adoption. I think the navy and air force, the biggest uses the RemoEye-15 (2004) and can conduct reconnaissance and we be unable to reach the goal of we can all be glad that those burden will fall upon the 483,000- the RemoEye-006 UAV (2006) execute strikes, while in the long reducing noncombatant casualties; steps were taken. strong army, which will soon be was developed and deployed at term robots can be tasked with but for now we are better served reduced to 365,000 troops. The the battalion level. Defense reforms search missions. by a moratorium, at least until we South Korean military, especially emphasizing drone warfare could The Army also installed a sci- can agree upon definitions regard- the Army, will not be able to cope expand the operational area of entific boundary system south of ing what we are regulating and with the changing military structure corps and divisions by a factor of the demilitarized zone (DMZ). it is determined whether we can and potential future threats unless three to four in the future. This Its high-performance surveillance indeed achieve humanitarian ben- it seeks innovation through drones would also enable network-centric cameras and optical fiber network efits through the use of this technol- and automation. warfare based on a drone system. automatically alert the situation ogy. A preemptive ban ignores the Although the South Korean Furthermore, carrying out the room when infringed upon, allow- moral imperative to use technology military has pushed for defense Kill Chain pre-emptive system ing immediate military reaction. to reduce the persistent atrocities reform since the late 1990s, it has operations to strike down North Once this new boundary system and mistakes that human war- repeatedly failed due to changing Korea’s ballistic missiles in near is fully operational, the number of fighters make. At the very least it administrations, the 1997 Asian real-time now requires raising soldiers stationed in the area can is premature. financial crisis and instability the reconnaissance capabilities of be decreased drastically. Even blind Alternative considerations include resulting from North Korea’s fre- forward units in a groundbreak- spots along the inter-Korean border the following: Regulate autono- quent military provocations. But ing manner. The Army corps also will then be effectively monitored. mous weapons usage instead of a current sense of crisis is pushing needs a next-generation drone When taking into consideration prohibiting them entirely; consider the South Korean government and the demographic cliff, threats restrictions in well-defined circum- the military toward speedy reform. from North Korea and China, and stances rather than an outright ban The development of drones and IT KIM MIN-SEOK the harsh reality of the need for and stigmatization of the weapons technology has caused the South was a researcher at defense reform, the build-up of a systems; do not make decisions Korean army to establish a drone the Korea Institute of drone and robot system in South based on unfounded fears – remove combat unit. General Kim Yong- Defense Analyses from Korea becomes a realistic prospect. 1982 to 1994 before woo, the South Korean Army chief becoming a journalist. Korea’s drone and robot technol- of staff who assumed the post last He was spokesman for ogy is nearing 80 to 85 percent RONALD ARKIN year, intends to change the struc- the South Korean Ministry of the level reached by the highly is Regents' Professor and ture of the Army and shift to new of National Defense advanced countries like the United director of the Mobile warfare tactics based on technol- from 2010 to 2016. He States. Setting up a drone army Robot Laboratory at ogy. The Moon Jae-in government is now a military and will be facilitated by South Korea’s the Georgia Institute of security expert as well as is backing this concept. industrial prowess in such fields as Technology’s School of opinion columnist for the Interactive Computing in The US and Israeli armies were Korean daily newspaper materials and battery technology as Atlanta. the first to deploy unmanned JoongAng Ilbo. well as electronics and communica- aerial vehicles (UAVs) for military tions technologies.