Focused Question

Artificial Intelligence Opportunities and Conditions for European Land Forces

AN EXPERTISE FORUM CONTRIBUTING TO EUROPEAN CONTRIBUTING TO FORUM AN EXPERTISE SINCE 1953 ARMIES INTEROPERABILITY European Army Interoperability Center This text was drawn up with the help of Mrs. Laurane Chédaleux, member of the Finabel research team, under the supervision and guidance of the Head of the Permanent Secretariat. This focused question is a document that gives an initial reflection on the theme. The content is not reflecting the positions of the member states, but consists of elements that can initiate and feed the discussions and analyses in the domain of the theme. It was drafted by the Permanent Secretariat and is supported by the organisation.

LIST OF ACRONYMS AND TABLE OF CONTENTS ABBREVIATIONS Introduction 3 AI Artificial Intelligence BATX Baidu, Alibaba, Tencent et Xiaomi A wide field of possible CCP Chinese Communist Party applications for AI DTIB Defence Technological and within land forces 4 Industrial Base EU European Union - AI implementation out of GAFAM Google, Amazon, Facebook, Apple, the battlefield 5 Microsoft - AI implementation on ICT Information and Communication the battlefield 7 Technology IO International Organisation A large set of conditions requested IOT Internet of Things for AI implementation 7 LAWS Lethal Autonomous Weapons Systems - Technical issues 8 NATO North Atlantic Treaty Organisation - Human resources issues 9 NBIC Nanotechnology, Biotechnology, - Ethical issues 11 Information technology and - A need for regulation 13 Cognitive sciences NGO Non-governmental organisation An undeniable need for cooperation NPT Non-Proliferation Treaty through Europe and the world 14 OODA Observe, Orient, Decide, Act PESCO Permanent Structured Cooperation - Cooperation between States 14 UK United Kingdom - Cooperation between private UN United Nations and public sector 18 US United States of America USSR Union of Soviet Socialist Republics Conclusion 19 WWII World War II References 19

Artificial Intelligence 2 On the first of September 2017, Vladimir software perform perception, reasoning, Putin – President of the Russian Federation – knowledge representation, planning, com- stated, “Artificial intelligence is the future, munication or autonomy in order to achieve not only for Russia, but for all humankind”. specific goals. If Machine Learning appears as “It comes with colossal opportunities, but also the most advanced and usable AI approach, threats that are difficult to predict. Whoever be- it is mostly given its high media coverage and comes the leader in this sphere will become the its functioning. To summarize, a program can ruler of the world”1. The Artificial intelligence become intelligent in two ways. is indeed no longer a science-fiction perspec- tive, but an on-going revolution for societies, On a formal way, the program is coded and industries and eventually armies. any computer scientist can check down the parameters and way of “reasoning” of the ma- From 1770 to 1850 were developed steam chine. AI systems are then based on automa- engine as well as railway lines and thus, from tion, to make a system process automatically. 1870 to 1910 were born aviation, automo- As an example, on machine vision, the pro- bile, electricity and telephony. The twentieth gram will make a system captures and analy- century has been the century of the develop- ses visual information using a camera, process ment and the individualisation of these new an analogy-to-digital conversion and give an technologies in the new mass consumption outcome. societies. The European armies has been deeply transformed by these technologies. A Another mechanism is to feed the system with third revolution is currently occurring, since data. As we “show” it multiple examples, it the 1990s with NBIC technologies and the then automatically identifies the features and new world most valuable resource: data. Just learns “by association”. Deep Learning shows as the previous ones, this technological and up as a technique of Machine Learning, using economic revolution is going to disrupt our a neural network directly inspired by the hu- world and the ways we live, work, and ulti- man brain functioning. mately make war. Given the growing amount of data worldwide As a result of exponential progress in Nano- companies like the American GAFAM or the technology, Biotechnology, Information tech- Chinese BATX are collecting everyday as well nology and Cognitive sciences, scientists have as the enhancement of the storage capabili- been able to move forward an old scientific ties3 and the improvement of computer pro- dream: AI. This must be considered as a “con- cessing, AI systems based on Machine Learn- cept for improving the performance of automat- ing capabilities can be considered as exponen- ed systems for complex tasks”2. As a result, the tial. Indeed, the more these systems are fed system is adapting itself to a situation and re- by data, the more efficient and precise they acts autonomously – independently from hu- become. Given the NBIC technology im- man operators –. From this concept, system provements these past decades, whether AI is designers have developed several approach- going to happen is not a question. It is only a es – that include Machine Learning – to make matter of time. Data are in this century what

1 The editorial board, Putin: Leader in artificial intelligence will rule world, CNBC news website, 4th September 2017, available on https://www.cnbc.com/2017/09/04/putin-leader-in-artificial-intelligence-will-rule-world.html 2 Stoney Trent and Scott Lathrop, A primer on Artificial Intelligence for Military Leaders… 3 Moors law – it is going to stop

Artificial Intelligence 3 oil was to the last one: a driver of growth and applications of AI within armies, the debates change. Then, the debates should focus on the remain focused on the LAWs question. These following questions: how do we want AI to so-called “autonomous weapons” or “kill- be implemented? To what end? Under whose er-robot”, which have been described as the responsibility and according which rules? If third revolution in warfare, after gunpowder Transhumanism perspectives appear exces- and nuclear arms, are the main catalyst of fear sive, technophobia is just its negative twin. from AI implementation into defence sector. However, if this issue legitimately rises up, it Everyone has in mind a Hollywood’s image of should not restrict the debate of AI imple- AI, in particular due to the success of movies mentation within the European armies. An like Her, A space Odyssey or Matrix. None- assessment of the opportunities of AI imple- theless, such representations of IA are mostly mentation within land forces must be drafted fantasies, reductive, and obviously not in line (I) and technical, ethical and legal issues (II) with the projects and current applications of have to be solved through cooperation (III) to this new technology in our societies. In 2017, allow European armies to safely but promptly AI technologies have become a major topic enter this new technological age. following the defeat of a Go world champi- on against the intelligent program Alpha Go. This was proving that AI was “more intelli- A WIDE FIELD OF POSSIBLE gent” than the human playing at that par- APPLICATIONS FOR AI WITHIN ticular game.4 If this is an impressive move, LAND FORCES talking about intelligence remain tricky as this program took part in a game with limited outcomes and is not able to do anything else. General Schwarzkopf, commander-in-chief General intelligence, which can be considered of the allied coalition during the Gulf War, as “possessing common sense and an effective was the first beneficiary of the technological ability to learn, reason and plan to meet com- growth at the end of the twentieth century. plex information or processing challenges across He explained that technology has allowed a wide range of natural and abstract domains”5, him to gain omniscience on the battlefield might be the future of AI. However, for the and to dissipate Clausewitz's "fog of war". years to come, AI applications will remain Since that time, the technological progress narrowed intelligence systems, solving precise has gone on. Today, multiple sensors on a sol- issues they are programmed for better that dier’s equipment inform the commandment the human brain, but excluding any broader in real time on its environment and drones understanding. The private sector – especially are allowing intelligence to get information business, communication or financial compa- without sending any soldier. But at the time nies – has already started to implement kinds of hybrid war and guerrilla, omniscience ap- of AI to gain efficiency and make economies. pears as a utopia. The development of AI, temporarily with robotisation, might enhance The European Land forces have no choices prospective abilities and allow Land forces to but to follow the progress or to face an un- better overcome their opponents out (A) and avoidable demotion. Among all the possible on the battlefield (B).

4 Website of Deepmind, “Discover more about AlphaGo”, https://deepmind.com/research/alphago/, accessed on the 23rd September 2018 5 Nick Bostrom, Superintelligence – Paths, Dangers, Strategies, 2014, Oxford, Oxford University Press, p. 3

Artificial Intelligence 4 AI implementation out of the become truly useful. As an example, the mul- battlefield tiple legal documents accompanying external operations are often complex and sometimes According to the French General Barrera, tricky. Legal advisors are always trying to find innovation makes sense only if it helps “to the breach in them. For instance, Westerners maintain the operational superiority of our forc- have more or less diverted the UN mandate es, to make our organizations more agile, sim- during the Libyan conflict. With intelligent plify our operating methods, especially in daily tools, legal officers might gain efficiency. work”6. Here is the good news: AI can truly be an asset for the Armies on a daily basis, Thus, in the prospect of the increasing level especially out of the battlefield. of cooperation to come between European countries, language barriers remain a difficult Like in the private sector, the Armies could challenge. Even if European armies were led implement AI within their medical care struc- by officers speaking C1-level English, there tures and processes. AI is currently tested to would still have specific issues, specific top- assist health professionals by analysing faster and more precise medial images, by deeply monitoring patients via smart devices or supporting clinical decisions with predictive analytics.

In the industrial sector, AI de- vices are just about to assist maintenance services. Predic- tive maintenance actions could then be better schedules and sensors will be able to warn technicians about dysfunc- tions the machine is facing. The European civil project SERENA and several private companies’ products are nearly or already available. Such systems would help military capabilities to remain roadworthy.

AI can also be applied to legal devices. Some intelligent systems have already been created – like Hyperlex for example – to analyse legal contract and give advices and feed- back about it. If this kind of software need to be deeper developed, they can

6 “En somme ça n’a de sens que si l’innovation concourt directement à conserver la supériorité opérationnelle de nos forces, à rendre plus agile nos organisations, plus simple notre fonctionnement, en particulier celui du quotidien”, Général de corps d’armée Bernard Barrera dans “Scorpion : nous préparons déjà le jour d’après”, Defense et Securité Internationale, Hors-série n°61, août-septembre 2018, p. 22

Artificial Intelligence 5 ic on which the stakeholders would not feel appears risky. The human kind should always comfortable speaking in another language be able to understand its actions. As a result, than their mother tong. A lot of documents such system would be an advisory tool for also need to be translated in all the 24 Eu- commandment to take a decision. ropean languages, which include important costs. However, AI is also improving on this To feed such system, Intelligence cells of issue. As an example, Google launched in Armies need to gather as much information 2016 its Neural Machine Translation system. as possible: defeating its opponents will in- But more should be achieved to help officers creasingly depend on tackling the best data. understanding each other, like audio auto- There, AI can make a huge difference and mated translator systems. might revolutionize warfare more than AI used in the application of lethal force. Several AI may also become a way to spare money projects are already in development or imple- and protect the environment. Google has an- mented. Facebook uses AI systems to find and nounced that it has entrusted its intelligent remove terrorist content from its platform, system Deepmind to manage and optimise which could inspire armies in their cyber the energy consumption of its huge data cen- fight against terrorism. The EU funded civil tres. Energy management, which is increas- project BRIDGET is already over, and allow ingly costly, is emerging as one of the new ca- visual search engines to find and recognise pabilities of AI and could be used in military monuments, locations, buildings, products, Headquarter as well. books, logos or objects. Such AI applications should be developed for the European Land On typical military related issues, some experts forces too. Indeed, in 2014 while soldiers of consider AI will be a game changer for Intel- unknown nationality occupied the Donbas ligence, Strategic Planning and Surveillance. in Ukraine, the military intelligence servic- es were not the first able to determine with In a faster and uncertain world, the State still certainty who they were. However, a Polish have to predict the coming issues in their envi- researcher using facial recognition software ronment. However, the decision making pro- found a proof they were Russian on the basis cess and the OODA loop need to be imple- of pictures taken in Chechnya of these same mented as fast as the situation evolves and the men wearing Russian uniforms. AI may then human brain is then really challenged. Some ensure intelligence service a better assessment strategic military planning tool have then been of collecting information. Such systematic designed to help commandments, but an in- analyses would allow the detection of some telligent system able to adapt itself constantly precursor signals that are yet impossible to to the environment evolutions would be even perceive and improve Armies’ abilities regard- more effective. Decision-making would end ing simulation and modelling. The Pentagon up more technical and rational as based on is already working on this opportunity as pro- predictive analysis and Armies would be able ject Maven monitored by Google is dealing to compress and accelerate the OODA loop. If with these technological possibilities. some analysts consider that soon enough tech- nicians and engineers would replace strategist, AI devices and systems implementation in Ar- it remains a whimsical idea: AI systems won’t mies daily services and functioning could defi- be mature enough to enable this before a long nitely reduce some costs and make processes time and a complete reliance on the machine faster in an uncertain and changing world.

Artificial Intelligence 6 AI implementation on the battlefield during a NATO-led mission. As a result, 10 soldiers died and 21 were wounded. To- Some innovations are already semi-auton- day, drones can achieve reconnaissance mis- omous by coupling some AI and robotic sions to avoid such trap. Armies will be able technologies. As an example, South Korean to anticipate the enemy positions and capa- troops are assisting in the Korean Demilita- bilities. Some robots are even able to check rized zone by SGR-A1 robots with surveil- down buildings and undergrounds. Drones lance, tracking, firing and voice recognition are already more and more implemented skills. According to some experts, Israelis within some Land forces around the world. In would also detain some really innovated ca- Europe, it should be implemented too. With pabilities implementing AI systems. LAWS – AI capabilities, the drones would follow their military robot, which autonomously search mission autonomously without any soldier and engage targets based on programmed guiding it. Soldiers’ skills could then be better requirements – rise as a very hot topic since a employed. few years. Indeed, there are many ethical and legal concerns about it. However, such inno- AI systems are still under development, but vation has undeniable assets. If robots replace expert already predicts they will make an im- humans on the battlefield, they would pro- portant disruption. As European countries tect both physical and mental health of sol- are trying to invest more in their Armies, such diers. They also might be more efficient as a investments might be a way to gain efficien- machine has no emotions, cannot feel stress, cy and to reduce some cost on the long term. hatred, be tired or hungry. Therefore, if sol- However, the implementation of AI the Ar- diers are well trained, they may have a differ- mies comes along with a long list of issues that ent behaviour on the battlefield. A machine need to be tackled. will always perform in the same way. These are the advantages of unmanned systems, whether they are vehicles, drones, LAWS, etc. A LARGE SET OF CONDITIONS The possibilities are huge and these abilities REQUESTED FOR AI can also be “mixed” to get, for example, un- IMPLEMENTATION manned vehicles equipped with autonomous offensive capabilities. AI implementation in land forces, including In the case of weapons with a non-auton- automation of weapons systems, is the ine- omous lethal system, some intelligent au- luctable future and European armies have to tomation can still be implemented to assist prepare this coming reality. Today, if AI tech- soldiers in their decision to fire and to en- nologies are still at their early stage, consider- sure more precision. This can still improve able financial resources are provided for their the soldier resilience during its mission and development. But implementing AI without efficiency. dealing with some crucial issues into the ar- mies – and even in our societies – would be Drones with intelligent devices are the solu- a mistake. Technical issues, on which securi- tion for dull or dangerous missions of recon- ty relies on (A), Human resources challenge naissance and surveillance. For instance, on (B), Ethical aspects (C) and Regulation needs the 18th of August 2008, French troops were (D) should rise as AI pre-implementation top ambushed in the Uzbin Valley, Afghanistan priorities.

Artificial Intelligence 7 Technical issues or pulse, that is disruptive or damaging to elec- tronic equipment, the armies should obviously One of the biggest fear when it comes to AI be trained to continue their mission in a de- is the loss of control. A lack of loyalty of the graded or contested environment. As it would software could come from the fact that the not harm civilians directly, such weapons could AI system would no longer act according to be quite easily used by the armies and the risk the way it was programmed or parameters to be drown into the fog of the war arise as were settled. To ensure the correct function- a major risk. Specific training toward this is- ing of systems, computer scientists have two sue already exists in some armies regarding the approaches. If the intelligent program is cod- damage of communication or localization sys- ed, experts can quite easily check the lines of tems, but it should definitely be extended. code. However, when AI is operating with Machine Learning, the only way to evaluate In the military sector, ensuring security also its behaviour is to execute as much as possible deals with ensuring confidentiality of classified the system. If we can reasonably trust the pro- data and AI system must be programmed to gram, it will nonetheless never be tested on respect and disclose information according mil- every possible situation. itary pre-existing classification. But AI technics may also produce high value outcomes from If a loss of control happens, it could also be as common data. From there, any machine should a result of a security breach, cause by human be able to estimate the level of confidentiality of mistake – i.e. Stuxnet virus in Iran – or a hard its own outcomes and to classify them. cyber-attack on the system. As a consequence, the implementation of AI implies ensuring cy- AI could also become an issue of safety, as ber security. Cyber security issues are already the RAND Corporation assessed in its arti- an important topic for the European armies, cle How Artificial Intelligence Could Increase especially toward soldier use of social media the Risk of Nuclear War7, by looking back to and IOT. AI of things will soon enough be a the story of the Russian Lt Col Stanislav Pet- reality in soldiers both professional and per- rov. This officer was working with a machine sonal lives. The damages are already annoying, meant to detect a missile attack on the 26th but with AI coming to such an ever more con- September of 1983 when its siren clanged nected while uncertain world, armies cannot off as it detected a launched missile. How- afford such security breaches anymore. In case ever, he knew the computer had glitches and of cyber war, if AI systems dealing with intel- was convinced there was an error. In fact, the ligence or military planning, for example, are machine misread sunlight glinting off cloud breached and a malware introduced to gener- tops and no missile had been launched on the ate a wrong outcome, the consequences can be soviets. If he had not recognized it as a false dramatic for an operation. Indeed, a slight al- alarm, at a time of high tensions between the teration of the system, even imperceptible for US and USSR, the soviet nuclear response the operators, can provoke a spill over effect. could have led to a disaster. Through that little piece of History, one must remember In case these tools would turn out to be unusa- that the reliability on the machine that can ble or in the case of an Electromagnetic bomb makes mistake as well as men, but that we

7 Doug Irving, “How Artificial Intelligence Could Increase the Risk of Nuclear War”, RAND corporation, 24th of April 2018, https://www.rand.org/blog/articles/2018/04/how-artificial-intelligence-could-increase-the-risk.html, accessed on the 23rd of September 2018

Artificial Intelligence 8 trust may make us do terrible things. And as Considerable technical concerns need to be AI would be considered as “more intelligent” discussed before AI applications get usual than the human being, we may rely even more within armies. Ensuring AI safety, security and blindly on its sensors and conclusions. As a confidentiality are the first steps. Then, logis- consequence, we must ensure, first, that the tic issues regarding this new type of tools or employment of such machine is preceded by a weapons need to be assessed, especially when clear assessment of its technological maturity considering their implementation during ex- and proceed according regular audits. Then, ternal operations. First, one of the major char- must be determined to which extend and to acteristics of the armies’ equipment is adapt- which tasks AI autonomy’s may be acceptable ability. Indeed, armies need capabilities that from an ethical point of view, but also to en- can be deployed in specific areas under specific sure safety. More generally speaking, safety conditions. From a technical point of view, AI protocols and precautions, of electricity, for applications have to be designed for high resist- example, have been established. The same will ance to weather conditions as well as energetic have to be thought and put in place for AI. lacks. Another logistic issue secondly arises : to implement in good conditions AI within land The “safety” of robots implementing AI sys- forces, soldiers must present specific skills. tems should also be ensured. Indeed, these technologies developed in the robots – i.e. drones – and acting autonomously in their Human resources issues environment could become a true strategic target for the enemy. Given the foreseeable Outsourcing and subcontracting are becom- costs of these capabilities, they should be able ing common practises in western armies, but to avoid incidents or the enemy’s attempt to with AI implementation in the European put them down. armies, the integration of AI expert within

Artificial Intelligence 9 them seems inescapable. If external experts AI biases or errors, the Armies cannot allow coming from subcontracting companies themselves to completely rely on machines. could theoretically realize audits and controls Then, computer scientists should establish of the systems, such a solution contains vari- a regular audit, test of loyalty and review of ous weaknesses. the machine algorithm. But operators should also be able to detect abnormal situations or To begin, with the integration of a growing outcomes from the system. Then two types of amount of external experts in their regular education should be provided. processes, the Armies will have to deal with serious confidentiality issues given the sensi- The education of the computer scientists will tive data the external auditors could access. probably be provided by universities or high This should arise as a bigger and bigger issue schools as the private sector skills on AI ap- with the spreading of AI technologies within pears in advance compare to the military one. military tools. The need of AI operators will However, digital jobs require high skilled pro- extend and these positions will need to be ful- files that are not yet really widespread. Coun- filled by soldiers able to take part in external tries around the world are already increasing operations on the battlefield. Ultimately, in- their research and recruiting to hire top scien- dependence and non-reliance of the armies on tist. Japan, China or the US are attracting a lot external companies for technical aspects of AI of Europeans eager to work with high finan- systems should be ensured. This should none- cial investments. As a proof, even enterprises – theless not exclude any cooperation between that usually offers better opportunities and re- private and public sector on AI system main- muneration than the public sector – have diffi- tenance or processing. culties to recruit experts. (Figure 1) As a result, the European armies should consider offering Hiring computer experts and AI experts will more than they usually do to their experts in emerge as a top priority for European armies order to become competitive. Once these ex- in the coming years. Even if avoiding the perts are recruited, updating their skills on a “black box”, i.e. not understanding AI deci- regular base should also be an important fea- sions, will mostly means to be able to detect ture. Several European armies have already es-

Artificial Intelligence 10 tablished in the past years a Cyber command lines or open letter to warn the international and therefore face similar issues. society of AI dangers and to ensure its appro- priate use. One of the main initiatives was The education of AI systems operators, which launched in 2017 by the Future of Life In- may be regular soldiers, also needs to be pro- stitute and its 23 Asilomar AI Principles9 that vided. Just as soldier are receiving cyber secu- have been endorsed by hundreds stakehold- rity training, they should receive information ers including scientists, AI researchers and and specific sessions on AI, depending on industries. Besides, at the end of 2017 the which systems they are working with. Effi- University of Montreal drafted a Declaration ciency and performance will indeed highly for a Responsible Development of Artificial In- hang on human-machine teaming. telligence10. These declarations are particularly insisting on the necessity to research, by the Anticipating skills needs, managing coming time we are researching for AI, on questions vacancies, recruiting new agents and retain- in computer science, economics, law, ethics, ing the best despite competition from private and social studies related to it. There is also a companies, improving the salaries are all chal- clearly wish for cooperation, trust and trans- lenges human resources will have to face with parency regarding on-going and future re- the implementation of AI within the Armies. searches. AI systems are expected to get along with the ideals of human dignity, rights, free- doms and cultural diversity, as well as under Ethical issues Human control. On a specific military aspect, LAWS are clearly not recommended. One of Ethical debates regarding AI are taking an the main ideas toward it remains that human- increasingly important place in the scientific kind, in the name of human dignity, deserves sphere, as well as in public one through me- not to be killed by a machine. dia. Given AI dual-use, these debates are way wider than the military application of these The military applications of AI are consid- new technologies. However, it is what arouses ered as such a threat that Google, which was most passions. Some study, like The malicious working on a project for the Pentagon, had to use of AI: forecasting, prevention and mitiga- justify itself and developed its principles to- tion8, are forecasting dangerous features of AI. ward AI systems elaboration. Indeed, Google Among main fears stand population monitor- got several resignations and a petition from ing, democratic elections breaching, or terror- employees that did not get along with its ac- ism. Overall, the idea that such a technology tivities. In its statement, the company gen- would fall into the wrong hands is the main erally speaking declares itself in compliance concern. with human rights and liberties and wish to avoid harmful applications. It nonetheless Recently, several stakeholders gathered them- declares its will to continue its collaboration selves and published ethical principles, guide- with the Pentagon in other areas than AI for

8 Future of Humanity Institute and others, “The malicious use of AI: forecasting, prevention and mitigation”, February 2018, available: https://img1.wsimg.com/blobby/go/3d82daa4-97fe-4096-9c6b-376b92c619de/downloads/1c6q2kc4v_50335.pdf, accessed on the 23rd of September 2018 9 Future of Life Institute, “Asilomar AI Principles”, 2017, available: https://futureoflife.org/ai-principles/, accessed on the 23rd of September 2018 10 University of Montreal, “Montreal Declaration Responsible AI”, 2017, available: https://www.montrealdeclaration-responsibleai.com, accessed on the 23rd of September 2018

Artificial Intelligence 11 use in weapons. Cyber security, training, hu- activated, can select and engage targets without man resources, health care have for instance further intervention by a human operator”12. been mentioned. As a consequence, if a strong international campaign has been launched to In the future, Armies should always carefully warn and fight against AI possible bias, it choose how and whether to delegate decisions should not hold AI technology implementa- to AI systems. Nevertheless, future adversaries tion in the Armies down. might not subscribe to the ethical standard European armies follow. This rise as a major However the importance of some ethical issue, as of course Armies will not accept to be questions cannot be put in question. Deci- demoted and to take the risk of loosing on the sion-making and eventually LAWS remain battlefield. From this situation, if other coun- issues where the debates are not yet conclud- tries get autonomous weapons capabilities, ed. The main doctrinal question regarding AI then they may have to equip themselves too, military applications is “Should humans stay in order to remain competitive. Moreover, in the decision making loop?”. the definition of an “autonomous weapon” appears as a bottleneck as different points of Nowadays, it appears that militaries do not views already confront themselves. intend to use fully autonomous weapons and want to remain responsible for lethal-force Belgium’s Parliament already voted in order decisions. The US Department of Defense to prohibit any development of LAWS and stated in that sense that “autonomous and many scientists refuse to work on AI pro- semi-autonomous weapon system shall be grams linked to the defence sector. However, designed to allow commanders and opera- AI researchers need to do more than opt out tors to exercise appropriate levels of human if they want to bring about change. Indeed, judgment over the use of force”11 More gen- they should rather explain their concerns to erally, people are recognizing that "black box" policymakers and military professionals. They AI approaches are unacceptable. If such sys- need to figure out AI systems’ limitations, tems are to be deployed in domains where the which would be possible only if scientists consequences can be serious and dangerous bring their technical expertise. A constructive – LAWS, military planning… – independ- dialog between all stakeholders – scientist, ent experts must have the ability to analyse NGOs, policymakers, industrials – appears the system and its operation. However, this as the way to implement in the best way AI is yet impossible to do. Most of AI systems within Armies. A solution, both in national, decisions path cannot be examine and there- European and international scene, will prob- fore their outcomes might then not be un- ably include regulation. Vincent C. Müller derstandable. Then, Human control remains and Thomas W. Simpson13 wrote about some essential, especially given the early age of AI. interesting possible legal solutions to estab- It should also be accounted that, from an eth- lish. They proposed to create an international ical and legal point of view, LAWS present agency dealing with LAWS regulation, to de- some features of mass killings weapons, and fine common technical requirements and to particularly swarming LAWS, which consist delimit specific situation where LAWS should of a dozens of unmanned aircraft that, “once be allowed.

11 US Department of Defence, Autonomy in Weapons Systems, Directive n°<3000.09, 21st of November 2012, P2 12 Ibid 13 Vincent C. Müller and Thomas W. Simpson, “Réguler les robots-tueurs plutôt que les interdire”, Multitude 2015/1, (n°58), P77-81

Artificial Intelligence 12 A need for regulation The first legal question is then whether the pre-existing general principles and rules will As all new technology, AI is raising new le- have to be amended or modified to fit new gal issues related to responsibility and deci- realities or if they remain valid and resilient. sion-making. The EU, true regulation cham- It turns out that the following principle may pion, already stated that new technologies be interpreted in a way that ensure true lim- did not come with new values. Consequently, its to the development of some AI systems, the Commission promised guidelines on AI such as LAWS. The principle of humanity, implementation from a legal point of view directly inspire by the Martens Clause of the in accordance with the European Charter of 1899 Hague Convention, prohibit the use of Fundamental Rights. On AI regulation in autonomous weapons by the simple fact that the defence framework, two tracks exist that they lack emotion as well as legal and ethical might put some restriction on these new judgement. Indeed, the principle of humani- technologies. On the one hand, general prin- ty requires respect for human life and human ciples – and rules applying on every kind of dignity. Would an AI system be able to recog- battle have been established through custom- nise a soldier hors de combat? Doubts also arise ary law or “contracting” law between States. when it comes to the principles of distinction The Geneva Conventions of 1949 and its between soldiers and civilians, as it might be additional protocols are the cornerstones of an issue for automated systems. AI systems International Humanitarian Law (jus in bello) would have to comply with principles of pro- and aim to protect civilians during a conflict. portionality and necessity too. On the other hand, several conventions reg- ulate or prohibit the use of specific kind of The UN and the meetings have initiated a weapons, like the Biological and Toxin Weap- reflexion on a specific regulation on the top- ons Conventions of 1975 or the Convention ic of AI autofocus weapons in the frame of on Cluster Munitions of 2008. the Convention on Certain Conventional

Artificial Intelligence 13 Weapons. The idea of a meaningful human between States (A) as well as between private control on weapons has been approved, which and public sector (B). de facto disapproved LAWS14. But the docu- ment presents only recommendations and is not binding any State. In fact, one of the Cooperation between States main issues remains to define what are au- tonomy and LAWS. As a result, there is no AI is going to give to any actor detaining it a specific regulation on LAWS or on AI system competitive strategic advantage, from a mil- in general. However, this should rise as a top itary point view, but also from an economic priority in the perspective of AI capabilities one. The industrial revolution made the UK development. If they were implemented with- the global hegemon in the 19th century. Nu- out any regulation, how to put in place a legal clear power and ICT development crowned responsibility in case of disrespect to the jus in the US as its successor since the end of the bello by AI? State, soldier, technician or selling WWII. AI may interfere with the current company may all be potentially responsible. world’s geopolitical landscape. Then, and as International Law remains a contractual law; there is a genuine need for For that reason, several countries have cooperation. launched themselves into a race – which ap- pears rather economical for the moment – aiming the development of AI. As a part of AI AN UNDENIABLE NEED FOR systems are developed with Machine Learn- COOPERATION THROUGH ing, having data is already a sign of power. Eu- EUROPE AND THE WORLD rope is late on AI advancements and do not host any big company holding a significant amount of data, which could arise as an issue As AI and data are becoming conveyor of for European companies that wish to train society revolution, economical growth, new their system. strategic issues and new geopolitical land- scape, it may become a new issue of interna- The two leading countries, in terms of invest- tional relations. According to Realism, States ments and intentions mostly – it is difficult would always act in accordance with their to assess and compare the technical progress national interest, and then, trust and coopera- of countries in such sensitive topic – appear tion among stakeholders seems to last as long to be the US and China. China has declared as they share common interests. In addition, its intention to grow as the global leader in sovereignty prevails when it comes to Defence AI by 2030. This appears achievable given the and International Security. Nonetheless, AI level of investments the government is pro- is such a growing issue in an evermore un- viding to its companies. In fact, the Chinese predictable world, that it becomes a priority model of military-civil fusion is a true asset to collaborate on this matter, to ensure that for the country as AI researches in the civil primary conditions for AI development are – but deeply connected to public power – sec- fulfilled. Even with trust issues, common un- tor will directly feed the military industries. derstanding and collaboration should occur US are not in such an efficient situation as

14 Fifth Review Conference of the High Contracting Parties to the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, Report of the 2016 Informal Meeting of Experts on Lethal Autonomous Weapons Systems (LAWS), 10 June 2016, CCW/CONF.V/2

Artificial Intelligence 14 the polemic around Google and project Ma- tation, especially in the defence sector, there is ven demonstrate it. Russia recently started to a crucial need for cooperation. Cooperation invest in AI, but benefits from a key alliance appears as the only way to established ethical with China in the military sector, as well as and legal borders to systems that – if not into in the private sector that may enhance their the right hands and for right purposes – can abilities to develop their capabilities. South become dangerous. It is also a way for medi- Korea, Israel or Japan can be considered as um powers to refill the gaps that have already medium AI powers too. appears between them and the US or China. None of European Armies and European Europe is late compared to these powers, as DTIB are able today to develop on its own private investments in AI only reach €2.4-3.2 such technical capabilities. Eventually, given billion in 2016, compared to €6.5-9.7 bil- the importance of coalitions and common ex- lion in Asia and €12.1-18.6 billion in North ternal operations for States nowadays, inter- America.15 operability between the AI capabilities should be ensured. Several national units dealing with AI are cre- ated around the world. The Pentagon request- Cooperation at the global level is essential. ed $70 million (even though $10 million have However, these years are tough times for mul- been authorized) to establish a National Se- tilateralism. Indeed, the step up of populism curity Commission on Artificial Intelligence, through the democratic world brought back supplemented by a Joint Artificial Intelligence inward-looking attitude of the States. The Centre (JAIC) and estimated that it could perfect example of that process is Brexit. Set- have a total cost of up to $1.75 billion over ting up a global international treaty assessing the next seven years. It will be tasked with the future developments of AI, as the NPT assessing how to improve the use of artificial did, would then seem an arduous challenge. intelligence capabilities for national security, But it may worth to try, especially to precisely its effects on national security risks and ethical define LAWS and establish an international considerations. The French Defence Minister legal framework about it. NPT has several Florence Parly announced on 16 of March weaknesses – recently, North Korea and Iran 2018 the creation of a "Defence Innovation cases – and is based on discriminations. Still, Agency", with a budget of 100 million euros it avoids nuclear bomb explosion since the for artificial intelligence, mostly to rapidly re- end of the WWII and ensure a minimum cruit of 50 experts in the field. France may level of international security. This outcome be, with the UK, one of the most innovative should inspire the international community. Army in Europe, but is clearly far from the real AI powers investments. Several meetings on LAWS occurred with- in the UN framework, gathering until 70 A Battle has started between stakeholders. Be- countries. Different kinds of position have tween States, which want to develop their AI emerged. China, Russia, and the United States related skills and capabilities, both in private reject a politically binding code of conduct and public sector, but also with NGOs and or a legally binding treaty on AI or LAWS, IOs which try to influence the way it will be and rather wish to focus on the definition of developed. But to ensure the right implemen- LAWS. On the contrary, some countries like

15 European Commission, “Factsheet: Artificial Intelligence for Europe”, 25th of April 2018, https://ec.europa.eu/digital-single-market/ en/news/factsheet-artificial-intelligence-europe, accessed on the 23rd of September 2018

Artificial Intelligence 15 Austria, Brazil or Belgium – which already facilitate AI researchers exchanges across the prohibit the development of these weapons in Channel. its army – wish a prohibition treaty, as well as a moratorium. In the middle of these opposed At the other side of the world, India and views stand some EU countries like the UK, Japan are cooperating on defence applica- France or Germany. They mostly pronounce tions of AI systems. The two Asian partners, themselves against LAWS but their position is along with Australia and the US, are pre- not completely clear when it comes to the ide- paring themselves to counter Chinese grow- as of politically binding agreement or Treaty. ing power. In the mean time, a next level To protect their sovereignty, the States are yet of strategic cooperation between China and not ready to commit themselves in any way. Russia have been reached given the four days of military drills in Central and East- Meanwhile, bilateral cooperation, which ern Russia, called Vostok-18 that occurred is already happening, is a way for armies to in September 2018. get more interoperable and to develop their AI capabilities. Following the path of the so- The EU and NATO are relevant organisations called “Special relationship”, the UK and the to organise a strategic cooperation between US have launched a common artificial intel- Europeans in order to face the new geopolit- ligence hub, which host new joint Defence ical reality. This strategic cooperation should Innovation Board meeting in May 2018. As involve the development of common AI capa- a consequence, experts can share experienc- bilities – to reduce their industrial costs – but es and innovation priorities, as well as deal also the elaboration of a common doctrine with industrial features. In July 2018, UK toward them. Indeed, technological revolu- and France also announced a partnership to tions bring a lot of change from the strategic

Artificial Intelligence 16 to the tactical level. The American Revolution civil project Galileo – the public regulated in military affairs is a proof of it. service: a encrypted signal with restricted ac- cess to provide information to European ar- NATO is already dealing with the future ca- mies and polices – or on the PESCO projects. pabilities of the alliance. Indeed, its military exercise to be held in Norway in October and The European Commission has already estab- November 2018 will test new technologies, lished a European Strategy about AI in April including AI systems. General Denis Mercier, 201817, even though it is focused on civilian Head of the Allied Command Transforma- aspects. It is proposing an approach follow- tion, is already calling for in-deep collabora- ing three objectives: increase public and pri- tion between the allied as he stated that “the vate investments and boost Europe’s scientific mistake would be for each of the NATO coun- base and industrial capacity, prepare the Eu- tries to develop its programs alone, which would ropean society for the socioeconomic chang- ultimately raise issues of technical, ethical and es brought by AI and ensure an appropriate political interoperability”16. However, uncer- ethical and legal framework. Meanwhile, a tainty is rising around the organisation as Eu- Declaration of cooperation has been signed ropeans experts show more and more doubts by 24 Member States to ensure a coordinated about the resilience of NATO and the reliance European plan on AI by the end of the year, on the US Umbrella. to increase the impact of investment at the EU and national levels and to exchange best Consequently, German and French Govern- practices. In addition, the European Group ment are pushing for more European cooper- on Ethics in Science and New Technologies, ation. During its annual speech to diplomats which is an independent advisory body for the French President Macron stated, “Europe the European Commission presidency, cur- can no longer rely on the United States for its rently research on Artificial Intelligence, Ro- security. It is up to us to guarantee European botics and Autonomous Systems. security”. The German Foreign Minister Maas also urged to boost military cooperation aim- The EU has started several projects, like ing to “form a counterweight” to Washington Smokebot – Civil robots to support fire bri- and “take an equal share of the responsibility” gades in search and rescue missions to perform as Europe-US relations cool. in harsh conditions – or Serena – an AI system to predict maintenance of industrial equip- A European cooperation on AI appears then ment. The EU funding for AI-related project as an accurate solution for European Armies, and research is over 1.5 billion of euros18. to develop their capabilities and doctrines. Since a few years, the European policy on de- If these initiatives yet have a civil purpose, fence and security took a slight step forward. they can be extended to the defence sector. The European States proved their ability to Most of the European Armies, which are just collaborate on some sensitive aspects of the at the early stage of developing Innovation

16 “L’erreur serait que chacun des pays de l’OTAN développe ses programmes seul, ce qui poserait à terme des problèmes d’interopérabilité technique, éthique et politique.”, in Nathalie Guibert, “Comment l’OTAN se prepare aux guerres du future”, 30th of March 2018, https://www.lemonde.fr/europe/article/2018/03/30/pour-la-guerre-de-demain-la-ressource-strategique-est-la- donnee_5278849_3214.html 17 European Commission, “Factsheet: Artificial Intelligence for Europe”, Op. cit. 18 European Commission, “Artificial intelligence: Commission outlines a European approach to boost investment and set ethical guidelines”, Press release, Brussels, 25th of April 2018

Artificial Intelligence 17 cells, are looking for interesting solutions for Collaboration between Armies and the pri- their future and civil projects can find some vate sector is required even though the work- application within them. The next steps force and public opinion are not really ready should then be to establish a concrete Europe- to see these types of companies working for an cooperation on Innovation, including on defence. (I.e. Google scandal with the Ma- AI issues, related to military environments. ven project) The key for a good collabora- States that show interest and true commit- tion will then be communication. Dialogue ment to a European cooperation could then should be reinforced within these companies aspire the development of their common stra- and as much as possible with public opinion. tegic autonomy. Today, no European State A bona fide dialogue should also be ensured is able to develop on its own AI capabilities: between the armed forces and the companies the cost of such programs is too high and no to deliver equipment with high-level of tech- country detains all the skills to conceive and nologies, that fit military requirements for an develop them. Then, there is a true need for a affordable cost. In the US several companies more unified European defence industry. – like Intel – are already looking for assist- ing, influencing and exchanging with the new National Security Commission on Artificial Cooperation between private and Intelligence. public sector Usually, traditional defence industries are If the traditional national defence industries criticised for being slow to move and for the have been able in the past to foster some of long time delivery of their products. None- the major advanced technologies – nuclear, theless, in a new industrial ecosystem gather- internet, aeronautic – they are overcome by ing new types of companies (i.e. start-up) and the private sector on AI. Even the American researchers, the defence industries might shift Defence Department has to negotiate and themselves to comply with more competitive launch programs to buy from the Silicon framework. As an example, Thales sets up its Valley. new AI global hub in Montreal – Cortex – to collaborate with several Canadian research in- Many projects exist yet. Amazon is providing stitutes. A strong European hub on AI should to the US government a facial recognition sys- also be established, gathering European com- tem, Google is working on the project Maven panies and funds to get a true AI European which aim to analyse drone image and Booz defence industry. Allen Hamilton recently sealed a contract with the Pentagon with the objective to allow European industrial autonomy toward AI a rapid employment of AI for intelligence, has to be ensured for obvious strategic issues. surveillance and reconnaissance operations. However, to develop a strong new cooperation In China, President Xi Jinping has considered between private and public sector, research and the civilian and military integration issues development budget, both at national and Eu- as one of the main topic to solve. The CCP ropean level, have to be reinforced. After the wishes to coordinate the Chinese economy 2008 economic crisis, governments have made with defence to give birth to a strong strategic important cuts in defence budgets. As a result, military planning. the European defence investments in research

19 “A corporate mind-set favourable to innovation is critical”, Interview of Marwan Lahoud in European Defense Matters, A magazine of EDA, 2016, Issue 10, P18-19

Artificial Intelligence 18 and development have declined by 30%19, REFERENCES even though the international security context makes the investments moving toward im- • Nick Bostrom, Superintelligence, Paths provements. On the other sides of the world, dangers and strategies, Oxford, Oxford the US invests 54.6 billion of euros per year, University Press, 2014, 432p. which represent four times EU member States • Neil Davison, “A legal perspective: spending combined, Russia had doubled its Autonomous weapon systems under investments between 2012 and 2015 and ac- international humanitarian law”, cording some prospections China will override International Committee of the Red Cross, its American competitor by 2022.20 2017, 7P. • Stephan De Spiegeleire and others, To try to fill this gap, the EU announced a “Artificial Intelligence and the future of 500 million of euros support in 2019 and Defense: strategic implications for small- 2020 in favour of the European Defence In- and medium-sized force providers”, The dustrial Development Programme.21 Achiev- Hague Centre for strategic studies, 2017, ing an integrated European DTIB is the cor- 140P. nerstone of European defence and security. • Denise Garcia, “Future arms, technologies, The fund is then expected reach €1 billion and international law: Preventive per year from 2021. Despite these efforts, the security governance”, European Journal of amount is definitely too low compared to the International Security, 2016 (1), P94-111. other powers investments. • Doug Irving, “How Artificial Intelligence Could Increase the Risk of Nuclear The European States and their Land forces are War”, RAND corporation, 24th of not playing the same role as the US or China April 2018, https://www.rand.org/ in the world. However, their interests need to blog/articles/2018/04/how-artificial- be protected in the same way. To that end, AI intelligence-could-increase-the-risk.html, represents many opportunities for European accessed on the 23rd of September 2018 Armies. Today, they are in the early phase of • Richard Kemp, “Legal Aspects of Artificial understanding what benefits it could bring Intelligence”, Kemp it law, November as well as the conditions to implement it in 2016, 32P. a right way. Being afraid or banning coming • Nicolas Mazzucchi, « Les implications technologies are surely not solutions. It is still strategiques de l’intelligence artificielle », unclear for many leaders about how AI could Revue internationale et strategique 2018/2 be integrated into existing infrastructure and (N° 110), p. 141-152. legacy systems. Then, European States should • Yannick Meiller, « Intelligence artificielle, gather themselves, provide financial support sécurité et sûreté », Sécurité et stratégie to advanced study on AI related issues and co- 2017/4 (28), p. 75-84. operate on research and development. How- • Motomu Shimoda, “Chapitre 3. Brain, ever, their sovereign concerns and their dis- mind, body and society: Autonomous agreements are obstacles to a deep European system in robotics”, Journal International collaboration. de Bioethique 2013/4 (Vol. 24), p. 41-48.

20 Ibid 21 Martin Banks, “EU offers up cash infusion to European defense industry”, 23rd of May 2018, available at : https://www.defensenews. com/global/europe/2018/05/23/eu-offers-up-cash-infusion-to-european-defense-industry/, accessed on the 24th of September 2018

Artificial Intelligence 19 improve thedailytasksofpreparation, training,exercises, andengagement. freely appliedby itsmember, whoseaimistofacilitateinteroperability and address thetopics.Finabel studiesandFood forThoughtsare recommendations the landforces andprovides commentsby creating “Food forThoughtpapers” to also athinktankthatelaboratesoncurrent events concerningtheoperationsof Groups for situations that require solutions. In short-term addition, Finabel is In the current setting, Finabel allows its member states to form Expert Task engagement ontheterrainshouldbeobtained. shared capabilityapproach andcommonequipment,ashared visionofforce- the harmonisationoflanddoctrines.Consequently, before hopingtoreach a focused oncooperationinarmament’s programmes, Finabel quicklyshiftedto structures butcontributestotheseorganisationsinitsuniqueway. Initially coalition;Finabelad hoc neithercompetes norduplicatesNATO orEUmilitary framework oftheNorth Atlantic Treaty Organisation (NATO), theEU,and Finabel contributestoreinforce interoperability amongitsmemberstatesinthe understanding viaannualmeetings. among memberstates’ officersandChiefs of Staff in aspirit ofopen andmutual on consensusandequalityofmemberstates.Finabel contact favours fruitful European ArmiesoftheEUMember States. The isbased Finabel informalforum Finabel aimstobeamultinational-, independent-,andapoliticalactorforthe future ofitsmembers.Finabel, theonlyorganisationatthislevel, strives at: reflections, exchange studies,andproposals oncommoninterest topicsforthe for cooperationbetween European for Armies:itwasconceived asaforum Created in1953,theFinabel organisation committeeistheoldestmilitary • • Finabel focusesondoctrines,trainings, andthejointenvironment. Contributing toacommonEuropean understandingoflanddefenceissues. bring togetherconcepts,doctrinesandprocedures; Promoting interoperability andcooperationofarmies,whileseekingto You findour studies will at GSM: +32 (0)483 +32 GSM: 712193 E-mail: [email protected] Quartier Reine Elisabeth Tel: (0)2 +32 441 79 38 www.finabel.org B-1140 BRUSSELSB-1140 Rue d’Evere 1

Responsible publisher: Mario BLOKKEN - Finabel Permanent Secretariat - QRE - Rue d’Evere,1 - B-1140 Brussels - +32 (0)2 441 79 38