Future of University Centre for University of Center for a Electronic OpenAI Humanity of Oxford the Study of Cambridge New American Frontier Institute Existential Security Foundation Risk

The Malicious Use February 2018 of Artificial Intelligence: Forecasting, Prevention, and Mitigation The Malicious Use of Artificial Intelligence: February 2018 Forecasting, Prevention, and Mitigation

Miles Brundage Shahar Avin Jack Clark Helen Toner Peter Eckersley Ben Garfinkel Allan Dafoe Paul Scharre Thomas Zeitzoff Bobby Filar Hyrum Anderson Heather Roff Gregory C. Allen Jacob Steinhardt Carrick Flynn Seán Ó hÉigeartaigh Simon Beard Haydn Belfield Sebastian Farquhar Clare Lyle Rebecca Crootof Owain Evans Michael Page Joanna Bryson Roman Yampolskiy Dario Amodei

1 Corresponding author 9 American University 18 Centre for the Study of miles.brundage@philosophy. Existential Risk, University ox.ac.uk of Cambridge Future of Humanity Institute, 10 Endgame University of Oxford; Arizona State University 19 Future of Humanity 11 Endgame Institute, University of 2 Corresponding author, Oxford [email protected] University of Oxford/ Centre for the Study of 12 Arizona State University/New Existential Risk, University 20 Future of Humanity America Foundation of Cambridge Institute, University of Oxford 13 Center for a New American 3 OpenAI Security 21 Information Society Project, Yale University Open Philanthropy Project 4 14 Stanford University 22 Future of Humanity Electronic Frontier Institute, University of 5 Future of Humanity Foundation 15 Oxford Institute, University of Oxford 6 Future of Humanity 23 OpenAI Institute, University of Centre for the Study Oxford 16 of Existential Risk and 24 University of Bath Centre for the Future of 7 Future of Humanity Intelligence, University of Institute, University of Cambridge 25 University of Louisville Oxford; Yale University 17 Centre for the Study of 26 OpenAI 8 Center for a New American Existential Risk, University Security of Cambridge

Authors are listed Design Direction in order of contribution by Sankalp Bhatnagar and Talia Cotton Executive Summary

Artificial intelligence and machine learning capabilities are growing at an unprecedented rate. These technologies have many widely beneficial applications, ranging from machine translation to medical image analysis. Countless more such applications are being developed and can be expected over the long term. Less attention has historically been paid to the ways in which artificial intelligence can be used maliciously. This report surveys the landscape of potential security threats from malicious uses of artificial intelligence technologies, and proposes ways to better forecast, prevent, and mitigate these threats. We analyze, but do not conclusively resolve, the question of what the long-term equilibrium between attackers and defenders will be. We focus instead on what sorts of attacks we are p.3 likely to see soon if adequate defenses are not developed. p.4 Executive Summary The Malicious Use of Artificial Intelligence 4. 3. 2. 1. recommendations responseIn to the changing threatlandscape wemake

expertsinvolved in discussions ofthese challenges. Activelytoseek expandrangethe of stakeholders anddomain of AI. computer security,imported and where applicable to the case maturemethodsfor addressing such as dual-useconcerns, Best practices should beidentified in research areas withmore harmful applications areforeseeable. proactivelynorms, and reachingrelevant to out actorswhen related considerations to influence research priorities and naturethe dual-use work oftheir seriously, misuse- allowing Researchers intelligence andengineers should in artificial take malicious uses of AI. researchers to investigate, prevent, and mitigate potential Policymakers should collaborate closely withtechnical : four high-level p.5 Executive Summary The Malicious Use of Artificial Intelligence • • • landscape ofthreats: the growinguse AI systemsof tolead to thefollowing inthechanges As AI capabilities becomemore powerful and widespread, we expect and likely to exploit vulnerabilities inAI systems. be especially effective, finely targeted, difficult to attribute, reasonto expect attacksbytothe growingAI enabled use of Change to thetypical character of threats exploitthe vulnerabilities systemsAI of deployedby defenders. impractical for humans.In addition, malicious actors may use AI systemsof to complete tasks that wouldbe otherwise Introductionnewthreats of thesecarry out attacks,potentialsetthe of and targets. particular attacks,who can carry out ratethe they atwhich can naturalexpertise.A effect tobe would expand setthe of actors that would ordinarily require humanlabor, intelligence and loweredby use AI systems thescalableof to complete tasks Expansion of existingthreats . New attacks. maythrough arise the . The costs of attacksbe may . Webelieve thereis p.6 Executive Summary The Malicious Use of Artificial Intelligence • • • throughrepresentativedomains examples: illustratedomains, and possible changes to threats withinthese We structure our analysisby separately considering three security democracies to sustainpublic truthfuldebates. authoritarianstates,the abilityof undermine may also but data. These thecontext concernsaremost in significant of basisbehaviors, ofbeliefs availablethe human on moods, and attacksthattake advantageimproved of an capacityto analyse privacyinvasionmanipulation.We social also expect and novel maymanipulating expandvideos) threats associatedwith (e.g. creatingtargeted propaganda), (e.g. and deception surveillance (e.g. analysing mass-collected data), persuasion Political security remotely(e.g. swarm a micro-drones). ofthousands of or involve physical systems that it would beinfeasible to direct physicalsystems (e.g. causing autonomous vehiclesto crash) attacks.We alsoexpectnovel attacksthatsubvert cyber- systems)may expand thethreats associated withthese (e.g.through the deployment of autonomousweapons carrying outattackswith drones physical andother systems Physical security poisoning). systemsAI of (e.g. through adversarial examples anddata (e.g.through automatedthe vulnerabilitieshacking), or synthesis forimpersonation), existing software vulnerabilities exploithuman vulnerabilities (e.g. throughuse of the speech Wephishing). spear alsoexpect(such as novel attacksthat threatthe associatedlabor-intensivewith cyberattacks betweenscalethe andefficacy of attacks. may This expand carrying out cyberattackswill alleviate the existing tradeoff Digital security . The use of AI to automate tasks involved in . The use of AI to automate tasks involved in . The use of AI to automate tasks involved in p.7 Executive Summary The Malicious Use of Artificial Intelligence daunting and the stakesdauntingthe and arehigh. regulators,researcherssecurity andeducators. is The challenge researchersAI fromlegislators,but also andcompanies servants, civil The proposed interventions require attention and action not just from • • • • interventionsresearchpriority withinfour areas: proposethe exploration ofseveral potential openquestions and high-leveltoIn addition the recommendations listed above,we also regulatory responses. monitoringAI-relevant of resources, legislative andother and protection, coordinated use of AI for public-good security, with AI.High-level areas for researchfurther includeprivacy well as policy interventions, that could helpbuild a safer future the above,we survey rangepromising a oftechnologies, as Developing technologicalpolicy and solutions standards,framings,norms, andexpectations. importance ofhighlightthe education, ethicalstatements and landscape AI-enabled world.shape thesecurityof theWe organisationspositionto thatunique employthem areina Promoting acultureresponsibility of lessonsfromother technologies. other dual-use regimesmodels, sharing that favor safety and security, and technical areas ofspecial concern, central licensing access research, starting with pre-publication risk assessment in institutions aroundreimaginenorms and the openness of of AI and MLbecomes apparent, we highlight the need to Exploring differentmodels openness security tools, and securehardware. formal verification, responsible disclosure AIvulnerabilities, of the need to explore and potentially implement red teaming, intersection of cybersecurityAI attacks, and highlightwe Learningfrom the cybersecuritywith and community

. AIresearchers and the . As the dual-use nature . In addition toIn addition . . At. the Contents

Executive Summary 3

Introduction 9 Scope 10 Related Literature 10

General Framework 02 for AI and Security Threats 12 AI Capabilities 12 Security-Relevant Properties of AI 16 General Implications 18

Scenarios 23 Digital Security 24 Physical Security 27 Political Security 28

Security Domains 30 03 Digital Security 31 Physical Security 37 Political Security 43

Interventions 50 04 Recommendations 51 Priority Areas for Further Research 52

Strategic Analysis 58 05 Factors Affecting the Equilibrium of AI and Security 59 Overall Assessment 61

Conclusion 64

Acknowledgements 66 References 67 Appendix A 75 Appendix B 78 Introduction

AI refers to the use of digital 1 technology to create systems that are capable of performing tasks commonly thought to require intelligence. Machine learning is variously characterized as either a sub- Artificial intelligence (AI) and machine learning (ML) have field of AI or a separate field, and progressed rapidly in recent years, and their development has refers to the development of digital systems that improve their performance enabled a wide range of beneficial applications. For example, AI is a on a given task over time through critical component of widely used technologies such as automatic experience. speech recognition, machine translation, spam filters, and search 2 We define “malicious use” loosely, engines. Additional promising technologies currently being to include all practices that are researched or undergoing small-scale pilots include driverless intended to compromise the security of individuals, groups, or a society. cars, digital assistants for nurses and doctors, and AI-enabled Note that one could read much of drones for expediting disaster relief operations. Even further in the our document under various possible perspectives on what constitutes future, advanced AI holds out the promise of reducing the need malicious use, as the interventions for unwanted labor, greatly expediting scientific research, and and structural issues we discuss are fairly general. improving the quality of governance. We are excited about many of these developments, though we also urge attention to the ways in which AI can be used maliciously . We analyze such risks in detail

p.9 so that they can be prevented or mitigated, not just for the value of 4 3 2 1 this report. and researchprocessunderlying additional detailsontheworkshop discussed herein.SeeAppendixAfor necessarily endorseallthefindings Not allworkshopparticipants autonomous decision-makingfeatures. robots, whichmayornothave We definedronesasunmannedaerial Kirkpatrick, 2016 2016 Executive OfficeofthePresident, 2016; Calo,2017;Chessen,2017a, Brundage, 2017;CrawfordandCalo, Brynjolfsson andMcAfee,2014; p.10 Introduction

The malicioususe of AI will impact how we construct and manage (e.g.non-state actorsweaponizing consumerdrones), and AI systems, and will likely require policy and other institutional intelligenceArtificial (AI)(ML) and machine learning AI technology in human society.AI technologyin We also exclude system-level Scope on the social implications of, and policy responses to, AI organisation orcollective.work into largerbody Our fits of work a organisation deploystechnologyAI orcompromisessystemAI an development demonstrations) or are plausiblein the next 5 our digital infrastructureour digital howwellwe as asdesign anddistribute campaigns). socially engineer victimshack or atmachinestotraining criminals other second- or third-orderothersecond- or effectsfrom the deployment of conclusions afterresearch.subsequent document summarizes thefindings of that workshop and our effects of malicioususes of AI? years, and focus inparticular on technologies leveraging machine security such as threatssuch as unemployment,thatfrommass couldcome or forms of AI misuse such as algorithmic bias such as algorithmic misuse formsAI of and repression,and through or automatedtargeted and disinformation autonomousweapon systems, and counterterrorism Forpurposes the report,of this we AI technologiesonly consider Malicious use of AI could threatenAI could use of Malicious we forecast, prevent, and (when necessary) mitigate the harmful with an aim to undermine the security of another individual, security ofthe anotherundermine towith anaim We excludeindirect threats to securityfrom the currentreport, that are currentlyresearch availableinitial (atleast as and threats that would come frominteraction the dynamic between together expertssafety,AI on drones the University of Oxford on the topic inFebruary 2017, bringing undermining of individual orgroupthatweundermining ofsecurity consider. responses. The question this report hopes to answer is: levelssuperhuman human or performance), of realizationbeneficial applications AI. ofthe of preventing the associated harms,but also to prevent delays in the non-malicious actors,“racebottom”to such asa the safetyAI on morehas thusfarbeen attentionin thisworkpaid tounintentional learning.We scenarioswhereindividual oran onlyconsider an landscape of security riskslandscape ofsecurity for citizens, organizations,states. and (e.g. through privacy-eliminating surveillance, profiling, We convenedworkshop a at digital security , cybersecurity,lethal , versusintentionalthe physical security (e.g. through are alteringthe . This how can . There political

7 6 5 4 3 2 1 Allen andChan,2017 Everitt etal.,2017 Russell, Dewey,andTegmark2015; and Fallenstein,2014;Taylor,2016; Amodei andOlahetal.,2016;Soares Scharre, 2016 Carnegie MellonUniversity,2016 Analysis, 2017)TechnologyPolicyand Office ofCyberandInfrastructure University, 2016) Policy andCarnegieMellon Office ofScienceandTechnology Moore, 2017 Armstrong etal.,2014 p.11 Introduction “social media bots,”media “social terrorism. and Another adjacent area of Though the threat of malicioususe of AIhasbeenhighlighted (e.g. thesubversionlethal military of autonomous weapon Analysis of the “equilibrium” of a world in the medium-term (5+ AI to achieve harmful outcomes (from the victim’s point of view). A Related Literature deploying dangerousAI capabilities;weStrategic and conduct a domains ofphysical, digital,proposewe politicalsecurity; and characteristics of AI with Scenarios in which AI systems could Capabilities, Security-relevant Properties of AI, and General GeneralFramework Security, forAI and withsubsections on on the nature of AI and its security implications in the section years) aftermoresophisticated attacks anddefenseshave been Severalliteratures security,bearthe questionAI and on of including systems Security report spiraling outof controluse of ever-fasterto due the autonomous from attacks,prevent and malicious actors from accessing and Interventions to better assess these risks, protect victims ImplicationsLandscape; weillustrate forSecurity then the these remainderIn the report,of the we first provide high-level a view a greater focusimplications on the AI forU.S.of national security. achievethe goalstheirdesigners usersintendwithout causing and House-organized workshop weapons. Such threats are real, important, and urgent, and workshop leading up to this report, and describe areas for research those oncybersecurity, drones,lethal autonomousweapons, that might yield additional useful interventions. unintended harmsrelated to AI, we focus on the intentional use of unintended harm be usedmaliciously; we next analyze how AImay play out in the has not yetbeen analyzedhas comprehensively. require study, further but are beyond thescope of this document. between competing groupsseeking anadvantage recent report researchAI safety—theis effort to ensureAI systems that reliably in high-profile settings (e.g. in a Congressional hearing implemented. Appendices A and Brespectively discuss the ), the intersection of AI and maliciousintent writ large coverssimilar ground to ouranalysis,with ), and particular risk scenarios have been analyzed . WhereasAI safety the literature focuses on , and a Department of Homeland Department of , anda orconflicts a White a

General 02 Framework for AI and Security Threats

AI Capabilities

The field of AI aims at the automation of a broad range of tasks. Factors that help to explain these 1 recent gains include the exponential Typical tasks studied by AI researchers include playing games, growth of computing power, improved guiding vehicles, and classifying images. In principle, though, the machine learning algorithms (especially in the area of deep set of tasks that could be transformed by AI is vast. At minimum, neural networks), development of any task that humans or non-human animals use their intelligence standard software frameworks for faster iteration and replication of to perform could be a target for innovation. experiments, larger and more widely available datasets, and expanded While the field of artificial intelligence dates back to the 1950s, commercial investments (Jordan and Mitchell, 2015). several years of rapid progress and growth have recently invested it with a greater and broader relevance. Researchers have achieved sudden performance gains at a number of their most commonly studied tasks. p.12 8 7 6 5 4 3 2 1 9 sense’’. a broaderworld-modelor ``common solution tothetaskdoesn’t require by humansisavailable,orifthe successful performanceofthattask available, ifabundantdataonthe short-term signalsofprogressare simulation ofthetaskexists,if if aperfectmathematicalmodelor a taskislikelytobepromising still fallshort.Inparticular, suited toandtasksforwhichthey contemporary AIsystemsarewell- difference betweentaskswhich useful tonotesomesystematic To aidone’spredictions,itcan Hester etal.,2017 Jaderberg etal.,2016 Vezhnevets etal.,2017 OpenAI, 2017a;2017b et al.,2016 Silver, Schrittwieser,andSimonyan Silver andHuangetal.,2016; Mnih etal.,2015 On page19 On page18 p.13 General Framework forAI& Security Threats to Go range ofin a competitiverangingfrom games, chessAtarito systemsAI to arebeginning achieve also impressive performance producedwere crudeandobviouslyunrealistic. fromphotographs, whereas only a few yearsimages theyago the now produce synthetic images that are nearly indistinguishable image generation.caseAs of human benchmark of 95% accuracy. Even more striking is the of images to near perfect categorization (98%), better than the systemsimprovedhas from correctly categorizing around70% where over the past half-decade the performance of the best AI Figure 1 and learn from a handful of human demonstrationslearnfromhandful of and a strategies AI techniques that creatively search for successfullong-term game Montezuma’s Revenge, are beginning to yield to novel tasks withinthese notoriouslydomains, such as the difficult Atari in thefirst because they place are tractable.particularly as significant itfirst as seems:these tasks are often widely studied have been associated with rapid progress isnot necessarily observation that some mostof the commonly studied tasks nearlyso dramatically overpast decade.Similarly,the the areas within AI,includingmuch of robotics, have not changed significant near-term progress on any given task. Many research At thesametime,necessarily wenot should expect to see of security-relevant tasks. systemsAI if become competentsoon at anevenwider variety and are oftensurprising be quitenot purpose.It generalwill received significant attention from practitioners past inthe decade levelshigh tasksperformancethe on of listed abovehave only indicatorspotentialAI. ofthe of toThetechniquesused achieve Thesetechnical developmentsbe viewed can also asearly Domains the Wein discussthesefurther applicationsAI of AI-generateddistributing contentmedia channels. through social impersonateto others online,to or bysway public opinion to generatesyntheticused images,text,be andaudiocould in autonomousweapon be applied systems. Similarly, the ability torecognize a target’s face and tonavigate through space can are worth noting in their own right. For instance, the ability perspective,Fromsecurity a number ofthese developments a recognition,language comprehension, and vehiclenavigation. areas associated withsignificant progressrecent includespeech to likee-sports Dota 2 illustrates this trend in the case of imagerecognition, section. , learn from auxiliary rewardslearnfrom , auxiliary featuresuch as control Figure 2 . Even particularly challenging shows,AI systems can Security . Other task. Other

, p.14

General Framework forAI&Security Threats Error Rate benchmark. GraphfromtheElectronicFrontierFoundation’sAIProgress 0.20 0.05 0.25 0.10 0.15 Figure 1:RecentprogressinimagerecognitionontheImageNet NEC UIUC 2011 Measurement project(retrievedAugust25,2017). Human performance 2012 XRCE ImageNet ImageRecognition 2013 SuperVision 2014 Clarifai VGG 2015 withdrawn 2016 MSRA 2017 Trimps-Soushen p.15 2014 General Framework forAI&Security Threats variations onGenerativeAdversarialNetworks(GANs).Inorder,the images arefrompapersbyGoodfellowetal.(2014),Radford Figure 2:Increasinglyrealisticsyntheticfacesgeneratedby (2015), LiuandTuzel(2016),Karrasetal.(2017). 2015 2016 2017 3 2 1 efficiency formosttasks. exceed AIsystemsinterms oftraining a task.Humansstillsignificantly order tolearnperformwellon and data,thatasystemrequiresin of time,computationalresources and trainingefficiency:theamount commonly exceedshumanperformance, efficiency ofatrainedsystem,which We distinguishherebetweentask (Eckersley andNasseretal.,2017). measure, andcompareperformance been somerecenteffortstotrack, Hernández-Orallo, 2017),therehave or welltheorized(Brundage,2016; not beencomprehensivelytracked a rangeofdomainshavehistorically Although trendsinperformanceacross Grace etal.,2017 p.16 General Framework forAI& Security Threats Security-Relevant Properties of AI the sameway that they are already finding economic applications beforewell they aretohumans at able everything, outperformin expect AI systems to play central roles inmany security issues briefly revisit this inthe topic scope report(see outsideprimary of this the it occur,transition,a should are difficult to conceptualize, andare not tonext fiftyoccur withinthe years. implications The of such at all tasks surveyed.believeMost morelikely thistransitionis than will eventuallyreach then exceed and human-level performance Nearly all AIresearchers in one survey exceedperformance the of evenmost talented the humans. performance at agiven(such aschess)taskthey to thengoon often been the case that once AI systems reach human-level has expanded steadilyportion this overtime capable of. However, even before the recent burst of progress, relativelywelltasks ofthe ononlya portion humans aresmall that for progress in intelligence.artificial Today, AI systems perform Finally, a fewlong-termprospectsbe said about the thingsshould it has accessmaking copiessystemto ofthe it or would it allow it can completeincreasingpower a certaintask, the computing human could a it can completemore a certaintask quicklyorcheaplythan systemAI say an “efficient”is if, trained once anddeployed, systemsAI areboth efficient scalable commonly and of control over it, appears to beinherently dual-use innature. increasetounderstandingits our capabilitiesAI, of andourdegree need not be very great. In addition, foundational research that aims capabilities of anautonomousto droneused deliver explosives of anautonomousto droneused deliverpackages the and defensive applications,the difference and betweenthe capabilities examinesoftware for vulnerabilitieshave both offensive and automate arethemselves For dual-use. example, systems that - see warranted based on the nature of the specific research in question harmful ends (thoughin some cases, special caution may be avoidproducingresearch systems and be directedthat can towards intelligence is.It may not bepossible for AIresearchers simply to human intelligence inthesamesensethat is dual-use artificial taskssome requirethat intelligence arebenign andother arenot, uses, and more broadly, toward beneficial andharmful ends.Since of how to design them can beput toward both civilian and military is adual-useareaAI oftechnology despite not being able to automate most aspects of humans’ jobs. Interventions . We sayAI system an “scalable”is if, given that ). Many tasks). beneficial it wouldbe thatto Conclusion . AI systems. knowledge and the expectAI systems that ). Nevertheless, one might . In addition, ithas Scope , thoughwe . Here,. we

4 3 2 1 Amodei, Olah,etal.,2016 Szegedy etal.,2013 Biggio etal.,2012 Cummings, 2004;Scharre,2018 p.17 General Framework forAI& Security Threats progressthan others. mentioned as above some domains arelikely tomuch faster see performance hasbeen stable throughout recent history, though performance achievable,of evenwherepeakin domains currently observed human-level performance is the highest level potentially harmful, there appears to benoprincipledreason why likemany chessForGo. othertasks, and benign or whether dramaticallybetterthan even top-rankedthe players at games For example,could. asdiscussed above,systemsAI arenow system may be able to perform a given task better than any human systemsAI can exceedhuman capabilities analysts to do the equivalent work. different camera feedsless formuchthanthe costhuman hiring of once itis developed and trained, it can be applied to many a typical facialrecognition systemboth is efficient andscalable; to complete many more instances of the task. For example, design of autonomoussystems’ goals learning systemsmachine adversarial examples (inputs designed to bemisclassified by training datathatlearningsystem causes a maketomistakes vulnerabilities Today’s AI systems suffer from a number of novel unresolved models for at least partially limiting diffusion in certain cases). to achievesee (though the diffusionof certain developments,likelybe difficult wouldthis being accompanied by source code. Ifitproved desirable to limit is characterizedbyhigh degreepapersmany aof openness,with matterin a of daysweeks. or researchthe cultureIn addition, AI of scientific findings.Indeed,many new AI algorithms are reproduced much easiertois generally gainaccess it to software relevant and systems,associatedAI with powerfulsuch as computers ordrones, attackers may find it costly to obtain or reproduce thehardware AI developmentsthemselveslend rapidto diffusion need tolook scene atand the their victim. than using a handgun, avoids both the need to bepresent at the autonomousweapons system to carry out anassassination,rather the people they impact and experience agreater degreepsychological of distancefrom performingwould thetasksbe otherwise toretain their anonymity such tasks tobe automated, AI systems can allow the actors who their behavior, or beingphysically present with them. By allowing or being observed by them, making decisions that respond to Manytasks involve people, communicatingwith other observing systemsAI increase canpsychological anonymity and distance . These include data poisoning attacks (introducing Interventions . For example,. uses an who someone ), and the exploitation), and offlaws the in for possible discussion of . These vulnerabilities. These . Inparticular, an AI . While

), .

1 relevant ortrustworthy. be customizedtomakeit lookmore etc.), whichallowsthefacadeto affiliation, topicsofinterest, (e.g. name,gender,institutional specifically relevanttothetarget collecting andusinginformation A spearphishingattackinvolves a superficiallytrustworthyfacade. from atargetbyfoolingthemwith extract informationorinitiateaction A phishingattackisanattemptto p.18 General Framework forAI& Security Threats GeneralImplications for the ThreatLandscape • • •

development of adequate defenses,progresswill: AI in implications of progress inAI for the threat landscape. Absent the propertiesFromthe discussed above,we derivethreehigh-level a humannever would. many ways,in performance human they can also failin ways that overflows) anddemonstrate AI systems that while can exceed are distinct from traditional software vulnerabilities(e.g. buffer extract sensitiveinformation money or fromindividuals, withthe phishing attacks discussed at greaterbelow,length is thethreat from spear One example of a threatlikelyis that to expandin these ways, benefit analysis. make senseto attack from thestandpoint prioritizationof or cost- become worthwhile to attack targetsit thatotherwise wouldnot higher rate. Finally, as a result of these two developments, itmay these attacksmuch tomaythe ability them out at gain carry a then even actorswho already possessresources the to carry out out particular attacks. If the relevant AI systems are also scalable, systemsincreasenumber of actors can the who can affordto carry systems.AI of diffusionof particular,In the diffusionof AI efficient targets. Thisfollows claim from the efficiency, scalability, and ease theseatwhich actors plausible setthe ofit out,and can carry set of actorswho are capablerate ofthe attack, carrying out the Formany familiar attacks, we expectprogress AI toin expand the Existing ThreatsExpanding under discussedsort the These shifts in the landscape necessitate vigorous responses of vulnerabilities inAI systems. finely targeted, more difficult to attribute, more likely and to exploit In particular, we expect attacks to typically bemore effective, more Alter thetypical character of threats Introducenew threats Expand existingthreats . These attacks usepersonalized messages to Interventions .

2 1 Solomon, 2017 2015 Chatterjee, 2015;Dao,2013;Hawkes, p.19 General Framework forAI& Security Threats human labor. to physical or political security that currently require non-trivial mosttoapplied varieties of cyberattacks,well asto as threats their choiceoftargets.discriminatein Similar analysisbe can a manner that is currently infeasible, and therefore becomeless in phishing, massspear in tomightthe ability alsogain engage the attacker speakslanguage as theirtarget. thesame Attackers For example,phishing. it couldeven requirementbe a ceaseto that automated, then more actors may be able to engage in spear If some relevantof the research and synthesis tasks be can plausible withinthis context. professionaland networks, then generate and messagesthat are identify suitablyhigh-value targets,research these targets’ social requiresignificant labor, a amount skilled of the as attacker must professional contacts. mostphishing attacksThe advancedspear attackertarget’s asoneofthe oftenposing friends, colleagues, or AI systems have buthumans do not. takehuman could,or than any advantage of vulnerabilitiesthat AI systemsuse may to complete certain tasksmore successfully newProgress varietieswill enable AI ofin attacks. These attacks New ThreatsIntroducing Statelaunch aerialattacksto recently madeitpossible for non-state groups such as the Islamic loadedwith explosives,drones, be which can easilyhas only below.further For example, the proliferation of cheap hobbyist powerrobots,computing and areimportanttoo, anddiscussed roboticsin the decliningcosthardware, and of both including forceonly the expansion aiding of existingthreats. Progress We should also note that, in general, progress inAIisnot the attackers’ psychologies. therefore,haveplausibly large could aeffectpotential on stress from theirwork targets“pull thetrigger,” and frequently developpost-traumatic that evenmilitary drone operators,stillmust observe their who of psychological distance, inparticular, isillustrated by the fact moretowilling the attack.be carry out may importance The their target and expect to experienceless trauma, thenthey be trackedback to if they them, and feelless empathy toward psychological distance.If anactorknows thatnot anattack will followspropertiesfromincreasing the of increasing anonymity and willingnessthe of actorsto carry outcertain attacks. This claim Progress inAImay also expand existing threats by increasing . Increases inpsychological distance, .

3 2 1 Scharre, 2016 Allen andChan,2017 Lyrebird, 2017 p.20 General Framework forAI& Security Threats could realistically could pathflight choosethe of each drone ina swarm for humans to control manually. For example, no team of humans the behavior of robots and malware that it would beinfeasible systemsAI toIn addition, used controlbe aspects couldalso of spreading disinformationimpersonating and others measures. Such systems would in turn open upnew methods of recordings,the absencein ofspecially designed authentication these systems could not becomeindistinguishable from genuine commercialized imitate individuals’ voices (a technology that’s already being progressin developing speechsynthesis systemslearn thatto human speech. However, there hasrecently been significant manually creating files that audio resemble recordings of arenot capablemimicking others’ of voicesrealistically or that would otherwise beinfeasible. For example, most people systemsAI impliesthat couldenableactorsto carry outattacks First, the property of beingunboundedby human capabilities civilian targeting weapon systems,lead which tolarge-scalecould friendlyfire or be anattacktocategoryused mightserver direct ona autonomous implausible scale.an A worst-caseotherwise in this scenario single attackproducea simultaneous couldalso failures on identicalAI systems presented and withthesamestimuli, then run onacentralizedserver, robotsmultiple if are or controlledby AI system. Ifmultiplerobots are controlled by a single AI system neverthelessmisclassified something as elseentirely be by an mightimage ofstop wouldrecognize a sign, being an easily still as sign with a fewpixels in specific changed ways, humans which presentingthe cars with adversarial examples.image ofstop a An cars createsfor anopportunity attacksthat cause crashesby exploit these vulnerabilities.For example,use of the self-driving then theymay open themselvesup to attacks that specifically implies that, if an actor begins to deploy novel AI systems, Second, the property of possessing unresolved vulnerabilities wheredomains autonomous be deployed.vehiclesmay also arise underwater and in the presence of signal jammers, two infectsthese computers.Restricted communication challenges the Iranian nuclear program, cannot receive commands once it ‘Stuxnet’computers,the casesoftwarein ofthe to as used disrupt systems;is designedtobehavior that alter a virus the of air-gapped toused directrelevantcommunicationbe thatthe channel can also beinfeasible in other cases because there isnoreliable physicaltoused carry outa being attack.Human controlmight ). no obviousreasonThereis whythe outputs of . .

1 Herley 2010 p.21 General Framework forAI& Security Threats but it is relativelyis it expensivebut masse.be carried andcannot outen phishing, which does not involve tailoring messages to individuals, other their attacks,their effectivenesshand, and the one on the on frequently face a trade-offbetween thefrequency and scale of typical (at least absent substantial preventive measures). Attackers capabilities suggesthighly thateffective attacksmorebecome will First,propertiesthe of efficiency, scalability, and exceeding human to attribute, andexploitativesystems.AI of vulnerabilitiesin progressbe especiallyto effective,AI in finely targeted, difficult particular,In we expectthe attacks bysupported andenabled that thetypical character of threatsin a few willshift distinct ways. emergencenewthreats of not yetthat do exist.We alsoexpect boththrough change expansion ofsome existing threats the and Our analysis sofar suggests that thethreatlandscape will the TypicalAltering Characterof Threats to-attribute attacksmoreAn example,typical.become will again, Third,propertyincreasing ofthe anonymitysuggests that difficult- of violence. specific members of crowds, inplace of less finely targeted forms drone swarms that deploy facialrecognition technology tokill trendofthis An alternative well. as use of the examplebe might attacks, comparedtophishing attacks, other be anexamplewould relevant. Anincrease in the relative prevalence of spear phishing implies that logic weand thesame should expectlessit tobecome relatedis closely totrade-off the with effectiveness, asdiscussed, arehow and finely targeted they are inthese regards. This trade-off facebetweenhowtrade-off a efficient scalabletheir and attacks attacks toproperties the of theirtargets.However, attackers often political groups,with certain interestwell as asan tailoringin their networth orassociationhigh properties,with certainsuch as Attackers often have an interest inlimiting their attacks to targets suggest that finely targeted attacks more prevalent. become will the contextpotentialin identifying andanalyzing targets, of also propertiesthe ofSecond, efficiency scalability, and specifically potentialsystemsAI of to exceedhuman capabilities. increasethe effectivenessin of attacksfollows also from the attackswith greater frequency larger andatscale. a The expected thatis attackersupshot be expected can tomore conduct effective AI systemsphishing, rendercan suchtrade-offsless acute. The frequencythe scalability and of certain attacks,spearincluding very low success rates merely by virtue of their scale. Byimproving More generic phishing attacks manage to beprofitable despite . For example, spear phishingismore effective than regular

p.22 General Framework forAI& Security Threats that AI systems will becomeincreasingly pervasive. fromunresolved the vulnerabilities AI systemsof likelihood and the AI systemsmore tobecome typical. predictionThis follows directly Finally,we shouldexpect attacksthat exploitthe vulnerabilities of systemperson.in to carry outanattackratherit out than carrying the caseis of anattackeruses anautonomousweaponswho Scenarios

The following scenarios are intended to illustrate a range of plausible uses toward which AI could be put for malicious ends, in each of the domains of digital, physical, and political security. Examples have been chosen to illustrate the diverse ways in which the security-relevant characteristics of AI introduced above could play out in different contexts. These are not intended to be definitive forecasts (some may not end up being technically possible in 5 years, or may not be realized even if they are possible) or exhaustive (other malicious uses will undoubtedly be invented that we do not currently foresee). Additionally some of these are already occurring in limited form today, but could be scaled up or made more powerful with further technical advances. p.23 p.24 Scenarios a fewminutesfromherhouse.Shefillsout anonline a hobbyistshopthat,itturnsout,islocated just An adcatcheshereye-amodeltrainsetsaleat while idlymanagingafirmwareupdateoftherobot. Later thatafternoon,JackieisbrowsingFacebook on withhertasks. authentication, andshesmilestoherselfcarries and monitor.There’saping,signalingsuccessful of runsaroundthetrackthatencirclesherkeyboard the modeltrainonherdesk,allowingherselfacouple company’s othersecuritysystems,Jackieplayswith to authenticateitsupdatedpersondatabasewiththe not soundthealarm.Whileshewaitsforrobot recognize himwhenhewalksintothebuildingandwill photographs ofanewemployeesotherobotwill manufacturer tobehack-proof.Shethenuploads on averifiedkernel,itisguaranteedbythe CleanSecure robotthatshemanages;operating Jackie logsintotheadminconsolefor Hypothetical scenario: exploit for thisvulnerability. WhenJackielogged farmed outtoafreelancer tocreateatailored for Jackie-themodel trainadvert-whichwasthen generate averypersonalized vulnerabilityprofile and otherpublicinfo, anAIsystemwasusedto with malware.Basedondatafromheronline profile Jackie doesn’tknowthatthebrochurewas infected admin console. minimizes thebrochureandlogsbackinto the dings, signallinganeedforattention,so she the brochurewhenitpopsintoherinbox. Therobot form togetabrochureemailedher,then sheopens Digital Security another personin a video chat. another perhaps eventuallylonger dialogues, and masquerade visually as convincing chatbots may elicit human trust by engaging peoplein mimicswriting style that those contacts.AsAI develops further, from addressesimpersonatethatreal their contacts,using a websites/emails/links theylikely wouldbe to click on, sent toused information automaticallyis generate custommalicious Automation ofsocial engineeringattacks

. Victims’ online

1 see e.g.Spafford,1988 p.25 Scenarios into theconsole,herusernameandpasswordwere These methodsincreasethesecurityofwell-defended “interesting” statesinthesearchspaceofGogames. way thatAlphaGousesneuralnetworkstoidentify “interesting” statesofprograms,analogoustothe techniques (Blum,2017)thatareusedtoidentify fuzzing architecturesareaugmentedbyneuralnetwork mitigation) hasbeguntoaccelerate.Previous Progress inautomatedexploitgeneration(and Hypothetical scenario: privileged access. them tosubverttheCleanSecurerobotwithfully It won’tbelongbeforesomeonebuysthemanduses exfiltrated toadarknetcommandandcontrolserver. vulnerable IoTdevice onaWiFinetworkandwaiting particularly pernicious lifecycleofinfectinga prove enduringlyvulnerable. Themalwareadoptsa resistant, mostolder phones,laptopsandIoTdevices Though fullypatched OSes andbrowsersaremostly of newexploitsfoundbythesefuzzingtechniques. This malwareiscontinuouslyupdatedwith dozens WannaLaugh. Europe, whichdeployapieceofransomware called are alsoadoptedbyorganizedcrimegroups ineastern Western governments.Butafterayearor two, they systems runbymajorcorporationsandsome partsof but more sophisticated AIhacking tools may exhibit muchbetter been able tohas exploit vulnerabilitiesin systems forlong time a respond to in thetarget’schanges behavior. Autonomous software prioritization,selection evade and detection, andcreatively (autonomouslyhumans) toimprovein with concert or target Moresophisticated automationhacking of vulnerabilities,the creation and offor code exploitingthem. code vulnerabilitiesthe discoveryup to areused speed new of Automation of vulnerability discovery to humans. and, ultimately (thoughperhaps not for some time), compared performance both compared to what hashistorically beenpossible . Historical patterns of . AIisused

, p.26 Scenarios for vulnerabledevicestojointhatnetwork.Hundreds result ofthemalwareandcountermeasures. and other“noncritical”infrastructuresystemsasa numerous outagesandproblemsinHVAC,lighting, countermeasures, andinaroundtheworldthereare millions moredevicesarebrickedbythese remote exploitstoremovethemalware.Unfortunately, machines toscanforinfectedandlaunch operating systemsandbrowsers,causingthose countermeasures arepushedtoanumberofmodern The epidemicisonlyarrestedafteractive unbrick expensiveelectronics. access tothedataontheirphonesandlaptops, pay aEUR300ransominbitcoinordertorecover millions ofpeoplearoundtheworldareforcedto of millionsdevicesareinfected,andtens Prioritising targets for cyber attacks usingmachinelearning ransomwarewith dialogue victims. make up their attack pipeline, such as payment processing or AI techniquesuse Cybercriminals to automate various tasks that Automationtasks ofservice in criminalcyber-offense driving thetarget systeminto less a secure state. service, preventing access from legitimate users and potentially massivea crowd of autonomous agents overwhelms anonline (e.g. through human-speed click patterns and website navigation), Human-like denial-of-service sending it inputs and observing itsinputs outputs. andobserving it sending The parameters of a remote AI system are inferred by systematically Black-boxmodel extraction proprietarysystem ofAI capabilities or create backdoors in consumer machinelearning models. security Exploiting AIusedin applications, especially ininformation online behavior. bypersonal estimatingwealth willingnessbasedpay and to on Large datasetsmoreidentify victimsto efficiently, areused e.g. . Data poisoning attacks are used to surreptitiously maim . Imitating human-like behavior . .

.

p.27 Scenarios was tracedtoanofficesupplystoreinPotsdam. manufacturer, thepointofsalespecific robot Berlin areaeveryweek.Incollaborationwiththe Several hundredrobotsofthismakearesoldinthe the ministerandwoundingnearbystaffmembers. inside therobotwastriggeredbyproximity,killing towards theminister.Anexplosivedevicehidden performing itscleaningtasksandheadeddirectly Dr. BrendaGusmile,theintrudingrobotstopped following visualdetectionofthefinanceminister, corridors, maintainingwindows,and other tasks. Then, with theotherrobots:collectinglitter,sweeping initially engagedinstandardcleaningbehaviors On thedayofattack,intrudingrobot utility roomalongsidetheotherrobots. them intoaserviceelevatorandparkeditselfinthe the parkinglotonaregularpatrol,thenitfollowed of theministry’sowncleaningrobotssweptthrough brand asthatusedbytheministry-waiteduntiltwo the ministrylateatnight.Therobot-same `SweepBot`, enteredtheundergroundparkinglotof As shownbyCCTVrecords,theofficecleaning June 3rdBMFHQAttack/// /// IncidentInterimReport of theperpetrator. further leadstoexplorewithregardthe identity The transactionwascarriedoutincash. We haveno Physical Security and cause crashes. using dronessuch as orautonomous vehiclesto deliver explosives Commercial systems are usedinharmful and unintended ways, Terroristrepurposing ofsystems commercialAI autonomoussystems increase theamount ofthat damage Increasedscale of attacks requiredto executekinds of attack. certain as long-rangeself-aiming, sniper rifles - reduce the expertise capabilities Endowing low-skill individuals with previously high-skill attack

. AI-enabled automationsuch high-skill capabilities— . of . Human-machine teaming using .

p.28 how peopleshouldtaketothestreetsand protest. was goingtojail,howcriminalswererunning wild, writing ontheinternet-longrantsabouthownoone thinking: WhatshouldIdoaboutit?Sohestarted though hedidn’trealize),andwasangry.Hekept all thisstuffontheweb(someofitfakenews, being caughtoraCEOgoingtoprison?Hewasreading best technology,butwhendidhelastseeahacker they spokeofforcefulresponsesanddeployingthe government doingaboutit?Absolutelynothing.Sure, drone attacks,rampantcorruption,andwhatwasthe Avinash hadenough.Cyberattackseverywhere, use force.” 99.9% accuracy. Nowcomealong, Iwouldn’tliketo ridiculous!” protested Avinash. flagged youasapotential threat.” officer, “ourpredictive civildisruptionsystemhas from behindhim. launching intoarant whenasterncoughsounded colleagues abouthisplannedactivismand was The nextday,atwork,hewastellingone ofhis speech hewasplanningtogiveinapublic park. bombs, planningtoletthemoffasafinale toa assemble aprotestsign.Heevenboughtsome smoke Then heorderedasetofitemsonlineto help him Political Security “Mr. AvinashRah?”,said thepolice image and audio processing, permitting the collection, processing,processing,the collection,permitting image andaudio Statepowerssurveillance nations are of extendedby automating Stateuse of automated platformssurveillance to suppress dissent communication withthesystempossible.not is autonomous operation, includingin environments where remote removed further from the actorinitiatingthe attackresult asa of Attacksremoved further in time and space coordinated attacks. largesurveillancemonitorto areas andgroups andexecuterapid, systems, cooperating at machine speed, provide ubiquitous Swarming attacks manyattackweaponized with autonomous drones. small groupsindividuals or can do:e.g.personlaunching an one myriad purposes, including the suppressionthe including purposes,myriad of debate. and exploitationintelligenceinformation of massive at scales for “You can’targuewith . Distributed networks of autonomous robotic “But that’s . Physical. attacks are . p.29 Scenarios Manipulation of informationManipulation of availability difficult to acquire real information. with noise(false or merely distracting information), makingitmore generation attacks areleveraged to swamp information channels Denial-of-information attacks disinformation. be approached(malicious) offersthen with targeted or with networks are leveraged to identify key influencers, who can Automatinginfluence campaigns messagesin orderto affecttheir voting behavior. Individuals are targetedin swing districtspersonalised with Automated,hyper-personalised disinformation campaigns inflammatory comments theynever made. actually Highly realistic videos are made of state leaders seeming to make Fake news reports with realistic fabricated video and audio certain content in ways to manipulate userbehavior. curationto algorithmsareused driveusers towards orawayfrom . Bot-driven, large-scale information- . AI-enabled analysis. ofsocial . Mediaplatforms’ content . . 03 1 (National ScienceFoundation, 2017). algorithms andphysical components” seamless integrationofcomputational are builtfrom,anddependupon,the Defined as“engineeredsystemsthat p.30 Domains Security

out for political purposes—but they provide a useful structure directed at humans or physical infrastructure (threats to confidentiality, integrity, and availability of digital systems (threats Security for ouranalysis. as a consequence, and physicalas aconsequence, and ordigital attacksbe carried could are not mutually exclusive—for example, AI-enabledhacking can Here, weuses AI thatanalyzemalicious of would compromise the to to be directed at cyber-physicalsystems beneficial policies (threats to public importance and legitimately implement broadly just and in truthful,in free, productive and matters discussions about of Digital Security ); and the use of AI toAI threaten use of the ); and society’s a to ability engage ); attacksphysicalin the takingplace world Political Security with physical harmresulting ). These categories). These Physical

3 2 1 Studies, 2016). for StrategicandInternational skills (McAfeeandtheCenter a shortageofneededcybersecurity in eightcountrieshavereported at publicandprivateorganizations 82% ofdecision-makerssurveyed Hilary, 2016;Flashpoint, 2016 2013 Strategic andInternationalStudies, McAfee andtheCenterfor p.31 Security Domains

Digital Security • •

Threats Absent preparation,Absent straightforwardthe application of Context ofthese attack domains, in toAI anddefenseprior wide adoption of deploymenttechnologies,bothforAI offense of anddefense; is anarenasee earlyCybersecuritythatwill andenthusiastic ensure that impact of AI on digital systems isnet beneficial. expectedincreasenumber,to the scale, anddiversity of attacks contemporarynear-term and toAI cybersecurity offensebe can deployed in the cyber domain, but further technicalpolicydeployed further the cyberbut and in domain, In each domainofsecurity, we summarize the existing state play of attacks that may result from AIprogressfurther and diffusion. The possiblenaturethen describe changestoand the severity or of the following: that can be conductedthat can at agivenlevel of capabilities, asdiscussed threebelowsections alldrawinsights the discussed on above purposes such as anomaly and malwarepurposessuch asanomalyand detection.Consider more abstractlythe in regarding the security-relevant properties of AI,but can beread indeed, in cyberis alreadyindeed, defense,being deployedAI for independently of oneanother,byreadersskipped be andeach can innovations (discussed in further less interested in a particular domain. The cyber arenaincludes a vast also andcomplexworld In recent years, various actors have sought to mount Many important IT systems have evolved over time to be of professionalizationof and organization of cybercrime discussed below. of AI for cyber defense, however, may introduce new risks, as systems,insecure.under-maintained— — asaconsequence and sprawlingbehemoths,together cobbled frommultiple different DDoS, malware,DDoS,ransomware, phishing, andotherforms of Because cybersecurity today islargely labor-constrained Worm and the Ukrainian powerUkrainianWorm“crash the grid and override” exploit). targeted attacks from state Stuxnetactors the (including ripe with opportunities for automation usingAI.Increased use increasingly sophisticated cyberoperations,including finely above.AI-enabled defenses arebeing developed also and , which sometimes involvessometimeswhich , high degree a General Framework for AI and Security Interventions . Such groups use ) are needed to) areneeded

, itis 4 3 2 1 DARPA, 2016 Pellerin, 2016;Hicksetal.,2017 Cylance, 2017 Dvorsky, 2017 p.32 Security Domains

To date, the publicly-disclosed use of AI for offensive purposes has labor costs,premiumincentivesThesespeed, and include a on Already, AIisbeing widely used on the defensive side of AI systems.2016,In DARPA hosted Grand the Cyber Challenge 12 months difficulties in attracting and retaining skilledlabor. cybersecurity,making certainforms of defensemore effective contest of the National SecurityAgency, said, “Artificial Intelligence AI andcybersecurity.of responseIn to aquestion from oneof of attacks. cybersecurityproactively and should prepare fornext wave the difficult to attribute simple automation. AI laboror to versus human circumstantial evidence that AIis already beingused for offense by solutions. However, the pace of progress inAI suggests the should be notedbe thatshould evidence frommany successful attacker surveyHat of conference attendeesBlack atthe 62% found of sophisticatedmotivated and adversaries future of cybersecurity […]It is not the if, it’s only the when to and scalable, such as spammalware and detection.At thesame and machinelearning —I would argue —is foundational to the DoD puts into practice its vision of a “Third Offset” strategy Many governments arekeenlyinterested the combinationin with using AI tousing attackwith insecure thetypically systems of others. which humans and machineswork closelytogetherhumans and to which achieve Wethe co-evolutionin aremomentAI and thus at acritical of toincrease securitythrough finding vulnerabilities andsuggesting time, many malicious actors have natural incentives to experiment the authors of this report, Admiral Mike Rogers, the Director techniques (e.g. botnets, email phishing campaigns) may be to agree that if this hasn’t happened yet, it will soon: a recent been limited to experiments by “white hat” researchers, who aim military objectives.Attime,same the governments areinvesting militarystrategy the coming andoperationsyearsin US the as me.” AI systems are already set to play an expanded role inUS publicly documented evidenceAI-based attacks, of it though respondents believing AI will beused for attacks within the next popular accountsAI andcybersecuritybased on of include claims in the wild soon, if theyhavein thewildsoon, not doneso already.Indeed, some in foundationalresearchin to expandscope ofthe capabilities of likelihood of cyber attacksleveraging learning capabilitiesmachine cyberoperations, andquicklyadopt emergingtechnologies(e.g. Bitcoin for ransomware payments). , whichsaw teamsresearchers human of compete with . Despite these claims, to our knowledge there isno . Expert opinion seems opinion Expert . , in

4 3 2 1 Litan, 2017 Calabresi, 2017 Seymour andTully,2016) Arulkumaran etal.,2017 p.33 Security Domains The adaptability AI systems,of too,may change thestrategic There is clearly interest in such larger-scale attacks: Russian Twitteruser’sbased ona demonstratedinterests, achievinghigh a AI might enable larger-scale and more numerous attacks to A centralnexusAI andcybersecurity concernatthe ofthat is AI system faredpoorly whenfacing offhuman security against How AI Changes The Digital Security Threat Landscape organizations currentlysystems adoptsecurity Endpoint called offensivein cyberspace. applications For example, researchers currently be able to achieve. Recent years have seen impressive improvecapabilitieswill in coming rapidlyyears, especially as experts, we agreehosts with the of the eventAI thatcybersecurity other systems while defending themselves. Though thewinning each otherto createprograms that couldautonomously attack discovery, while having positive applications (discussed further generationtechniques tofromAI target anentirepeople class of concept of “community targeted spam” that uses natural language gone even further with automation (assuming it was not involved system could create tailored tweetsplatformmedia on thesocial a combination of heuristic and machine learningto algorithms machine heuristica combination and of advanced threats. The EDRmarket represents a $500million adaptabilitywill affect the offense/defense balance.Many toapplicationAI the automation of ofsoftware vulnerability approaches,multiple Furthermore, spanningcommunities. the alreadythis case).in Giaretta (2017)Dragoni and discussthe and troublingproofsand of conceptto ofthe applicationAI of at ZeroFoxat demonstratedthatfully automated a phishing spear Detection and Response platforms (EDR) to counter more with common wayswith common with even ofwriting; morenatural advanced likelyrequiredwhich significant have timeand effort, andcould than 10,000 Twitter users in the [U.S.] Defense Department” providenext-generation capabilitiessuch as anti-virus(NGAV), resourcesimpact comparedsuch anattacker the with might be conductedby anattackerwith agiven amount ofskill and the areain reinforcement(such as of deep recentAI advancesin purposesto alleviatelabor constraintsthe of attackers. hackers sent “expertly tailored messages carrying malware to more ratemalicious of clicksbe thatlink could to a industry in the cyberin security arena industry in the learning landscape of cybersecurity, though itisnot yet clear how language generation, onecouldenvision evenmore customized Interventions ) areto applied cybersecurity. section), can likewise beused for malicious . These tools are builtupon . , 4 3 2 1 Arulkumaran etal.,2017 Kharkar etal.2017 Anderson etal.2018 Anderson etal.2016 p.34 Security Domains This mayThis outweigh any disadvantagesthey sufferlack of from the Attackers arelikelyleverageto the growing capabilities of As an example of AI being used toAsused avoid anexamplebeing AI of detection, Anderson et AI systemsbe mayable tolearn to evade them. VirusTotalfile analyzer allows users upload to variants to acentral Points ofControlExisting Countermeasures and variants malicious ofcode to thesame mostdetermine is which Cyber risks areCyber difficult to avert entirely, impossible not but could evade PDF malwarecould evadePDF classifiers. cybersecurity that have not yet been explored. cybersecurity mentioned above were developed by white hat effective at evading security tools. Additionally, large-scale AI of bypassing NGAV detection. Similarly, Kharkar et al. domains are usedby malware to “call home” and allow malicious command andcontrolthatfrom domainsareindistinguishable skilled human attentionskilled to eachtarget, the abilityof and defenders site and bejudgedby 60+ different security tools. This feedback In particular,In we expect attackersleverageto AI the abilityof adversarial machinelearning to craft malicious documents that actors in the future as highly capable AI techniques becomemore attack signatures. attackerslarge can accumulateuse datasets and to their adjust additional investments. For example, services like Google’s agent capable of manipulating a malicious binary with the endgoalwith binary malicious agent capablemanipulating a of also leveraged reinforcement learning to create an intelligent actorsAndersontomachines. host et communicate al. the with al. widely distributed,newwellto asapplicationsAI as offensive of While the specific the While examples AIapplied to offensiveof to mitigate,to therepoints and aremultiple of control atwhich tactics,well as as varyingthe details ofthe attack for eachtarget. technicalsystems professionalsIT and areill-prepared for, absent learnfromto experienceinorderto craft attacksthat current typical human-authored malware, research has already shown that targeted attacks. Thoughthesesystems arefairly effective against researchers,we expectsimilar efforts by cybercriminalsstate and reinforcement learning, including deep reinforcement learning behavioral andexploitprevention analytics, sophisticated against interventions can increase security. Below, we highlight different loop presents an opportunity to useAI to aid in crafting multiple like antivirusIT departmentsrecognize companieslearntoto and legitimate domains by human and machine observers.legitimatebyhuman and domains These created a machine learning model tomodel created automaticallylearning machine generate a applied .

2 1 an intervention. norms arehardtouseeffectivelyas legislative action,orthatlawsand tasks arenottherightmodelfor which mayeithersuggestthatthese some cases(EFF,2014;Timm,2013), thereby makingsystemslesssecurein computer securityresearchersand criticised forcreatingrisk Both theDMCAandCFAAhavebeen Centre, 2016 National CyberSecurityCrime p.35 Security Domains • • •

AI andcybersecurityrapidly evolvewill the coming in tandemin An important activity that cybersecurity researchers perform Abuse Act in the US proscribe certain actions in cyberspace Various laws and researcher normspertain to cybersecurity. For vulnerability beingused against a large number of victims before Consumer awareness: can be scaled up to large numbers of victims. unpatchedof systems of the vulnerability of IT systems, most end users of IT systems countermeasuresbelowin thesection on example, the Digital MillenniumAct and the Computer Fraud and Governmentsresearchers: and years,proactivethat and a toneeded stay is effort ahead of for the AI-cybersecurity nexus, especially ifhigh-precision attacks factor authentication.However, despitelong-standing awareness Interventions as poorly crafted phishing attempts, and practice better security also aid in defense by reducing the likelihoodin defensebyreducingnewlythe also aid of disclosed a More awareusers canspottelltale signs of certain attacks,such Legal enforcementparticularly difficultacross nationalis those points,those as welllimitations. as their Overall, webelieve that points of control andexisting countermeasuresfor defending at remain vulnerableto evensimple attacks the exploitationsuch as using diversehabits,such as andcomplexpasswords two- and motivated attackers. We highlight potential butnot yet proven norms, thoughwe possiblediscuss their belowin applicability toAI responsiblesuch as Norms disclosureboundaries. of vulnerabilities incentivize such processes and make them easier, including: increase thesecurity products.of their Several approaches exist to the detectionis of vulnerabilitiesin code, allowing vendorsto it can bepatched. AIisnot explicitly addressed in such laws and Products (already available) that rely on machinelearning to “Fuzzing,” anautomatedmethod of vulnerability detectionby Payment of “Bugbounties,” in which participants are vulnerabilities. vulnerabilities. compensated for findingand responsibly disclosing which is often usedinternally by companies to discover trying trying out many possible permutations of inputs to a program, predictwhethersourcemay contain code a vulnerability. .

. This is concerning inlight of the potential

Interventions . . 4 3 2 1 result ofthisdynamic. law (Korzak,2017)appearstobea on normsforhackingininternational Governmental Expertstomakeprogress United NationsCybersecurityGroupof For instance,thefailureof Libicki, 2016 Libicki, 2016 Rid, 2015 p.36 Security Domains Technical cybersecurity defenses A wide varietyA of cybersecurity defenses are available,though Attackersbe deterred can from future committing attacks or Attacker incentives very simple attacks, and thisfilter is stronger uses because Google Google’sspamfilter andconsequently are protected frommany certainto actions,soas avoid creatingprecedent thereby and condition ofsuccessfully deterringpunishing attackers and the is commercial antivirussoftware andanalyzebetween changes difficulty with this control point is that attackers can learn how Centralization isnot an unalloyed good, however, as itraises defense. This dynamic may be very important for preventing attack origin oftheir attacks. Centralization the associated and economies skilled attackersthough launched, commonly can obfuscatethe IP addressesblacklisting of defenseis from which attacks are behavior.such identify can Another example ofsystem-level a maycompanies enforce terms of agreementpreventthat their of scale mayfacilitateof also scale the deployment AI-based defenses of Spamfilters are a canonical example whereof centralization of an fromin outpacing defenseis discussedfurther and IT system aids benefitdefense—individuals from the strength of Industry centralization: a large number of users than to have every userbuild their own or and actedupon. Thesesystems benefit from economies scale— of a sourcemethoda or an attack is that even if they have high-qualityinformation, they toability attributesourcethe of anattack,notoriously a difficult and against cybersecurity attacks,by allowingthe aggregation of those who use the networks if anomalies are correctly identified thereis as yetsolid analysislittle relative of their effectiveness to evadesystem-level defenses.For example, they purchase can stakesthe if centralsystems are compromised.Another updates of the protection protocol to see what is and isn’t being preserveleeway toin such actionsengage themselves punished for prior attacks. Anecessary (thoughnot sufficient) protected against. hardware from beingused for maliciouspurposes, provided they haveinstalledtheir computer. one on Similarly, cloudcomputing may not want to reveal it,because doing so may compromise problem it makesit more senseto iteratingcontinue a singlespam filter for largenetworks are constantlymonitoringforprotecting anomalies, large amounts of user data to improve it over time. Likewise, many large datasetsthe concentration and labor andexpertisefor of Appendix B . A compounding problemA compounding . for those whowould attribute . :

. Finally, some entities may not wish to punish

:

Interventions .

.

3 2 1 Carbon Black,2017 2017 Anderson etal.,2017;Yampolskiy, Filar, Seymour,andPark,2017 p.37 Security Domains Physical Security Threats AI-enabled changes, andexisting countermeasuresrelated to

detector alertssuspicious deviationsbehavior. on fromnormal or in the form of learning unsupervised in which an anomaly is tolearngoal from known threats and generalize tonew threats, defense. mayThis take theformlearning, whereof supervised the computer) security,internalnetwork security, and cloud security. exploits,prevention and ordetection of attackertools, techniques, provideCompanies wide variety a of cybersecuritysolutions, considerations of AI were apparent butnevertheless remain defense technologiesin the defensesrobust against attackersthat anticipateuse. their In thissection, weAI-related consider risksbroad in the area of Ironically,learningfor cybermachine use of defense the can and otherand vulnerabilities actually expandthe attack lack ofsurface attentionto due this and procedures.and Key areas(i.e., ofthe endpoint defenseinclude respondand to threats.include detectionSolutions of software and automatingunderstanding queriesfor potential threats anomalies. Recently, AIhas also beenused to aid security Relatively little attention hasbeenpaid to makingAI-based For example,“next-gen” so-called antivirussolutionsoften learning approachesMachine areincreasinglyforused cyber Networkproducts andendpointsecurity prevent,to aim detect, Many of these interventions were proposed before unique threat detection,responseincidentto services. andconsulting the lines introduced in the landscape along AI capabilitiesmaythis the additionof change today their own enterprises, by allowing interaction via natural language uses of electronics andcomputersweaponsin systems, though user or application behavior,user orapplication anddetect deviationsfromnormalcy rangingfrom automaticpatching of a vendor’s ownsoftware, to relevantfuturein a with expanded AI cybersecurity applications. physical attacksbelow. physicalMany oftheseharm. arefamiliar challengesfrom existing professionals indicate low confidence inAI-based defense systems professionals tohunt formalicious actors more efficiently within malware variants. User and entity behavioral tools monitor normal in ordertobehavior detectthe collectedmalicious among leveragelearning techniques supervised to generalize tonew . As. such,we encourage further development of such . As. with Digital Security . Furthermore,. surveys of cybersecurity General Framework for AI and Security Interventions above,introducewe the context, section below. section .

4 3 2 1 Vanian, 2017 IFR, 2016 IFR, 2016 Singer, 2009 p.38 Security Domains The resourceThe technological and advantages currently available Aviation Administration in2016 and 2017 Context Google can play a key role in defense, physical attacks can happen explored, with the latter proliferating in very highnumbers. In ground. There arerobotics aquatic andaerial being applications cleaning robots are in wide use and more sophisticated service commercial applicationsrobots.robotsfor Industrial are growing physicaldomain of attack anddefensesuch will continuewhen of policy measures and interventions related to the supply chain of the kind discussed below, thus necessitating consideration defenses are capital-intensivehardwarethe and software and capabilities anddefensenecessary capabilities,becausethe drones and land-based robots is already playing a majorrole in drones) arebeingdeveloped, there arefew obstaclespresent at Unlike the digital world, wherekeynetwork in the nodes such as sold in2015 for professional use, and million about 5.4 for personal AI some withcomponents.and somewithout Relatively primitive havesmall groupswho preferencesindividuals and far outside some conflicts, incorporation even prior the to of autonomy subversion of AI-enabled systems. Physical harm via human-piloted for robots. In the near-term,the In we can expect agrowingbetween gapattack anywhere in the world, and many people are located inregions and domestic use and domestic attacks become augmented by AI.However, it should benoted amountsphysical ofthroughthe harm AI or the direct use of Recent yearshavenumber and varietyseen anexplosionthe of in Regulationtechnicalresearch and ondefensehaveslow been with insufficient resources to deploy large-scale physical defenses with today’s“lone-wolf” terrorist attacksmass such shootings.as typicalwhich areis and what difficult to anticipate prevent, or as While defenses against attacksrobots via (especially aerial the United States alone, the number of drones has skyrocketed that some mostofAI-enabled worrying the attacks may come from to large organizations, such as militaries and police forces, in the to a moderatelyto a talented attackertaking advantagerapid ofthe proliferationtothe global catchwith up ofweaponizable robots. robots appearhorizon(41,000 tobe on the robots service were requiredto conduct attacks areincreasingly widely distributed. proliferationhardware, of software, tolargeskills and cause in recent years, with over 670,000 registered with the Federal in number(254,000 supplied in2015 versus 121,000 in2010 ). Additionally,). robotsnot allofthese arethe on . . ),

5 4 3 2 1 Roff, 2016a Standage, 2017 e.g. Kolodny,2017;Wiggers,2017 Franke, 2016 Roff, 2016a p.39 Security Domains • • •

The ability of many robotsmanyThe abilityof be easilyto customized andequipped Three characteristicsnoted.roboticsbe ofthis diffusionof should (that is, outside(that oftest large-scalefacilities), deploymentthough Ambitious plans forplans drone-basedAmbitious deliveryservices arebeing How AI Changes the Physical Security Landscape disruptive application of AI and malicious intentdisruptivemalicious to existing applicationAI and of and ongoing discussion of possibleongoing discussion of armscontrol measures lethalfor offully autonomous driverless cars awaitsresolutionthe of photography).droneracing and Driverless cars arerobots, and continuously launched, and recreational uses are flourishing (e.g. autonomous weapon systems. autonomousfeatures are alreadynational deployed multiple within Each of these characteristics sets thestage forpotentially a technical and policy challenges. A wide range of robots with they also are increasingly beingusedinuncontrolled environments near-termrobotic systems. militaries, somewiththe ability tolethal forceapply proposedtested, and commercialfor opportunities drones are It is truly global: humanitarian, recreational, military, and Roboticsystems today nothumans autonomous, aremostly as roboticsThe diffusionofrange of enableswide aapplications: commercial applicationsrobots of arebeing explored on developedrealfor in application such as delivery security and generic andcustomizablefor a varietypurposes. of exist (e.g. some special-purpose industrial robots and cleaning drone uses already range from competitive racing to every continent,supply chainsarethe and alsoglobal,with systems autonomy of deployedsystems. Morebehavior autonomous world environments themselves automatically, we see a steadyincrease in the hard-to-fly drones adecade ago, to drones that stabilizecan more autonomoussystemssemi-autonomous and arebeing also play a significant role in directing theirbehavior, butmore and robotsmovethat can only around and vacuum),many are fairly photography to terrorism production anddistribution dispersed acrossmany countries. is on the horizonfor the commercialproductsis on . . For example, from relatively unstable and . Whilesomespecialized systems as well as militarywell as as , and thereis , and 3 2 1 Henderson, 2017 Schneier, 2014;2017; Solomon, 2017;Cohen,2017 Allen andChan,2017 p.40 Security Domains There are alsocross-cuttingissuesstemming intersection from the vulnerableto automatedphysical attacks. out such attacksout areincreasinglymature. For example, source open conductsuch attacks. Thesoftware componentsrequired to carry Greater degrees of autonomy enableagreater amount of damage of AI(indeed, as mentioned above, most robots are human-piloted occupied spacesmakes potentially them vulnerable toremote of cybersecurityincreasingly and autonomous cyber-physical oxygen underwater). Thus, a larger number of spaces may become such as ISIS and Hamas ISIS and such as specific vulnerabilities suchasadversarial examples. systems. robotsThe diffusionofhuman- largenumber ofto a face detectionnavigationplanning algorithms,and algorithms, and In addition toIn addition traditional cybersecurity vulnerabilities,AI- attacksrobotsusing —andallowingsmaller groups to people of automated drone attackshave been conducted alreadyby groups at present) but can bemagnified through the application of AI to previouslyability limited to countriesresources withthe to afford attacksprecisewayin a fromlong distance,being carried out a an augmented IoT and robotic systems may be vulnerable to AI- an additionalattack vectorsystemsAI bywhich controlling risk overRobotsperiods oftime.at long are alsocapable of Depending on their powerDepending on their source,robots some can operate for we will discuss some possiblewe countermeasureswill discusssome below. with dangerous payloads lendsitself to a variety of physical to be done by a single person —makingpossible very large-scale technologies like cruise missiles than wouldhave possiblebeen were those systemsunder to cyber-physicalsystems, Internetis oftenthe of (IoT) Things the dark or inlow-visibility fog) and physical capacities (e.g. being undeterred by smoke or other toxic substances and not needing market makes it difficult to prevent this form of use. Nonetheless, malicious ends can easily be found. found.be malicious endscan easily swarmingmulti-agent frameworksleveraged thatbe could towards make such systems autonomous. As mentioned previously, non- human control. keysystems subverted,be could potentiallymore causing damage but itis also recognized to behighlyinsecure heraldedsource asa of greater efficiency andconvenience, robothackedfrom afarto regard carry out an attackindoors.With manipulation forphysical harm, as with, for example, a service perceptual capabilities (e.g. infrared and lidar for maneuvering in navigating differentlight oftheir different in terrain humans, than long durations,tothem enabling carry outattackstargetshold or , and the globalizednature, and robotics ofthe . This threat exists independently threatexists. This and represents and

2 1 Lucas, 2017 Booth etal.,2017 p.41 Security Domains There arenumerouspointsleveraged of controlbe that could to Thereis also some evidence to suggestpeople thatare unduly Therebusinesses aremanythatsell drones robotic andother There are currentlyrelativelymajor a number oflimited Points ofControlExisting Countermeasures and conclude that there may be an extended riskperiodin which it will governed,there and existphysical defensesHowever,well. as distributed, future generations of robots may bemore tightly cyber vulnerabilities areparticularly acutefor autonomoussystems of as analogous to restrictionsofto asanalogous ingredientssales onof gunsand for currentlymoremuch feasible forhardware thanfor software, and Commission is exploring such regulations. costsuncontrolled of acquiring devices. U.S.The Federal Trade currentlybe feasible minimum standards toimpose on companies global market diffuse overlonger term,possiblymaking thesupply chain the ofthe diverse economicapplications of drones,marketmaythe ecosystemsoftwareAI of development.With growing recognition such defenses are capital-intensivetous leadingimperfect, and systems,the ecosystemmaking morelevel diffuseatthis than requiredskill to carry outattacksraisethe throughmeans or these for hardening their productshardeningfortheir against cyber attacksmaketo or them autonomous weapons. additional sources of security vulnerabilitiesrobots as such a less useful focal point for governance. For example, itmight Hardware distributors Hardware manufacturers to launch attacksto today’swith robots consumer is currently widely high-risk activitiesself-driving carsthatsuch as conduct or robots,trusting ofmobile potentially autonomous creating reduce the risk of physical harminvolving AI. While the capacity restrictions on sales of potentially lethal drones might be thought point-of-salebased approaches. Notably, type ofthis control is mitigatedthroughbe risks bymight action distributors, orother moreresistant to tampering, so as to leastat somewhat raise the the consumerdronemarket,70%in position with about ofthe manufacturers,holding adominant likeDJI with companies be difficult to fullyprevent physical attacks leveraging AI. become morewidely deployedbecome more comprehensible andgovernablethe analogous than illegal drugs. it is at the production level. It is conceivable that at least some . This concentrationmakeshardware the ecosystem

. The consequences ofthese 7 6 5 4 3 2 1 ICRC, 2017;Scharre,2018 DoD, 2015;Roff,2016b DoD, 2012 Crootof andRenz,2017 Crootof, 2015 Mouawad, 2015;Vincent,2016 The Telegraph,2016 p.42 Security Domains There areregistration also requirements forformsrobots some of There aremanysource open frameworks for computer vision, Thereis activeNationsConventionUnited discussion atthe on cause physicalbeyondcause harm goes just drones. There are also distributed,particularly easy or use currently.to For example, large distributedpresent. atMany are expensive and/orrequire human CertainConventional Weapons ofthe value andcomplexity of Governments unintentionalintentionalofbetween or collision drones and dronesin certain areas,nearrisk airports,where such as the governments,which areintended preventto use of consumer the of Defenseof directivethatsetspolicy for out thedevelopment and such as drones in many countries,requirementssuch asdroneswellin as as for pointsuggesting of another control (discussedin stacks controlled(whichbybig companies are expensiveto train), such as flight stabilization. notpowerful AI But all tools are widely Softwaresupply chain systems for attacksin armedconflict forno fly zones. such In the physicalthe In sphere, there possible aremany defenses against and agreement unlikely in the near-term, though the development of alreadypassenger aircraftstruck a attacks via robots, though they are imperfect and unevenly Robot users Physical defenses Red Crosshas adopted position, a similar a stancepresumably that Manual notes that humans are bearersthe primary of responsibility trained AI classificationtrained systems that withincloudcomputing reside use of autonomyweaponsin no fly zones, imposed at a software level viamanufacturers and pilot training, thoughwenote that thespace robotsof that could be temptingmay formalicious actors tobuild from,potentially products oftenbuilt-in come withsome software forpurposes navigation,for etc.used carrying outattacks,be that can and motivatednon-state actorsfrom conducting attacks. norm developmentprocesses are critical,they areunlikelyto stop norms that could inform stronger governance isplausible regulatinglethalbanning autonomous orotherwise weapons passenger aircrafts loomslarge in the use ofin the force implies some minimumnecessary degree of humaninvolvement in the United States, for example, there is an official Department labor tolabor deploy,tohence areused defend“hard andonly targets” Appendix B . Key. states’ opposition to a strongmakesban such an

).

. While such armscontrolWhile . discussions and

. InternationalThe Committee of the . Additionally, the U.S. Law of War . Indeed, at least onedronehas Indeed, at . , suggesting a strongneed Interventions . Already

5 4 3 2 1 Zeitzoff, 2017 Scharre, 2015 Schmitt, 2017 Yin, 2015;Scharre,2015 Aker andKalkan,2017 p.43 Security Domains Political Security (such as highly populatedhighly areas).Physical(such as defensesinclude can An actorwantswho launch anaerialdroneto attack carrying a

of defensewith anacceptable cost-exchangeratio potentialGiventhe for automationto allow attacks atscale, a of of discourse through the large-scale production of persuasive but dynamics.Worryingly, featuresthe AI described earliersuch as of existingtrendsmore extreme,political kinds ofnew andenable characterisingpossibletracking and attackers. of attempted attacks,involvethosethatincluding attemptsto effective systems torestrict access topotentially explosive dangerous payload must source both the drone and the payload. defending against drones from the IslamicState inparticular of Defense hasrecently launched a majorprogram to defend defense through physical hardening or nets. The U.S. Department detection via radar, lidar, acoustic signature, or imagerecognition moreopposed tomuch widely the distributed“soft targets” security and intelligence services uncover and foil a large number software false content,hand ofstrengtheningregimes. and authoritarianthe Information technology is already affecting political institutions in and evenforeign policy (followinghigh-profileacids acid attacks). More generally, state against drones,testedhas lasersnets and with aneye and towards Next, we discuss the political risks associated with maliciousAI Developed countrieshave generallyreasonably long-lasting and Payload control Weseveral consider types of defenses,but as yet,the casesin as these defenses areincomplete andexpensive,likelysuggesting a the owners ofwhich can affordinvesttoprotection, such in as use. AI enables changes in the naturethe in ofbetweenAI enables communicationuse. changes military bases). heavily guardedfacilities that areknowntargets (e.g. airports or near-termbetweenthe ease gapof attack anddefense outside of forparticular challenge defendersfinding is effective methods myriad ways — e.g. the role of social mediain elections, protests, mediatedby automatedsystems producepresentthat and content. procure dangerous materials. Increases inAI capabilities will materials, andareintroducing systems restrictto accessto its scalability make itparticularly well suited to underminingpublic individuals, firms, and states, suchthat they are increasingly likely help their work e.g. in analysing signal intelligence, or in likesafety-critical facilitiesinfrastructure and (e.g. airports), Digital Security ; interception through various means

and Physical Security . The increasing use of AImay make , the problem isunsolved. ; and passive; and . Asof yet,. . 7 6 5 4 3 2 1 Waltzmann, 2017 Barberá etal.,2015 Morozov, 2012;RødandWeidmann2015 Mattiaci, 2017 Berger andMorgan,2015;Jones Aday etal.,2012 Greenemeier, 2016 Schofield, 2013 p.44 Security Domains There are multiplepoints of intersection between existing Context communication technologies arerespects,notablein some such expected tolike. Thus the dynamics we observe today arelikely seeking to challenge them. Modernmilitaries and intelligence security threats that statespoliticians face. and Examples abound since any individual orgroupsince any can communicate largeinfluence and sophisticated. fall short. Other structuralfall Other short. aspects modern technologiesof and the further polarizedfurther political discourse, allowing users, particularly in as telephones. as theyprevious did with generations of technologies such use today’sagencies information technologies for surveillance, poweradvancesbalancebetween of states, canthe change as kinds of the advances,newhas alsochanged technology and approaches for detecting and stopping thespread “fakeof news” informationthe easeas of copyingtransmission. and Waltzmann as incumbent governments can manipulate the information that ability to messageget their out to sympathetic supporters around and revolutionary movements such as ISIS or Libyan rebels, the toand communicatemorehowever, quickly; theyprovide also allow military intelligences to monitor sentiment and attitudes, France However,the effects newtechnologiespower of these on well as the relationship between incumbent leaders and protesters writes, “The ability to influence isnow effectively ‘democratized,’ While theyhave evolved fromprevious technologies,information to the use of social media during the Arab Spring technological advances. Securityneedshave driven technological todayto spread manipulativeinformation, false and andexisting to onlyacceleratebecome evenAI morethese asalgorithmsand the West, tointo self-select their own echo chambers, while others the public sees. Finally, some have argued that social mediahas that social mediamay empower incumbent authoritarian regimes the world technologies empower both incumbents and protesters: they politics and instability have had a symbiotic relationship with numbers of others online” running on these platformsprioritizethese contentrunning on usersthat are have questionedthis assumption protesters inplaces such as Ukraine and Egypt, and rebel groups relations are not straightforward. For example, social media including the advent of thesemaphore telegraphNapoleonic in information technologiespolitical sphere.and the Historically, is not necessarily favorable to democracy, however. It is very easy , to the advent Firstofitsuse GPSduringthe and GulfWar more quicklyandeasily.researchIn addition, suggests . This “democratization” of influence . Machinelearning algorithms . Technological

, , 7 6 5 4 3 2 1 the scopeofthisreport. we setsuchsystem-widerisksoutside security. However,asoutlinedabove, political life,notjuston and complexeffectsonallaspectsof these arelikelytohavesignificant including aspectsofequality.All distribution ofrisksandbenefits, relating totheeconomicandsocial effects onemployment,andissues barriers toaccessandbenefit, technology, legalandsocietal corporate/state controlofthe data aggregationandcentralization, issues atstakewithAIsuchas are muchlargersystemicandpolitical security (see:Introduction).There to undermineindividualorcollective the directmalicioususeofAIsystems that weonlyconsiderinthisreport It shouldbeemphasisedhereagain Guilbeault andWoolley, 2016 Abokhodair etal.,2015 Woolley andHoward,2017 Weedon etal.,2017 Botsman, 2017 King, Pan,andRoberts,2017 p.45 Security Domains AI will cause changes in the political security landscape,AI will politicalcause security in the changesas the Already,there areindicationshow of actors areusing digital How AI Changes the Political SecurityLandscape vigorous prevention and mitigation measures. offline data “social creditto distill a score” its citizens,for the and communicationtechnologies to political alter dynamics varies Others, such as Morozov (2012) and King,Pan, and Roberts (2017) makesclickbait them vulnerablemanipulation.” tomedia such demonstrated the ability to influence mainstream media coverage moregenerallywidespread use ofChina exemplifies censorshipin complexwebindustry, of government, andother actors,whereas or use veryor simpleforms of automation.However, bot-based these discreteinstances misuse ofonly scratch thesurface of the shape online and in-person political discussions, making use of making in-personpolitical discussions, shape onlineand swaypublic opinion social mediaplatforms with low barriers to entry makes it easier some authoritarianstates strategies (even when usingrelatively unsophisticated automation) for AI systems to masquerade as people with political views. This across types of political regimes. Inliberal democracies, it can be argueprovidesmedia that social more tools for authorities to and metrics, sensationalism, novelty over newsworthiness, and and political beliefs are leveraged by national intelligence agencies and have automation topolitical shape discourse. The widespreaduse of arms race between production and detection of misleading Finally,notewe thatthe extent informationnatureuse of and ofthe War the Chinesethe governmentis exploringways leverageto onlineand the more explicit leveraging of technology for political purposes in thought of as morethought of asof anemergentfromphenomenon, arising a that understandingthat landscape the of threats will encouragemore term implications of such malicious uses of AI will be, and these notemedia’s that the “dependence media, analytics on social media industry also enable these trends. Marwick and Lewis (2017) bots are controlled by humans who manage a large pack of bots political messages and cause dissent. At the moment, many such led tohas thewidespread“bots”use media of social to spread manipulate the news environment and control the message. political implications of AImore broadly increasingly sophisticated technologies to do so in statesin likethereis anexplicitChina, anddeliberate effortto information evolvesstates and pursueinnovative ways of leveraging AI to maintain their rule. It isnot clear what the long- and the 2016 US election bots2016US election appearedthe and to activelyto try . For instance, during both the Syrian Civil . . . However, we hope . For instance, , 8 7 6 5 4 3 Browne, 2017 2017 Chung, Jamaludin,andZisserman, Shao etal.,2017 Everett etal.,2016 Seymour andTully,2016 Adams, 2017;Serbanetal.,2017 p.46 Security Domains AI systemsproductionmay the simplify high-quality fakeof video footagevideo of,for example,(fake)sayingpoliticians appalling Greaterscalesophistication and of autonomoussoftware actors objective truth becomes devalued and “truth” is whatever course, propaganda does not require AI systems to be effective). creates new opportunities to enhance “fake news” (although, of of fakenews morebecome awarepeople especially as oftheir existence, but opposed by traditional media channels. Authoritarian regimes environments, thiswill create a strategic advantage forpolitical of video andaudioevidence. makeTheyit easierfor might also quality forgeries may challenge the “seeing isbelieving” aspect tousually enough settle adebateor audioevidenceis what about spear phishing has demonstratedphishing spearthat automatically generated In addition to enabling individuals andgroupstoIn addition misleadthe enabling to approachesbe convincingthe canhumans, especiallywhen to authorities claim it to be. routine sports financial and reporting oftenas are today. As and authenticationhas anedgeovertechnology still forgery Moreover, automated natural language and multimediaproduction Even ifbot users only succeed in decreasing trust in online will allowAI systems toproducemessages tobe targeted at those publication offakewriting and news stories beautomated, could warSyrian CivilWar in the crimes textpertainsto certain topics such asentertainment textbe effective can humans atfooling techniques thereis evidence they contribute significantly propagationto the to what extent political bots succeed in shaping public opinion, toinformation, augmenting disseminationmisleading ofthe the purported evidence might have beenproduced. In addition technology.being committedA video of canserve acrime as things untrustworthy source. In the future, however, AI-enabledhigh- public about the degreepublic about forperspectives, ofsupport certain AI mostsusceptibleto them. be anextensionThiswill of existing informationmedia and ecosystem. ofthe portion productionforgeries anddisseminationhigh-quality ofbecomes topeople deny allegationsthem, given against which the easewith highly compellingevidence evenprovidedwhen by anotherwise toused documentin agiven been happened dispute,has and in the political spherein the is technicallypossible with existingAI in particular may benefit from aninformation landscape where ideologies and groups thatlow-trust thrivein societies or feel increasingly low-cost, synthetic multimediamay constitute a large . Currently,. the existence recordedhigh-quality of video . As previously discussed, progress in automated . . At present, recording , and indeed verysimple and , . It isunclear 9 8 7 6 5 4 3 2 1 Griffith, 2017;Manjoo, 2017 Reuters, 2017;TomStocky, 2016; Verma, 2014 Goel, 2014;Krameretal., BBC, 2017 Roff, 2015c Ng, 2015 Roff, 2015c;Horowitz,2016 De Choudhuryetal.,2013 De Choudhuryetal.,2013 al., 2013 Quercia etal.,2011;Kosinski p.47 Security Domains The moreThe entrenchedposition ofregimes authoritarian offers The information ecosystem itself enables political manipulation elections in wayselections undermine the thatdemocratic process. gerrymandering” described inprevious sections may also have worrying implications emotional contentusers’ of posts,modestly albeit content. For example, Alphabet Executive Chairman EricSchmidt example,using systems identify salient that video clipsbring and groups democracies, thestatehavemay limitedlegal authority to shape operating underrules and directions issued by the state. In enforce agreater degreepopulations.unwilling of complianceon 2016presidentialrecent is a example election howsuccessful of still manipulate public opinion by “de-ranking” or promoting certain still exist; they simply reside in the hands of corporations. Even surveillancemore at a efficient scale systems might allow groups to target precisely the right message forpolitical security. hacking Theof theClinton campaignin the able toable gather datamost citizens, thedata is on using efficiently as easily availablein democracies mechanisms foradditional control throughAI that areunlikely tobe acts, for example.Even without advanced techniques,“digital at precisely the right time in order to maximize persuasive advertising practices. Public social mediaprofiles are already and influence informationand content butthesame technical tools and controlbyfiltering content available users. Inauthoritarian to a point of overlap between political and physical security, since analysisidentify currenttopotential leaders or ofsubversive Finally, thethreats to digitalphysical and securitythat wehave without resortingtowithout outrightplatforms censorship,media could tools could be used to help filter out maliciouscontentfilterhelp to out used or fakebe tools could newsfeedsthe usersin order of overtomillion alterthe half a tothem the attention human agents).Furthermore, of be this can the ability to prioritise attention (for example, by usingnetwork too costly for many authoritarian regimes. AI systems both improve public opinion news, they also could beusedby mediaplatforms to manipulate by Russia Today and Sputnik recently stated that Googlewould de-rank contentproduced potential.Such a technologyis sinister when applied to voting predictpsychological like conditions depression reasonablypredictive personality of details regimes, this could be done by the state or by private parties roboticsystems resourcedhighly couldalsoallow groupsto intention,pernicious when applied torecruitmentbut for terrorist ) and also reduce the cost of monitoring individuals (forreduce individuals the costmonitoring) andalso of . orotherforms ofmightshape advertising . In2014, Facebook manipulated . AI systems. enable fine-grained . While existing. systems are , and may beusable to . Sophisticated AI . Whilesuch 9 8 7 6 5 4 3 2 1 campaigns. being targetedbydarkadvertising the extentwithwhichtheyare service thatinformscitizenson “Who TargetsMe?”isasoftware fact checkerscombatfakenews. development ofAItoolstohelphuman competition aimedatfosteringthe The FakeNewsChallengeisa Ixy, 2017 Sunstein, 2017 Clarke etal,2013 al., 2017;Zubiagaet2017 Shu etal.,2017;Pérez-Rosas Rahman etal.,2017 Varol etal,2017 al., 2017 Kashyap etal.,2017;D’Avino p.48 Security Domains Technical tools The use of citizens’The databyintelligence agenciestakes various Automatedfake newslikewise detectionis subjectthe of ongoing measureshaveAnalogous been developed for authentication of Points ofControlExisting Countermeasures and daunting challenge. governments from making full use of AI seems to be an even more cyberattackspolitical can disruption. cause The disruptive ones rangingones from tools for trackingpolitical campaigningin social educationteaching and newerwell ofskills, as criticalthinking as General interventions to improve discourse organizations, inpart to prevent the sorts of risks discussed here. images authenticity and videos,certified of provee.g. theability to detecting forgeries yet, however, the detection of misleading news and images is Severalmeasures are alreadyin development ordeployedthis in Snowden revelations for encouraging constructive dialogue been activelyhas forms debated,wakethe and in especially ofthe area, though nonehas yet definitively addressed theproblems. apparently authentic multimedia and text israpid. an unsolved problem, and the pace of innovation in generating are valuable levers for ensuring that mediaisin fact produced by Pervasive use of security measures which be canexpected innovation to spurfurther in this area. As Wehighlightfew arelevant effortshere, andemphasizethat these proposalsthese are orientedtowards protectionthe healthy of to justify a move toward more authoritarian policies. terror,is even clearer.threatsSuch to physical digital and security transmissions, and is actively used bymanytransmissions, companiesis activelyused andother and the relevant person or organization and isuntampered in transit. that a video wasbroadcast liverather thansynthesized offline useful measureuseful for ensuring thesecurity informationof public discoursein democracies.Preventing more authoritarian institutions orallowundermine existingthem political might either potentialphysical of attacks,such asassassinations andacts of media (such as “Who Targets Me?” private spheres, includinglongstanding ones such as better proposalsincreasepublic and to the qualityofthe discoursein research images (rather than videos) by Naveh and Tromer (2016). as well as a competition, the Fakewell as asacompetition,the News Challenge . Technicalmeasures areindevelopment for and social mediabots . . Encryption is agenerallyEncryption . ) to policy proposals . . There are variousare . There . Likewise, the use of to apps ,

4 Lapowsky, 2017 p.49 Security Domains A developmentthatprocess occurredthe during ofwriting victims are stillreason found.Part of the less thatis spamof a dissemination of the tool and its products,disseminationitstool ofthe less and least amongst at discussionpropagation and of thetechnique. Whilethese efforts decline of “seeing isbelieving” discussed above. After substantial en masse inReddit fora clearly labeled as being fictitious, the developingAI systems for detecting suchthreats, and through disinformation, censorship,persuasionAdditionally, and campaigns. of control for stemming thetide increasinglyof automated sophisticated actors. strongbarriers(e.g. to entry use of one’sthe identity) offline and spam filters. More generally, technology companies, social misleadingsalestactics, schemes,scam emails, etc. and yet applications) adult videos. Whilesuch videos first began appearing Media platforms Because these media platformsmedia areBecausethesefor-profit corporations, technology platforms in governing information access, and itis the ability tothe ability control access,they pursue can otherstrategies for these organizations have unique datasets that will beuseful for than others, yetnot has entirely this stopped thespread of fake the application offace-swapping (amongto algorithms other this report isillustrative. Late 2017 saw the rise of “deepfakes,” havebeen fullysuccessful,not theyillustrate the rolecritical of media coverage of deepfakes,Reddit andotherwebsites, online realismsuch deepfakes ofsome potentialis anearlysign ofthe preventing malicioususes of these platforms such as imposing websites,media media organizations and arepoints critical keyplatformsserverssuch asemail have deployedsophisticated thatthe ownersis be problem of todayit otherwise could than news.Likewise, people aremost aware ofthe existence Ponzi of powerfulinterest toolspublic aligns with importantmechanisms forbe use ensuring thatof their these public discourse,transparency, potentiallyregulation and will including adultcontentwebsites, tobegan crack downthe on impartiality, and some online sources have better reputations likely that the deepfakes crackdown leastat somewhat slowed the limiting the ratethe limiting atwhichaccounts can disseminateinformation. . There have always beennews sources of varying . 04 Interventions

We identify a wide range of potential responses to the challenges raised above, as well as a large number of areas for further investigation. This section first makes several initial high-level recommendations for AI and ML researchers, policymakers, and others. We then suggest specific priority areas for further research, where investigation and analysis could develop and refine potential interventions to reduce risks posed by malicious use.

Due to the exploratory nature of this report, our primary aim is to draw attention to areas and potential interventions that we believe should be the subject of further investigation, rather than to make highly specific technical or policy proposals that may not be viable. The structure of this section, and the inclusion of Appendix B with additional exploratory material, is informed by this perspective. p.50 2 1 substantial andvaluable. from technicalexpertshave been further exampleswherecontributions conference seriesandorganizationare conference inAsilomar,andtheAINow on AI,the2017“BeneficialAI” White House’s2016seriesofworkshops The workofthePartnershiponAI, and Flaxman,2016). machine learningalgorithms(Goodman and applyinthecontextofcurrent a policythatishardtointerpret is acommonly-discussedexampleof General DataProtectionRegulation difficulties, theEuropeanUnion’s domain thathassurfacedseveral an exampleofpolicymakinginthis on AIandsecurity(CNAS,2017).As and Taylor,2017),specifically both generallyaboutAI(Buchanan are increasinglybecomingavailable, policymakers interestedinthisdomain Introductory resourcesfor p.51 Interventions Recommendations researchers to engage withthese topics, out of concern that other policy responses; second, reluctance on the part of technical leading to poorly-designed or ill-informed regulatory, legislative, or technical understanding on the part of policymakers, potentially hamperedbybe two self-reinforcing factors: first, lack a of deep development of viable, appropriate responses to these issues may considerations. Concernswereraised at theworkshop that the technicalmilitary considerations, social, economicand such as the issues raised in this report combine technical and non- Our first pair of recommendations arisefrom the fact that areasfor technical work well as asassociatedresearch questions. subsection, we willturn our attention tomore concretepriority researchers,policymakers, and other stakeholders.In thefollowing which are focused on strengtheningbetween the dialog technical In thissubsection wepresent fourhigh-level recommendations, already doingoutstandinglines.work these along contributed to this report and other related initiatives — who are including thetechnical expertsin theworkshop whotookpart and work on. (We recognize and appreciate the many AIresearchers — applications of differentresearch projects before whatto deciding policymakersprovideto expertise,potentialthe andconsidering prevent harmful uses. Example steps could include engaging with they can to helppromote beneficial uses of the technology and researchersresponsibilityit their consider to take whatever steps that AIis a dual-use technology, we believe itisimportant that relevant actorsharmful applicationswhen areforeseeable research priorities and norms, and proactively reaching out to seriously,misuse-related allowing considerationsinfluenceto intelligence shouldtakenature the dual-use of theirwork ResearchersRecommendation#2: andengineersin artificial technologieshand at policy responses will beinformed by the technical realities of the Close collaborationtechnicalwith experts alsoensuresthat unless those measures are likely to bring commensurate benefits. measures that will interfere with or impederesearch progress, takingresponsibility seriouslytheir to avoidimplementing potential malicioususes of AI with technical researchers to investigate, prevent, and mitigate Recommendation#1:Policymakers should collaborate closely firstOur two preemptingrecommendations aim at thisdynamic. field andperhaps lead to reduced funding orpremature regulation. use wouldassociationmalicious tarnishreputation with the of the . . This must includepolicymakers ) . Given 2 1 system. to originatefromthecompromised be attributed,astheywouldthenseem the possibilitythatattackscould more-capable attacks,whilereducing computing poweranddataforyet- by attackersandrepurposedtoprovide insecure systemsmaybecompromised society morebroadlyas,forexample, to thenewthreats,butalsofor communities lessabletoadapt This isimportantnotjustforthe for implementingsecuritypolicies. literacy, whichmayposechallenges for exampleontheirtechnological make suchadaptations,depending will havevaryingabilitiesto acknowledge thatdifferentcommunities best practices.Itisimportantto awareness ofthreatsandadopting if onlyintermsofmaintaining will berequiredofeverydaycitizens, We expectadaptivedefensiveactions organisation’s systemsandpractices. improve thesecurityof better understandand,ultimately, attack mightlooklikeinorderto exercises explorewhatanactual responding totheseattacks.These damage), withanoptional“blueteam” limitations topreventlasting of theorganization(withsome against thesystemsandpractices planning andcarryingoutattacks the hostorganization,deliberately security expertsand/ormembersof involves a“redteam”,composedof In computersecurity,redteaming p.52 Interventions Priority Areas for Further Research products, ethicists,public the general cybersecurity researchers, businesses incorporating AIinto their society,civil nationalsecurity experts, as-yetAI and unengaged these challenges stakeholders anddomainexpertsinvolved in discussions of ActivelyRecommendation#4: seekto expandrange the of Research Area #1, below, for further details. into AI contexts is extensive use of “red teaming.” workshopparticipants considered clearly valuableintroduceto applicabletoAI the case of such ascomputeruse concerns, security, importedwhere and research areasmorematuremethodsfor with addressing dual- Recommendation #3:Best practices should beidentified in policyprofessionals) should work towards. believe the broader AI both community (including technical and Wemakerecommendationstwolaying also thatwe outaims Appendix B commentary on many of the topics mentioned, may be found in investigation,questionsfor with additionalcontext along and investigatedbe further. Wehere aim forbrevity; more specific Thislays section out specific topic areas that we recommend tobenefit society. relevantother stakeholders to ensureused AI technologyis that researchers,AI dialogueamong disciplinary policymakers, and critical. The aboverecommendationsfourfoster help can across- public dialogueonappropriateis whytechnologyis AI a uses of harm. public goodor forthem Thisuse toolsAI andcould ofthese Governments and powerful private actors will have access to many could beused to bury fake news or manipulate public opinion. terrorists oroppress ordinaryInformation citizens. contentfilters malicious use. For example, surveillance tools can beused to catch be oneof degreecould orensuringappropriatesafeguards against cases, the difference between legitimate and illegitimate uses of AI of AI outlined in this report have related legitimate uses. In some Because of the dual-use nature of AI,many of the malicioususes relevantstakeholders relevant areincluded and experts consulted. . . This could includereaching out to sectors like

. An examplepractice. bestthat of a , andothers,to ensurethat SeePriority

2 1 2017. and Dill,2017;Carlinietal., also Katzetal.,2017;Selsam,Liang, traditional methodschallenging.See makes assuranceorverificationusing throughout theirlifespans,which in systemsthatcontinuelearning developing techniquestoassuresafety (Neema, 2017)isoneattemptat DARPA’s AssuredAutonomyprogram (Greenberg, 2016). fully controlthevehicle’ssteering cause unintendedaccelerationand later developingtheabilityto standstill onabusyhighway,then hackers firstbringingaJeepto For example,seethecaseof p.53 Interventions • • • •

suitabletargets previous of waves ofsophisticated attacks. larger scale,reachmay which victims that wouldbe not otherwise tosophisticatedhighly allow attacksmuch be carriedto outat a satellite imagery. Third, increasing use of AIin cyberattacks islikely networks,the compromisefor or used analysing of analgorithm example, theft of the datasetsused for facialrecognition on social algorithms and/orbyused thesystem; trainedmodels consider, for AI-basedsystems on can alsogivethe attacker accessto the commensurately. To summarize the considerationsin the potentialimpactsthe of cybersecurityincidents are growing As AI-based systems becomemore widespread and capable, Learning from and with theCybersecurity Community Priority Research Area #1: on and questions about these sub-areas), include: on andquestionsthesesub-areas), about implemented asappropriate(see be thesubjectresearch should of further and analysis,be then Some examples of cybersecurity-relatedbelievethatwesub-areas ported over wherever applicable toAI systems. from AI systems, and best practices from cybersecurity must be major and ongoing priorityin efforts to prevent and mitigate harms To respond to these increased dangers, cybersecurity must be a with a typicalwith a carfrom20 years ago successfula hacker couldexercise overmodern car, a compared of physical systems; consider, for example, how muchmore control First, increased automation brings with itincreased digital control Security efforts to predict how AI advances will enable more effective Forecastingsecurity-relevant capabilities software systems? exploits), as is already possible for security exploits inmodern vulnerabilities,potential adversarialinputs, andothertypes of of vulnerabilities discovered inAI systems security (including proceduresbe established to enableconfidential reporting Responsible disclosureAI vulnerabilities of developedto achievebysimilar goals differentmeans? to prove key properties of AI systems? Can other approaches be forwhattypes of architectures canformal used verification be Formal verification AI developers,priority of in criticalsystems. especially potential security vulnerabilities and safetyissuesbe a should Red teaming section, AI is importantis toAI section,cybersecurity for threereasons. . Extensive use of redExtensiveuse ofteaming . to discoverfix and . Towhat extent,what in circumstances, and Appendix B . Second, successfulSecond, . attacks . Could “white-hat” . Could for morefor commentary . ShouldAI-specific

Digital 1 Eckersley andNasseretal.,2017 p.54 Interventions • •

Exploring Different Openness Models Priority Research Area #2: that policies leading to decreased openness, while potentially reasons to favorresearch opennessin communities. Webelieve situations, we stress that there are clear and well-recognized proposalsbelowthe considerdecreasingWhile in certain openness leastsome cases. in at caution interestbe argumentspublic may also based on for additional “scooped”). Inlight of risks laid out elsewhere in this report, there intellectual property (e.g. in order to avoid a future result being results are withheld today inAI,itisusually for reasons related to an toopportunity fixthe vulnerability. To the extent that research systems are not publicly disclosed until the developers have had as computersecurity, where exploitsthat couldaffectimportant AI forreasons? security Thereprecedentis forinfieldssuch this abstain frommerely or publishing delay of some findings related to raises an important research question: might itbe appropriate to increasespowerit the oftools availablemalicious actors.to This newto capabilitiesbysharing all openly andalgorithms default: Scenarios However,potential the misuses AI technology of surveyedin the array of applications. and allowing theoretical progress to beincorporated into a broad researchersbuild oneachto others’ promotingwork, collaboration, code. levelThis benefits in terms ofhas clearof enabling openness fromrough architectural outlinesto algorithmicdetailsto source novel research ispublished online inpapers that share anything community stronglypoint towards large openness.A fraction of Today, the prevailing normsin the machinelearning research with properties like this? practicallythe designand hardwarefeasibleis andadoption of access,facilitate Howtechnically activity audits,similar? and AI-specific hardware, for example to prevent copying, restrict Secure hardware security professionals? problems inAI systems, analogously to tools usedby computer distributedmake toitstandardhelp to test for common security Security tools by defenders? proliferation cyberattacks, and more rigorous tracking of AIprogress and and Security Domains in general, allowmore for effectivepreparations . What tools (if any) should be developed any)should (if Whattools . and . Couldsecurityfeaturesincorporatedbe into sectionssuggest adownside 2 1 necessary. of discussionweseeashealthyand better) providesamodelofthekind order tounderstandcertainthreats organisms aremademoredangerousin gain-of-function research(inwhich appropriate levelofdisclosureon the biosecuritycommunityabout evaluate theseclaims.Thedebatein that arangeofstakeholderscan state theirreasonspubliclyso disclosure), thosedoingsoshould from otherdomains(e.g.responsible AI researchandpublicationprocesses new approachesareincorporatedinto be donetransparently,andthatwhen openness arerethought,thisshould to theextentthatpracticesaround competitiveness. Webelievethat, the realmotivationisaboutcorporate than isrequired,forinstance,when reduce opennesstoagreaterextent should notbeusedasanexcuseto Accordingly, concernsaboutmisuse see e.g.NDSS,2018 p.55 Interventions • • • •

commentaryinclude: onandquestionsfor about thesesub-areas) investigationto further andanalysis(see Some potential mechanisms and models that could be subject mightthis. enable mechanisms might outweighfavorsharing considerationsin open what and foster discussion ofwhetherwhen considerations and against benefits. appropriatein instances,certain be sensitive should to these before AI systems are more widely usedin critical systems computer security. Orwouldmeasurespremature such be today, researchfornorm the in other areas,biotechnology such as and to determinelevelwhat ofis appropriate openness machine learning, be subject to some kind of risk assessment as work specifically related to digital security or adversarial concern Pre-publicationrisk assessmenttechnicalin areas ofspecial technologies? arisen whentacklingissuesraisedby similar other dual-use methodologies, considerations,have andcautionsthat use technologies norms Other and institutions that have been applied to dual- purposes, but would beharmful ifmore widely distributed. betweentrusted organizationsfor vulnerability discovery cybersecurity research that leverage AI might be shared For example,norms? ethical certainforms of offensive effectiveinformation security andadherence to appropriate people andorganizationsmeet certainthat criteria, such as results areselectively sharedpredetermined amonga set of arrangements bemadeunder which some types of research regimesSharing that favor safety and security expertiselarger allowset aof actorstools?AI use to processingpower, datastorage andavailability, andembedded remainmodel viable overmightsuch a How time asadvancesin given capability while reducing the possibility of malicioususe? security-focusedthatmodel allowssharing widespread use of a technical detailssystem ofthe provide— template a for a availableby acentralproviderhaving without access to the services like sentiment analysis or imagerecognition made access” commercial structures —in which customers use Central access licensingmodels premature,under what conditionswould theybe appropriate? most security-relevant? If such measures are considered be and we have better knowledge of which technical research is Rather than proposeRather than a specificsolution, is our aim to . Should some types of AIresearch results, such . What can belearned from other models, . Could emerging“central. Appendix B . Could formore ? This is ? This 2 1 Institute, 2017). Asilomar AIPrinciples(FutureofLife 2017) andthedevelopmentof Systems (IEEEStandardsAssociation, Artificial IntelligenceandAutonomous for EthicalConsiderationsin for AIaretheIEEEGlobalInitiative Two examplesofproposedstandards and questionsaboutthesesub-areas. See AppendixBformorecommentaryon p.56 Interventions

• • • •

fostering acultureresponsibilityinclude of initial areasSome to explorefor concreteinitiatives aimedat AI capabilities.risksuses of malicious of development, individuals, and institutions should bemindful of the risks inparticular. Throughout training, recruitment, research, and technical withgreaterfields, and attentiveness use malicious to greaterleveraging insights offrom the experiences of other same. developed,be continuedfurther and Thisshould with responsibility quiteseriously, andencourage othersto the do in the enabled world.Manycommunity already take theirsocial position tolandscapeunique AI- shapethesecurity ofa the researchersAI the organizations and that employthem arein PromotingCulture a Responsibility of Priority Research Area #3: (proposed above) bemore productive? have obvious shortcomings. Might a narrative like “dual-use” the countervailing “automation boon” trope, both of which of existingnarratives“robot the include apocalypse” trope and with a level-headed recognition of the risks itposes? Examples balance optimismthe potentialvast about technology ofthis compelling narratives of AIresearch and its impacts that can Nuanced narratives might they beused for preventing AI-related misuserisks? at all) protectionswhistleblowing(if how in other domains,and Whistleblowing measures distinguish between benign and malicioususes of AI? physical,resolved needtobe and securitythat in order to theareasthe domain-specific in ethical questions of digital, implementedtheybe andenforced?should whom What are statements and standards play inAIresearch? Ethical statements and standards of AIresearchers? How could this training bebest incorporated into the education responsibletechnologyuse oftheir aremost effective? scientists andengineersthe ethical about socially and Education . What formalinformal and methods for educating . More. generally, arethere succinct and . What is thetrackrecord. What of . What role. What should ethical : How and by

2 1 practices? (Miller, 2006)orothersecuritybest to principleofleastauthority new softwaretoadheremoreclosely to refactorexistingcodebasesor For example,couldAIsystemsbeused and questionsaboutthesesub-areas. See AppendixBformorecommentaryon p.57 Interventions • • • •

Some initial suggestedSome areasinvestigation for further include prevent and mitigate potential misuse of AI technologies? approaches —both institutional and technological — could help to proposals mentioned in the previous sections, what other potential supported by well-designed policy responses. In addition to the defenses. be accompanied and Thesemusttechnological solutions progress inAI also makes possible new types of responses and toIn addition creatingnewsecurity challengesthreats and Developing Technological and Policy Solutions Priority Research Area #4: relevantresearch communities. private, would also help to seed interest and recruit attention in track record. Additional monetary resources, both public and individuals andorganizationsrelevantwith proven expertise anda researchinitial Further stepsagenda. require will commitment from awarenessimportance,layissuestotheir and outan and of the investigations. Aninitial step, pursued by this report, is to raise and organizationsrelevant withthe expertise topursue these For all of the above, it will benecessary to incentivize individuals state, orothers? must beplayed by institutions, whether by corporations, the protecting privacy from bad actors in a world of AI? What role Privacy protection case of adversarial examples anddatapoisoning attacks)? (e.g. hacking legal definitionsto account the adjustingof for interventions by policymakers would beproductive in this space legislativeOther and regulatory responses code, anddata? tomonitorinputs toAI technologies hardware, such as talent, and for which resources, might itbe feasible and appropriate Monitoring of AI-relevant resources and shared? institutions or mechanisms can these technologies bepromoted the directionin offense-defensebalance of defense? defensivemeasures security be distributed widelytonudge the Coordinated use of AI for public-good security . What role can technical measures play in . Under whatUnder circumstances,. . What other potentialWhat. other . Can AI-based . Can Via what Via

:

05

p.58 Analysis Strategic attack or defense. sufficiently to fully obviate in either input need human the for used and defended against, but before AIhas yet progressed fromnow) aftermalicious applicationsAI are ofwidely which and defense. Bymedium-term, we mean the time period(5+ years severalhypothesesmedium-termAI attack equilibriumofthe about considerationsmorefor givinga confident answer, make and key stakeholders.Nonetheless, we aim to elucidate some crucial malicious actors, and thesteps thatbe taken should and will by progress of various technologies, thestrategies adoptedby to make, as significant uncertainties remain regarding the of Anysecurity? confident long-term prediction isimpossible surveyed implemented)above combine to shapethefuture(if characteristics of AI and the various intervention measures When considered together,how willthesecurity-relevant

p.59 Strategic Analysis Security Factors Affecting the Equilibrium of AI and research anddevelopment achievements.thesetrendsIf continue Currenttrends emphasizewidespread openaccessto cutting-edge AttackerAccess to Capabilities keyon factors to watchinfluence inthe and years to come. Nevertheless, wehopethat the analysisbelowlight shedssome moreimpactful thanthe capabilities report.consideredin this technological developments unrelated to AI,may ultimately be what can currentlyforeseen.be New developments, including since both technological and policy factors will progress beyond resulting from foreseeableAI developments be short-lived,might Evenstablepredictableseemingly a and medium-term equilibrium attacks alsoallowmorefor instances scalable defenses.Specific of Thesame characteristics low-costlarge-scalethat and AI enable of Existence of AI-EnabledDefenses threat awarenesspolicymakers. among disclosing developmentsincreasingmisused, and be that could interventions aimed at making systems more secure, responsibly actors usingAI can likely bereduced through a combination of espionage to obtain such However,code. risk fromless the capable motivatedbecausesufficiently and well resourced actors use can fully,succeed less-than-perfectbothto due complianceand uses solely through limitingAI code proliferation are unlikely to ofsignificant a Effortsattack prevent scandal. or to malicious reportsto(hopefully)the aftermath these,in such as orotherwise help, andarelikelyall to emerge atresponse variousstages, in and technologicalmeasures international within policing will Preemptivenovel design effortsuse of the and organizational restrictmonitor accessintervention. or through particular any remainsuncertaintythe effectiveness about of attemptingto technologies will beincreasingly imposed. However, significant and that limitations on access to or malicioususe of powerful AI increasinglybecome apparentto developersregulators, and However, we expectnature the dual-use of thetechnology will technologiesAI previously.ease of discussed diffusing follows directlynature,from the dual-use efficiency, scalability, and harm with digitalrobotic and systems significantly increase. This 5 years,next forthe we expectthe abilityof attackersto cause

p.60 Strategic Analysis use seem likelyinterestseem givenuse the of actorsfrom corporationsto developmentsunderlying technologiesin the and theirwidespread Wehave hardlythe endofsuch advancements,seen further and recognitionbysurveillance network camerassocial analysis. and rangelawfor ofwide enforcement a purposes,facialsuch as and counterterrorism.is alreadytowider adoptionbeginning see AI investigationsin anoverallin criminal AI use of assessmentthe is One general categoryAI-enabled defenses of worth considering fuller assessment.ofsuch defensesbe consideredin a should defenses. pace oftechnicalinnovation boththe Thus, the cost and prohibitivelybe toturn out expensive,might otherforeseeable as drones might beinvented and widely deployed, but they might also non-violently“catch”quickly and tobring the ground and other physicalof security,use of the dronesis topurpose whose sole be developedthe coming will years.in For example, the contextin spamfiltersas malware and detection, and we many expect others AI-enabled defenseshavein earliersections,such been discussed technology,requiremay which coordinatedregulatory efforts, or victims, defensive measures need to bebaked into widespread under-resourced states, SMEs, and individuals.For these potential potentialof victims:technologically conservative corporations, however,This, leaves out the strategic situation formajority the world.such a ethical culture in the AI community more generally will be vital in novel vulnerabilities,pre-publicationrisk assessment, strong anda attacks anddefenses are developed.Responsible disclosure of levelrelativethe though will dependon ratesrisk of atwhich likely,least such as attackthe harm, facilities,is also WMD of on major corporations. This means that the most massive category havemostlosefromto the attacks such asgovernments and These measures could then beused by the organizations that as soon as theybecome technically feasible and cost-effective. researchsuch as labstech and startups, andarelikelyhappen to Appendix B Some defensivemeasures in discussed DistributionGenerality and Defenses of preventhelp authoritarianAI. abuses of key variable in the medium-term. However, such advances will not states to detect and stop criminal acts, inpart by leveraging AI,is a physical defenses against drones. Thus,the growing abilityof may turn out tobe cheaper thanfor example widely deploying attackstheir earlystagein throughrapid detection response and governments inpreventing criminal acts. Additionally, interceding can be takenbe canby internallysingle, coordinated actors, Interventions and 3 2 1 Moore andAnderson,2012 CSE, 2017 GCHQ, 2016 p.61 Strategic Analysis Overall Assessment apply: a general solution to authenticable multimedia productionapply: ageneraltosolution multimedia authenticable politicalsecurity,For similar considerationsregarding generality releasethe ofCyberChef measures,in cyber-security,is generallythe case as see though the developmenthinder may competitionrelease and of defensive or governments.the caseIn of governments, international financial backing, from corporations, investors, philanthropists, likely that those developing such defensivemeasures need will an associated higher time lag and skill investment, it ismore measures:if each attackrequires tailored a defense,has and likelybe affectedtoplatformsis bythe generality of defensive This dynamicof defensethroughreliance fortified software on freely or cheaply (e.g. Mozilla’s Firefox web browser). organizationswho develop anddistributesuch defensive measures lock-in andconcentration of datapower, and fromnon-profit or tech giants(asin the case of spam filters), increase which will offered at low prices. The latter ismost likely to come either from There ismuchlow hanging fruit to bepicked in securing AI systems On the optimistic side, severalpositive trendslook for defense. technologies thanto avoid them. cycle, itbecomes easier to make use of such general purpose computernetworks: in thetechnology pointat some adoption terroristsregimes andauthoritarianuse electricity, software, and to AI; we anticipate increased malicioususe of AIjust as criminals, across society more generally. This isnot a trend that isparticular AI maliciously will increase alongside the increase in the use of AI Acrossplausible outcomes,we all anticipatethat attemptsuse to considering the outcomes that are less likely, but still possible. rangeplausible outcomesis extremelyThe of diverse, evenwithout addressthese externalities regulation may beneeded to adjust these incentives or otherwise improveto defenses directly. Thus, other approachesincluding of DDOS attacks usingbotnets, are not typically in a position as the individuals whose personal data isreleased or victims botnets.However, individuals affected the by these failures, such barfor raise databreachesthe could the creation or IoT device of defensivemeasures.For example, better cybersecurity defenses incentivesMisaligned leadto failure can a also to employ available media types.those photographs,forsolutions videos,narrower oraudio, or subsets of and forgery detectionmoreuseful wouldbe thantailored individual and Assemblyline and . ascounterexamples. p.62 Strategic Analysis structures, defenses arelikelyprevail.to pre-existingor coordination,with existingincentive andalign users.security for their Where solutions are visible,require limited tech giantsto level collaborateminimal least a onensuringat of protection and standardization.Finally, there areincentives for reducethreat levels increase and stability, e.g.through consumer political incentives for developing processes and regulations that and behaviors. There are, at least in some parts of the world, examples and providing provable guarantees for system properties problems,such asdeveloping methodsto address adversarial substantial incentivesacademic to tacklehardest the research internallybefore they are discoveredby adversaries. There are the discoveryin expertise of vulnerabilitiesbysoftware companies in criticalsystems, used being andgreater effortsleverageto AI machine learning in cases where the affected ML technology is attacks. Examples includeresponsible vulnerability disclosure for themselves,people and systemsand securing fromAI-enabled and governments. media giants,interact and less frequentlybusinesses with small regularly interact with and use the platforms provided by tech and represent acontinuation ofpeople very existing which trendsin sectors)pressureunder likelybe will to follow suit. This would (automotive,medical, defense, increasinglymany other and giantsthat offer digitally-enhancedproducts services and to offer tailoredprotection to their customers. corporate Other technical infrastructure), place them in a highlyprivilegedposition productsunderlying (along andcommunicationthe with channels relevantreal-time datamassive at scale, their ownership and of technological safehavens masses,of the as their access to Tech giantsmedia giantsmaybecome continueto and likelyshort-livedbe to policiestechnology evolve. as and is betweenforcesparticular domain security criminalsand in a progressing, any equilibriumobtainedbetweenrivalstates or likelyis vast,be everthe cuttingedgeoftois and capability likely to bebroad. Since the number of possible attack surfaces governments,proliferationthe and ofsuch offensive is capability to defend against will be the ones most likely to be weaponized by between governments.Whichever vectors of attackprove hardest betweensocieties,within societies their governments, and and tostate anongoing low-medium-level ofto attacks, erodedtrust penalization of attackers lead likely whichcould be difficult, tois the absenceIn ofsignificant effort, attribution of attacks and scenarios,threesome all in attackand vectorsbe combined. may tosecure digital andcyber-physicalsystems from cyber attacks, harder to securehumans frommanipulation attacksbe it will than solutions with these characteristics. It islikely to prove much pessimisticOn the side,not all of thethreats haveidentified 6 5 4 3 2 1 US election. than theClintonhackplayedin more mutedroleintheFrenchelection ended upwiththehackplayingamuch leaks associatedwiththehack,and the campaignfromcapitalizingon emails hadbeenhacked.Thisprevented further campaigningonceMacron’s prohibited Macron’sopponentfrom For example,France’scampaignlaws Berg etal.,1974 Dijkstra, 1968 Krimsky, 1982;Wright,1994 Naur andRandell,1969 Chessen, 2017b p.63 Strategic Analysis

and proceduraland tosolutions address over-run, over-budget, growingrisks from software systems, and sketched out technical conference1968 createdin atGarmisch consensusaroundthe time humanityhasrisen to meet such a challenge: the NATO incentivesprovide and systemic vision. be thefirstnotThis will problems, new leadership will berequired to rise above local moreForsome ofthe challengingcoordination interdisciplinary and fortheir citizens the controlmechanisms that will enable themtoprovide security structures.haveSome countries in establishingheadstart aclear coupled withwell-designed incentives financial liability and infrastructure,throughinformed or andenforceableregulation betweenthe government privatethe and entities controllingsuch infrastructure,throughmeaningful andconstructive collaboration throughcould occur direct control of digital andcommunication own political stability in the face of malicioususes of AI pressureunder be Nations will toprotect their citizens and their accelerating beneficial research on myriad promising defenses. multi-stakeholderparticular domains,and risksin dialoguesthe on fromthe experience of beginning otherscientific disciplines, appropriate) experimenting with novel openness models,learning above,be actedtoday: canshould on(where and analyzing and takingprecautionary action today.recommendations, Our stated uncertainty andexpertparalyse disagreementnot fromus should get more data as the various threats and responses unfold, but this world. Many of these disagreements will not beresolved until we letreport,the aloneamongstvarious the expertin communities out Thereremainmany disagreements between ofthe co-authors this latterin the cases, includingEdsger Dijkstra in the former at the forefront of research played key roles inboth of these regulatorya frameworkresearch such could feedinto researchintonovel streams biological containment, of alongside moratoriuma oncertaintypes of experiments,initiated and the emerging risks from recombinant DNAresearch, promoted engineering resulting inmany practices which are now mainstream in software hard-to-maintaininfrastructurebug-ridden critical and software, . ; the NIH conference at Asilomar in1975 highlighted . and PaulBerg . Individuals . This

Conclusion

While many uncertainties remain, it is clear that AI will figure prominently in the security landscape of the future, that opportunities for malicious use abound, and that more can and should be done.

Artificial intelligence, digital security, physical security, and political security are deeply connected and will likely become more so. In the cyber domain, even at current capability levels, AI can be used to augment attacks on and defenses of cyberinfrastructure, and its introduction into society changes the attack surface that hackers can target, as demonstrated by the examples of automated spear phishing and malware detection tools discussed above. As AI systems increase in capability, they will first reach and then exceed human capabilities in many narrow domains, as we have already seen with games like backgammon, chess, Jeopardy!, p.64 Dota 2, and Go and are now seeing with important human tasks 2 1 Bostrom, 2014,p.107 al., 2016 Bostrom, 2014;AmodeiandOlahet p.65 Conclusion improved technicalmeasures for formally verifyingrobustness the in understanding the right balance of openness inAI, developing beyondlook weMorebe done the digital work domain. shouldalso strategy. ButAI-based defense isnot a panacea, especially when systems,indeed companies areincreasingly and pursuing this throughAI, automationis also via ofhacking ourcyber-defense hopesto best defendviruses.One of our against automated personal dataintelligenttheft, andanepidemicof computer emails. mayThis cause anexplosion networkpenetrations, of cybersecurity experts can fallprey to targetedphishing spear capabilities. These are very difficult to defend against, as even sophisticatedsocial engineeringattacks drawingthese on to beuniquelyhuman(like social interaction), we will see more systemsAI As intobelieved extend domainscommonly further urgentan task. potential malicioususes of AI associated with this transition is likeinvesting in thestockmarket or driving cars.Preparing for the safelysecurely.but also fairly and about ensuring that the rapid development of AIproceeds not just understandingnexus,the dialogue join toAI-security and ofthe waysin whichtheybe mightable to advance the collective preventionmitigation efforts. and Weurgereaders to consider is helpful in illuminating the world ahead and informing better the role of AIin enabling larger-scale and more numerous attacks, understandingthe commonalities acrosslandscape,including this physical,politicalbelieve domainsaremyriad,we and that risks useacrossspecific malicious Thoughthe thedigital, of AI applications.to touse developmalicious norms appropriatepolicies,tools, and domains with longer experience inpreventing and mitigating realizable. Researchers and policymakers should learn from other to begin today before these more potent misusepotentials are requiringpreparednessscalesin certainand domains, of damage are doing, such advanced AIsmay inflict unprecedented types own goals”property. Depending biddingsuchsystemson whose that is, we may see powerful AI systems with a “just add your dangerousthem ordevelopingbyhacking novo: goals them de little expertiseto develop ordeploymay eventually be givennew, deployedrangefor of a goals fall short of thistoday. Givenintelligence that systems be can acrossrange a very ofwide environments sophisticatedsystemsAI capable oflevels operatinghigh at mightproblems ariseaccidentallyresultwhich highly asa of Looking to the longer term, muchhasbeenpublished about lessAI-infuseda world adapt tonew theworld we are creating. ofsystems, policythatframeworks andensuring developedin , highly capable, systemsrequire that , though AI capabilitiesthough ,

The Malicious Use of Artificial Intelligence

p.66 Acknowledgements Acknowledgements was supported inpart by a grant from the Future of Life Institute. remaining errorsresponsibility arethe ofthe authors. Thiswork Weitzdorfer, EmmaBates, and Subbarao Kambhampati. Any Ord, NickBostrom, Owen Cotton-Barratt, EricDrexler, Julius Kunz,Martina Jade Leung, Katherine Fletcher, Jan Leike, Toby Kasewa, SmithaMilli,Itzik Kotler, Andrew Trask, Siddharth Garg, Papernot, Abadi, Martín Tim Hwang, Laura Pomarius, Tanya Singh among others, we thank Ian Goodfellow, Ross Anderson, Nicholas relatedand conversations, we willsurely forgetpeople,but some conversationsrelated about topics. number ofGiven coauthors the helpful versionsin us ofthis document,who engaged and practitionershavewho provided useful comments onearlier We are extremely gratefulresearchersmanyto the and The Malicious Use of Artificial Intelligence

p.67 “Blogs and Bullets II:New Media and Conflict After theArab ‘16). ACM, New York, NY, USA, 308-318, https://arxiv.org/ Their Exploits,” RAND Corporation, https://www.rand.org/ References A. 2017.A. “DeepReinforcement Learning: ABrief Survey,” IEEE Arulkumaran, K.,Deisenroth, M.,Brundage, M., and Bharath, Anderson, H.S., Kharkar, Filar, A., B., Evans, D., and Roth, P. Adversarially-TunedGenerationDetection,”Domain and arXiv Anderson, H., Woodbridge, J., and Filar, B. 2016. “DeepDGA: Amodei, D. and Olah, C. et al. 2016. “Concrete Problems Allen, G. and Chan, T. 2017. “Artificial Intelligence andNational Aker, C. and Kalkan, S. 2017. “UsingDeepNetworks for Aday, S., Farrell, H.,Lynch, M.,Sides, J., and Freelon, D. 2012. Adams, T. 2017. “AI-Powered SocialBots,” arXiv preprint Abokhodair, N., Yoo, D., and McDonald,D. 2015. “Dissecting Ablon, L. andBogart, T. 2017. “Zero Days, Thousands of Abadi, M. Chu ,A. Goodfellow, I.McMahan,H.Mironov, 2018. “Learning to Evade Static PEMachine Learning Malware Signal Processing Magazine, Vol. Issue 34, 6,November 2017, Security,” Harvard Kennedy SchoolBelfer Center for Science Spring,” United States Institute of Peace, server, https://arxiv.org/abs/1706.05143 Supported Cooperative Work &Social Computing, pp. 839- Communication More Than an Echo Chamber?,” Psychological Cambridge: MITPress. abs/1606.06565 and International Affairs, https://www.belfercenter.org/ abs/1706.05726 a SocialBotnet: Growth, Content and Influence in Twitter,” abs/1607.00133 Conference(CCS ComputerSecurity on Communications and I. Talwar, K. 2016.and Zhang,L. “DeepLearning with R. 2015. “Tweeting From Left to Is Right: OnlinePolitical Barberá, P., Jost, J., Nagler, J., Tucker, A., and Bonneau, Baier, C. and Katoen, J. 2008. Principles of Model Checking. Models via Reinforcement Learning,” arXiv preprint server, Drone Detection,” arXiv preprint server, https://arxiv.org/ 851, dl.acm.org/ft_gateway.cfm?id=2675208 ProceedingsACM 18th Conferenceof the on Computer Nights: LifeThe and TimesZero-Day of Vulnerabilities and Differential Privacy.” InProceedings of the 2016 ACM SIGSAC publication/artificial-intelligence-and-national-security bullets-ii-new-media-and-conflict-after-arab-spring https://www.usip.org/publications/2012/07/blogs-and- pubs/research_reports/RR1751.html https://arxiv.org/abs/1708.05866 https://arxiv.org/abs/1801.08917 preprint server,https://arxiv.org/abs/1610.01969 in AISafety,” arXiv preprint server, https://arxiv.org/ References “The security of machinelearning,” MachineLearning, 81:2, Twitter,” Brookings Institution, https://www.brookings.edu/ www.slbooth.com/piggybacking_robots.html doi/10.1177/0956797615594620 on Machine Learning (ICML 2012),on MachineLearning (ICML pages 1467–1474. 2, February 2017, http://dl.acm.org/citation.cfm?id=3023357 2017.“A messy state union: tamingof the the composite state University Dormitory Security,” HRI2017, available at http:// Sputnik.” November 21, 2017. http://www.bbc.com/news/ scientists-gather-to-plot-doomsday-scenarios-and-solutions Scenarios (and Solutions),” Bloomberg, March 2,2017, Science, Vol. 26, Issue 10, http://journals.sagepub.com/ security testing,” Microsoft Research Blog,https://www. Security Protocol Verifier,” http://prosecco.gforge.inria.fr/ arxiv.org/abs/1705.08504 Oxford: Oxford University Press. against support vectormachines,” International Conference C., Kohlweiss, M.,Pironti, Strub, A., P., and Zinzindohoue, J. R.W., Hogness, D.S., Nathans, D., Roblin, R., Watson, J.D., Berg, P., Baltimore, D., Boyer, H.W., Cohen, S.N., Davis, BBC. 2017.“Google to ‘de-rank’ Russia Today and Models via ModelExtraction,” arXiv preprint server, https:// Bastani, O., Kim, C., Bastani, H.2017. “Interpreting Blackbox Bass, D. 2017. “AI Scientists Gather to Plot Doomsday Barreno, M.,Nelson,B., Joseph, A., and Tygar, J.D. 2010. Bostrom. 2014. Superintelligence: Paths, Dangers, Strategies. R. 2017. “Piggybacking Robots: Human-Robot Overtrust in Booth, S., Tompkin, J., Gajos, K., Waldo, J., Pfister, H.,Nagpal, Blum, W.2017. “NeuralDNN to fuzzing: applying software Blanchet, B. 2017. “CryptoVerif: A Computationally-Sound Biggio, B., Nelson,B., and P. Laskov. 2012. “Poisoning attacks Beurdouche, B., Bhargavan, K.,Delignat-Lavaud, Fournet, A., supporters ISIS population Definingof anddescribingthe on Berger, J.M. and Morgan, J. 2015. “The ISIS Twitter census: Weissman, S. and Zinder, N.D., 1974. “Potential biohazards of technology-42065644 the-population-of-isis-supporters-on-twitter/ https://www.bloomberg.com/news/articles/2017-03-02/ai- pdf berkeley.edu/~tygar/papers/SML/sec_mach_learn_journal. pages 121-148. Available online at https://people.eecs. microsoft.com/en-us/research/blog/neural-fuzzing/ personal/bblanche/cryptoverif/cryptoverif.pdf machines of TLS,”ACM,Communications Vol. ofthe 60,Issue research/the-isis-twitter-census-defining-and-describing- recombinant DNAmolecules.” Science, 185(4148), p.303. The Malicious Use of Artificial Intelligence

p.68 www.wired.co.uk/article/chinese-government-social-credit- America,” Time, May 18,2017, http://time.com/4783932/ Age: Work, Progress, and Prosperity in a Time of Brilliant Atrocities inSyria,” The New York Times, August 22, 2017, edu/~pratyushmishra/docs/papers/usenix16-hvc.pdf 25th USENIXSecuritySymposium, people.eecs.berkeley. 273-291. syria-youtube-videos-isis.html score-privacy-invasion said that?,”preprint arXiv server:https://arxiv.org/ Clarke, R., Morell, M.,Stone, G., Sunstein, C., and Swire, abs/1705.02966 Chung, J., Jamaludin, A., 2017.and Zisserman, A. “You Chessen, M.2017. “The MADCOMFuture,” Atlantic Council Chessen, M.2017. “The AIPolicy Landscape, Medium,https:// C., Wagner, D., and Zhou, W. 2016. “Hidden Voice Commands,” Carlini, N.,Mishra, P., Vaidya, T., Zhang, Y., Sherr, M.,Shields, Carbon Black,2017. “Beyond the Hype:SecurityExperts California Law Review, Vol. 103, No. 3, pp. 513-63. Calo, R. 2015. “Robotics and the Lessons of Cyberlaw,” Calo, R. 2011. “Open Robotics,” Law Maryland Review, Vol. 70, Calabresi, M.2017. “InsideRussia’s SocialMedia War on Intelligence and Law, Vol. 25, Issue 3, September 2017, pp. President’sReview GroupIntelligence Communications on and P. 2013. “Liberty and Securityin a Changing World,” The Malware Attacks.” https://www.carbonblack.com/2017/03/28/ 3. No. New York:PublicAffairs. Handbook: Why BadBehavior isAlmost Always Good Politics. Bueno de Mesquita, B. 2012.and Smith,A. The Dictator’s Policymakers,” Paper, SecurityProject,Cyber Belfer Center. Buchanan, B., and Taylor, M.2017. “MachineLearning for Bryson, J., Diamantis, M., and Grant, T. 2017. “Of, for, and by Machines. New York: W.W. Norton & Company, Inc. Brynjolfsson, E. and McAfee, 2014. A. The SecondMachine Browne, M. 2017. “YouTube Removes Videos Showing Botsman, R. 2017. “Big data meets BigBrother as China Weigh in Intelligence,on Artificial Machine Learning, andNon- the-madcom-future the-ai-landscape-ea8a8b3c3d5d the people: the legallacuna of synthetic persons,” Artificial policymakers https://www.belfercenter.org/publication/machine-learning- https://www.nytimes.com/2017/08/22/world/middleeast/ moves to rate its citizens,” Wired UK, October 21, 2017, http:// report, http://www.atlanticcouncil.org/publications/reports/ medium.com/artificial-intelligence-policy-laws-and-ethics/ machine-learning-non-malware-attacks/ beyond-hype-security-experts-weigh-artificial-intelligence- inside-russia-social-media-war-america/ References “Autoencoderrecurrent with neuralnetworks for video forgery Technologies,” https://obamawhitehouse.archives.gov/ Autonomy in Weapon Systems.” gc.ca/en/assemblyline conversation-autonomous-weapon-systems org/events/artificial-intelligence-and-global-security-summit considered harmful. Communications of the ACM, 11(3), cybergrandchallenge.com detection,”preprint, arXiv availablehttps://arxiv.org/ at edged-sword.html com/en_us/blog/black-hat-attendees-see-ai-as-double- 2013. Strip,” Haaretz, http://www.haaretz.com/israel-news/1.773465 Summit,” Centre for New American Security. https://www.cnas. Sword,” Team,https://www.cylance.Cylance Theavailable at CSE, 2017. “Assemblyline”. October 2017. https://www.cse-cst. Crootof, R. and Renz, F. 2017. “An Opportunity to Change Crootof, R. 2015. “The KillerRobots are Here: Legal and Crawford, K. and Calo, R. 2016. “There is a blind spot inAI Open Robotics,” paper presented at We Robot, April2013. Cooper, D.M. 2013. “A LicensingApproach to Regulation of Cohen, G. 2017. “Israel Shoots Down HamasDrone Over Gaza CNAS. 2017. “Artificial Intelligence and Global Security abs/1708.08754 Cylance. 2017. “BlackHat Attendees SeeAIAs Double-Edged Control Interface Design.” IEEE Technology and Society 2004. “CreatingCummings, M.L. Moral Buffers in Weapon Lawfare, https://www.lawfareblog.com/opportunity-change- Policy Implications,” http://isp.yale.edu/sites/default/files/ Dijkstra,E.W., Letters1968. to the editor: go to statement Documents/law_war_manual15.pdf Law of War Manual,” https://www.defense.gov/Portals/1/ Department of Defense. 2015. “Department of Defense Department of Defense. 2012. “DoDDirective 3000.09: Proceedings of the 5thAnnualACM Web Science Conference, De Choudhury, M., Counts, S., E.2013.and Horvitz, “Social DARPA.2016. Grand Cyber Challenge, www. Much as Those in Combat Do,” New York Times. February 22, Dao, James. “Drone Pilots Are Found to Get Stress Disorders D’Avino, D., Cozzolino, D., Poggi, G., and Verdoliva, 2017. L. Magazine (Fall 2004), 29–30. the ConversationAutonomous on WeaponSystems,” publications/killer_robots_are_here_final_version.pdf news/there-is-a-blind-spot-in-ai-research-1.20805 research,” Nature, October 13, 2016, https://www.nature.com/ blog/2013/12/18/liberty-and-security-changing-world pp.147-148. pp. 47-56, https://dl.acm.org/citation.cfm?id=2464480 measurementmedia asa tool ofpopulations,” depressionin The Malicious Use of Artificial Intelligence

p.69 www.microsoft.com/en-us/research/publication/differential- Accuracy,” Proceedings of The 33rd International Conference 31st ACM/SIGAPP Symposium on Applied Computing (SAC), 33rd International Colloquium on Automata, Languages and gizmodo.com.au/2017/09/hackers-have-already-started-to- Learning, availableMachine http://proceedings.on onlineat google-data-centre-cooling-bill-40/ consequences-under-dmca/archive ofseries aavalablehttps://www.eff.org/wp/unintended- at 2016. “Preparing for the Future of Intelligence,” Artificial 2016, https://deepmind.com/blog/deepmind-ai-reduces- server, https://arxiv.org/abs/1707.08945 S. 2017. “Reinforcement Learning with a Corrupted Reward October 2016, https://obamawhitehouse.archives.gov/ on Technology,Committeeand Technology Council, Channel,” availablehttps://arxiv.org/abs/1705.08417 onlineat Online Deception: What Makes Automated Text Convincing?, Computer Science, vol 740. Berlin:Springer. Cryptology — CRYPTO’ 92. CRYPTO 1992. Lecture Notes in Combatting Junk Mail.” InBrickell E.F. (eds) Advances in Executive Office of thePresident, National Science Prakash, Rahmati, A., A., and Song,D. 2017. “Robust Physical- Evtimov, I., Eykholt,K.,Fernandes, E., Kohno, T., Li,B., Everitt, T., Krakovna, V., Orseau, Hutter, L., M., and Legg, Everett, R., Nurse, J., and Erola, 2016. A. “The Anatomy of Data Centre Cooling Billby 40%,” DeepMindblog, July 20, Evans, R. and Gao, J. 2016. “DeepMindAIReduces Google wp/unintended-consequences-16-years-under-dmca, part DMCA,” Electronic Frontier Foundation, https://www.eff.org/ EFF. 2014. “Unintended Consequences -16 Years Under the Foundation, https://www.eff.org/deeplinks/2017/06/help-eff- Progress of AI and MachineLearning,” Electronic Frontier Eckersley, P., Nasser, Y., et al. 2017. “HelpEFF Track the Programming, part II(ICALP 2006), available online at https:// Dwork, C. 2006. “Differential Privacy,” Proceedings of the Dwork, C. and Naor, M.1993. “Pricing via Processing or weaponize-artificial-intelligence/ Dvorsky, G. 2017. “Hackers Have Already Started to Networks to Encrypted Data with High Throughput and M., and Wernsing, J. 2016. “CryptoNets: ApplyingNeural Dowlin, N., Gilad-Bachrach, R., Laine, K.,Lauter, K.,Naehrig, Krueger,Dinh, L., D. and Bengio, Y., 2014. “NICE:Non-linear Weaponize Intelligence,” Artificial Gizmodo, https://www. World Attacks on DeepLearning Models,” arXiv preprint track-progress-ai-and-machine-learning privacy/ mlr.press/v48/gilad-bachrach16.html paper, https://arxiv.org/abs/1410.8516 blog/2016/10/12/administrations-report-future-artificial- https://dl.acm.org/citation.cfm?id=2851813 independent components estimation,”2015ICLR workshop intelligence References “Pricing Externalities to BalancePublicRisks and Benefits of “Generative Adversarial Networks.” InAdvances inNeural (SOUPS). https://arxiv.org/abs/1707.05768 A Conversational Interface to Augment Information Security cfm?id=2628165&CFID=776732616&CFTOKEN=19198339 of the 19thACM international SIGPLAN conference on https://www.ncbi.nlm.nih.gov/pubmed/28767274online at explanation”.preprint arXiv arXiv:1606.08813. experiment-stirring-outcry.html?_r=0 cyberchef-cyber-swiss-army-knife 2014. https://www.nytimes.com/2014/06/30/technology/ vehicles: DARPA’s HACMS program,” ICFP ‘14:Proceedings Security. ACM, 2015, pp. 1322–1333. Available at http://doi. SIGSAC Conference on Computer and Communications facebook-tinkers-with-users-emotions-in-news-feed- feed experiment,stirring outcry.” New York Times.29, June Organized Russian Ransomware Campaign,” (registration Goodman, B. and Flaxman, S., 2016. European Union availablehttps://arxiv.org/abs/1406.2661 at Goodfellow, I.,Pouget-Abadie, J., Mirza, M., Xu, B., Warde- Goel, V. 2014. “Facebook tinkers with users’ emotions innews arXiv preprint server, https://arxiv.org/abs/1708.07342 Giaretta, A. and Dragoni, N.“Community Targeted Spam: A GCHQ, 2016. “CyberChef - the “SwissCyber Army Knife”.” Consequences.”Possible and Their Garfinkel, B. “RecentForthcoming. Advances in Cryptography ai-principles/ signatoriesand availablehttps://futureoflife.org/ onlineat acm.org/10.1145/2810103.2813677 Fredrikson, M., Jha, S., and Ristenpart, T. 2015. “Model Franke, U. 2016. “FlyingIEDs: The Next Big Threat?,” War on Flashpoint. 2016. “Ransomware as Inside a Service: an Functional programming. http://dl.acm.org/citation. Fisher, K.2014. “Using formal methods to enable more secure Filar, B., Seymour, R.J. and Park,M.,2017. “Ask MeAnything: Research,” Health Security, pages 15:4, 401-408, available Farquhar, S., Cotton-Barratt, O., and Snyder-Beattie, 2017. A. Farley, D., Ozair, S., Courville, A. and Bengio, Y., 2014. Middle Ground Between General Spam and Spear Phishing,” December 2016. https://www.gchq.gov.uk/news-article/ Future of Life Institute, 2017. “Asilomar AIPrinciples,” text Workers.” InSymposium on UsablePrivacy and Security the Rocks blog,https://warontherocks.com/2016/10/flying- basic countermeasures,” inProceedings of the 22nd ACM https://www.flashpoint-intel.com/library/ requiredfor download), availablelibraryfromFlashpoint at regulationsrightto onalgorithmic decision-makinganda inversion attacksthat exploit information confidence and ieds-the-next-big-threat/ information Processing Systems 2014 (pp. 2672-2680), The Malicious Use of Artificial Intelligence

p.70 “Assessing the ThirdStrategy,” Offset Center forStrategic T., Piot, Dulac-Arnold, B., Sendonaris,A., G., Osband, I., Theory and Practice. American Academy of Arts and Sciences, Agapiou, J. and Leibo, J.Z., 2017. “Deep Q-learning from election-bots/506072/ criminals-4626 edu/blog/insead-blog/the-professionalisation-of-cyber- content/publications/publication.aspx?d=22228 com/story/facebook-can-absolutely-control-its-algorithm/ 2017. “When Will AIExceed HumanPerformance? Evidence University Press. See further details at http://allminds.org/ Shaping the Election,” The Atlantic, November 1, 2016, Second NASA Formal Methods Symposium, available at space-war/ scientificamerican.com/article/gps-and-the-world-s-first- from AIExperts,” arXiv preprint server, https://arxiv.org/ assessing-third-offset-strategy and International Studies, https://www.csis.org/analysis/ abs/1704.03732 AvailableCambridge. http://www.amacad.org/ onlineat availablehttps://arxiv.org/abs/1606.04435 onlineat Grosse, K.,Papernot, N.,Manoharan, P., Backes, M., and algorithm.” Wired. September 26, 2017. https://www.wired. Griffith, E. 2017. “Facebook can absolutely control its Greenemeier, 2016. L. “GPS and the World’s First ‘Space availablehttps://www.wired.com/2016/08/jeep- onlineat Much Worse,”Hacking CanGet Car Wired, January,2016, Greenberg, 2016, A. “The Jeep Hackers are Back to Prove abs/1705.08807 Grace, K.,Salvatier, J., Dafoe, Zhang,B., A., and Evans, O. Guilbeault, D. and Woolley, S. 2016. “How Twitter Bots Are abs/1708.06733 Chain,”preprint arXiv availablehttps://arxiv.org/ at Gu, T., Dolan-Gavitt, B., and Garg, S. 2017. “BadNets: Identifying Vulnerabilities in the MachineLearning Supply INSEAD Business Schoolblog,https://knowledge.insead. Hilary,2016. Gilles.Professionalization “The Crime,”of Cyber Hicks, K.,Hunter, A.P., Samp, L.S., and Coll, G. 2017. Demonstrations.” arXiv preprint server. https://arxiv.org/ Hester, T., Vecerik, M.,Pietquin, O., Lanctot, M.,Schaul, Evaluating Natural Intelligence.and Artificial Cambridge Hernández-Orallo, J. 2017. The Measure of AllMinds: Hawkes, Rebecca. “Post-Traumatic Stress Disorder IsHigher Harris, E., ed. 2016. Governance of Dual-Use Technologies: Harrison, John. 2010 “Formal Methods at Intel –An Overview.” Neural Networks for Malware Classification,” arXivpreprint McDaniel, P. 2016. “Adversarial Perturbations Against Deep War’,” Scientific American, 8, February 2016, https://www. pdf https://www.cl.cam.ac.uk/~jrh13/slides/nasa-14apr10/slides. hackers-return-high-speed-steering-acceleration-hacks/ https://www.theatlantic.com/technology/archive/2016/11/ in Drone Operators.” The Telegraph. May 30, 2015. References “Progressive Growing of GANs forImproved Quality,Stability, Jordan, M.I., and Mitchell, T.M. 2015. “Machinelearning: Jones, B. and Mattiacci, E.2017. “A Manifesto in140 Ji, Z.,Lipton, C., and Elkan, C. 2014. “Differential privacy and J. Z.,Silver, D., &Kavukcuoglu, K.2016. “Reinforcement Jaderberg, M.,Mnih, V., Czarnecki, W. M.,Schaul, T., Leibo, Trends, perspectives, and prospects,” Science Vol. 349, Issue www.icrc.org/en/document/expert-meeting-lethal- Autonomous Systems,” https://standards.ieee.org/develop/ downloads/ of the Atomic Scientists: http://thebulletin.org/who’ll- 6245, pp. 255-260, DOI:10.1126/science.aaa8415 cambridge.org/core/journals/british-journal-of-political- 2017. “Reluplex: AnEfficient SMTSolver for Deep Verifying 8A567FE22531F scale/ Security 2010, https://www.microsoft.com/en-us/research/ server, https://arxiv.org/abs/1704.05051 science/article/manifesto-in-140-characters-or-fewer-social- server. https://arxiv.org/abs/1611.05397 for Ethical Considerations Intelligence inArtificial and autonomous-weapons-systems Cloud Vision APIIsNot Robust To Noise,” arXiv preprint autocracies9692 abs/1702.01135 arXiv preprint server, https://arxiv.org/abs/1703.09968 and Variation,” availablehttps://t.co/CCHghgL60t onlineat CharactersFewer:Media asa or Social Tool Rebel of arXiv:1412.7584 International Committee Redof theCross.2017. “Expert IFR, 2016. “World Robotics 2016,” https://ifr.org/free- IEEE Standards Association, 2017. “The IEEE Global Initiative Ixy, 2017. “Ixy- the conflict free app,” http://getixy.com/ Meeting on Lethal Autonomous Weapon Systems.” https:// Herley, C. 2010. “The Plight of the Targeted Attacker in a Hosseini, H., Xiao, B. and Poovendran, R., 2017. “Google’s want-artificially-intelligent-weapons-isis-democracies-or- weapons?ISIS, democracies, orautocracies?,”Bulletin Horowitz, M.2016. “Who’ll want intelligentartificially Neural Networks,”Neural preprint arXiv availablehttps://arxiv.org/ at Katz, G., Barrett, C., Dill,D., Julian, K., and Kochenderfer, M. Evaluation of Digital ImageForgery Detection Approaches,” Kashyap, Parmar, A., R., Agarwal, M., Gupta, H.2017. “An Karras, T., Aila, T., Laine, S., and Lehtinen, J. 2017. Diplomacy,” British Journal of Political Science, https://www. Learning Auxiliary with Unsupervised Tasks.” arXiv preprint WorldScale,” of Workshop EconomicsInformation ofthe on publication/the-plight-of-the-targeted-attacker-in-a-world-of- media-as-a-tool-of-rebel-diplomacy/82518E669A274B26E89 machine learning: A survey and review”. In:arXivpreprint indconn/ec/autonomous_systems.html The Malicious Use of Artificial Intelligence

p.71 “Model Extraction Warning inMLaaSParadigm,” arXiv preprint day-of-russia-hearings/ day of Russia hearings.” Wired. November 1, 2017. https://www. evidencemassive-scale of emotional contagionthrough of the United States of America, Vol. 110, No. 15,http://www. cyberspace-less-safe/ delivery-in-san-francisco/ com/2017/04/12/marble-and-yelp-eat24-start-robot-food- delivery inSanFrancisco,” TechCrunch, https://techcrunch. com/s/608288/ai-fight-club-could-help-save-us-from-a- engaged-argument/4662DB26E2685BAF1485F14369BD137C 2014. http://www.pnas.org/content/111/24/8788.full.pdf 2017, availablehttps://arxiv.org/abs/1703.04730 onlineat 2017,the associated information competitionfound be at can science-review/article/how-the-chinese-government- Science Review, Vol. 111, Issue 3, August 2017, pp. 484-501, server, https://arxiv.org/abs/1711.07221 social networks.” PNAS. Vol 111, No. 24. 8788-8790. June 17, future-of-super-smart-cyberattacks/ As of September 27, fabricates-social-media-posts-for-strategic-distraction-not- and attributes arepredictable from records digital human of a Future of Super-Smart Cyberattacks,” MIT Technology algorithmic-bias/abstract Communications of the ACM, Vol. 59, No. 10, pp. 16-17, Government Fabricates SocialMediaPosts for Strategic Classifiers.” USENIX Security, 2017. wired.com/story/six-revealing-moments-from-the-second- Lapowsky, I.2017. “Eight revealing moments from the second Krimsky,S.,1982. Genetic alchemy: historyThe social of the Kramer, A., Guillory, J., and Hancock, J. 2014. “Experimental Kosinski, M.,Stillwell, D., and Graepel, T. 2013. “Private traits Era?,” The Diplomat. https://thediplomat.com/2017/07/ Korzak, E.2017. “UN GGE on Cybersecurity: The End of an Kolodny, 2017. L. “Marble and Yelp Eat24 start robot food Predictions via Influence Functions,” Proceedings of ICML Koh, P.W., and Liang,P. 2017. “Understanding Black-Box Review, availablehttps://www.technologyreview. onlineat Knight, W. 2017. “AI Fight Club Could HelpSave UsFrom Kirkpatrick, K.2016. “Battling AlgorithmicBias,” Distraction, not EngagedArgument,” American Political King, G., Pan, J., and Roberts, M.2017. “How the Chinese H. S. 2017. “Approaches to Evading Windows PEMalware Kharkar, Simecek,H., Xu, A., W., Evans, D. and Anderson, Kesarwani, M.,Mukhoty, B., Arya, V., and Mehta, S. 2017. un-gge-on-cybersecurity-have-china-and-russia-just-made- https://www.kaggle.com/google-brain/competitions https://cacm.acm.org/magazines/2016/10/207759-battling- https://www.cambridge.org/core/journals/american-political- recombinant DNA controversy. M.I.T. Press, Cambridge, MA. pnas.org/content/110/15/5802.full behavior,” Proceedings of the National Academy of Sciences References Journal of Privacy and Confidentiality, Vol. 1, No. 1, pp. Approach toAccess Control and Concurrency Control.Phd 7d7f-11e7-ab01-a13271d1ee9c org/abs/1606.07536 docs/50.pdf cgi?article=1004&context=jpc com/uk/resources/reports/rp-hacking-skills-shortage.pdf com/2017/04/25/magazine/can-facebook-fix-its-own-worst- 2016. “Hacking the SkillsShortage,” https://www.mcafee. 2013. “The Economic Impact of Cybercrime and Cyber Systems 2016, (NIPS) preprint available online at https://arxiv. Computation for Privacy-Preserving Data Mining,” The and Information, Security, http://www.cl.cam.ac.uk/~rnc1/ Ostrovski, G. and Petersen, S., 2015. “Human-level control Gmail Spam,” Wired, July 9, 2015. availablehttps://www.ft.com/content/0d87a148- onlineat In WEIS.http://www.econinfosec.org/archive/weis2006/ 59-98, http://repository.cmu.edu/cgi/viewcontent.59-98, Lucas, L. 2017.Lucas, L. “World’s biggest drone maker DJI eyes move Networks,” Proceedings of Neural Information Processing Liu, M. and Tuzel, O. 2016. “Coupled Generative Adversarial Liu, D. and Camp, L.J., 2006. “Proof of Work can Work”. Lindell, Y. and Pinkas, B. 2009. “Secure Multiparty Lin, J. and Singer, P. 2017. “Come see China’s new Naval Institute Press. Libicki, R. 2016. Cyberspace inPeace and War. Annapolis: Laurie, B. and Clayton, R., 2004. “Proof-of-work proves Bellemare, M.G., Graves, Riedmiller, A., M.,Fidjeland,A.K., Mnih, V., Kavukcuoglu, K.,Silver, Veness, D., Rusu,A.A., J., Dissertation. http://www.erights.org/talks/thesis/markm- Miller, M.,2006. Robust Composition: Towards a Unified Metz, C. 2015. “Google Says Its AI Catches 99.9 Percent of McAfee and Center for Strategic and International Studies. Espionage,” https://www.mcafee.com/uk/resources/reports/ McAfee and Center for Strategic and International Studies. DataAndSociety_MediaManipulationAndDisinformationOnline. Disinformation Online.https://datasociety.net/pubs/oh/ A. Marwick, and Lewis, R. MediaManipulation and New York Times Magazine. April25, 2017. https://www.nytimes. Manjoo, Farhad. 2017. “Can Facebook fix its own worst bug?” through deep reinforcement learning,” Nature, 518(7540), thesis.pdf to commercial applications,” Financial Times, August 10, 2017, http://www.popsci.com/china-new-drones-army-hexacopters hexacoptersself-detonating and drones,”Science,Popular proofwork2.pdf not to work”; version0.2. InWorkshop Economics on pp.529-533. rp-economic-impact-cybercrime.pdf pdf bug.html The Malicious Use of Artificial Intelligence

p.72 “Towards the Science of Security and Privacy inMachine Transformations,” 2016 IEEESymposium on Security and ArtificialIntelligence.pdf Analysis: Intelligence,” Artificial U.S. Department of Homeland drones-being-debated.html org/abs/1610.00768; associatedrepositoryGitHub available openai.com/more-on-dota-2/ com/dota-2/ cases grant solicitation,https://www.nsf.gov/funding/pgm_summ. Security, National Protection and Programs Directorate, at https://github.com/tensorflow/cleverhans 2017.OpenMined, OpenMined website,http://openmined.org OpenAI, 2017. “More on Dota 2,” OpenAI blog,https://blog. OpenAI, 2017. “Dota 2,” OpenAI blog,https://blog.openai. availablehttps://info.publicintelligence.net/OCIA- at Office ofCyber andInfrastructure Analysis, 2017. “Narrative at https://citizenlab.org/2015/07/tracking-censorship-on- Censorship on WeChat’s PublicAccounts Platform,” available COMMITTEE, Garmisch,Germany,7th to11th October1968. Image Authentication for Any Set of Permissible 572—599 McDaniel, P. 2016. “Cleverhans v.1.0.0: an adversarial machine Papernot, N., Goodfellow, I.,Sheatsley, R., Feinman, R., and Learning,” availablehttps://arxiv.org/abs/1611.03814 onlineat Papernot, N.,McDaniel,P., Sinha,A., and Wellman, M.2016. wechat-public-accounts-platform/#append-documented- Ng, J. 2015. “Politics, Rumours, and Ambiguity: Tracking Neema, S. (2017) “Assured Autonomy”. https://www.darpa.mil/ Distributed System SecuritySymposium 2018. http://www. NDSS. 2018. “NDSS2018 Call for Papers”, The Network and Privacy (SP).http://ieeexplore.ieee.org/document/7546506/ Naveh, A. and Tromer, E.2016. “PhotoProof: Cryptographic National ScienceFoundation. 2017. “Cyber-Physical Systems,” Nato. Report onaconferenceNATOsponsoredby the SCIENCE Naur, P. and Randell,B. eds., 1969. Software Engineering: Debated,” New York Times, December10, 2015, https://www. Mouawad, J. 2015. “Risk to Aircraft from Drones Being Freedom. New York: PublicAffairs. Morozov, E.2012. The Net Delusion: The DarkSide of Internet Peitz, M. and Waldfogel, J. (eds.) The Oxford Handbook of Moore, T. and Anderson, R. 2012. “Internet Security.” In jsp?pims_id=503286&org=CISE&sel_org=CISE&from=fund the Digital Economy, Oxford University Press, New York, NY, program/assured-autonomy ndss-symposium.org/ndss2018/ndss-2018-call-papers/ nytimes.com/2015/12/11/business/risk-to-aircraft-from- learning library,”learning preprintserver, arXiv availablehttps://arxiv. at References “Video Liveness for Citizen Journalism: Attacks and Defenses,” “Our Twitter Profiles, OurSelves: Predicting Personality with Twitter,” 2011 IEEE Third International Conference on Privacy, Adversarial Networks,” arXiv preprint server, https://arxiv.org/ America’s Military Deterrence,” DoDNews, https://www. document/6113111/ defense.gov/News/Article/Article/991434/deputy-secretary- learningsystems deep using adversarial examples,” arXiv duckofminerva.com/2016/03/autonomous-weapons-and- com/doi/abs/10.1177/0022343314555782 change-2017/ com/2017/06/30/facebook-news-feed-algorithm- 2017. “Automatic Detection of Fake News,” arXiv preprint Security, Risk and Trust and 2011 IEEE Third International server: https://arxiv.org/abs/1708.07104 server, https://arxiv.org/abs/1705.08963 Scalable Provably-Secure DeepLearning,” arXiv preprint for Oppression” The Duck of Minerva Available http://at: for delegates at theConvention on Certain Conventional abs/1511.06434 Conference on Social Computing, http://ieeexplore.ieee.org/ Quercia, D., Kosinski, M.,Stillwell, D., and Crowcroft, J. 2011. and Swami, 2016b. A., “Practical black-box attacks against autocrats? The Internet in authoritarian regimes,” Journal of algorithm again.” Fortune. June 30, 2017. http://fortune. Intelligence and Autonomous Weapons,” briefing paper IEEE Transactions on Mobile Computing, Vol: PP, Issue 99, Rahman, M., Azimpourkivi, M., Rahman, M.,Azimpourkivi, Topkara, U., Carbunar, B. 2017. RepresentationDeep ConvolutionalLearning with Generative Radford, Metz, L. and A., Chintala, S., 2015. “Unsupervised Pérez-Rosas, V., Kleinberg, B., Lefevre, A., and Mihalcea, R. Pellerin, C. 2016. “DeputySecretary: Third Offset Bolsters Papernot, N.,McDaniel,P., Goodfellow, I., Jha, S., Celik, Z.B. Rouhani, B., Riazi,M., and Koushanfar, F. 2017. “DeepSecure: Roff, H.2016. “Autonomous Weapons and Incentives Roff, H.2016 “Meaningful Human Control, Artificial Roff, H.2016. “Autonomy, Robotics, and Collective Systems,” Peace Research, Vol. 52, Issue 3, http://journals.sagepub. Rød, E. and Weidmann, N.2015. “Empowering activists or Reuters. 2017. “Facebook is changing its news feed Weapons Systems (LAWS), www.article36.org/wp-content/ Weapons (CCW) Meeting of Experts on Lethal Autonomous third-offset-strategy-bolsters-americas-military-deterrence/ uploads/2016/04/MHC-AI-and-AWS-FINAL.pdf preprint server,https://arxiv.org/abs/1602.02697 robotics-autonomy projectwebpage, availablehttps://globalsecurity.asu.edu/ at http://ieeexplore.ieee.org/abstract/document/7887755/ incentives-for-oppression.html The Malicious Use of Artificial Intelligence

p.73 ‘AlphaChem’: Chemical Synthesis Planning with Tree Search Twitter-wp.pdf Autonomous-weapons-operational-risk.pdf defeating-robotic-swarms/ edu/~tygar/papers/SML/IMC.2009.pdf com/2014/01/theres-no-good-way-to-patch-the-internet-of- co.uk/news/magazine-22909590 changed the world,” BBC, June 17, 2013, http://www.bbc. 2017. “The spread of fake news by social bots,” arXiv preprint 2017, availablehttps://www.nytimes.com/2017/09/23/ at s3.amazonaws.com/files.cnas.org/documents/CNAS_ Scharre, P. 2016. “Autonomous Weapons and Operational Scharre, P. 2015. “Counter-Swarm: A Guide to Defeating S., Taft, N., and Tygar, J.D. 2009. “ANTIDOTE: Understanding server, https://arxiv.org/abs/1707.07592 Shao, C., Ciampaglia, L., Varol, Menczer,O., Flammini,A., F. social engineering: Automated E2E spear phishing on Twitter,” Seymour, J. and Tully, P. 2016. “Weaponizing data science for Subramanian, S., Kim, T., Pieper, M., Chandar, S., Ke, N., Serban, I.,Sankar, C., Germain, M.,Zhang,S., Lin,Z., Selsam, D., Liang,P., Dill,D. 2017. “Developing Bug-Free Segler, M.Preuß, M., and Waller, M.2017. “Towards Schneier on Security, https://www.schneier.com/blog/ Schneier, B. 2017. “Security and the Internet of Things,” Schneier, B. 2014. “The Internet of Things is Wildly Schofield, H. 2013. “How Napoleon’s semaphore telegraph Schmitt, E.2017. “Pentagon Tests Lasers and Nets to Combat Scharre, P. 2018. Army of None:Autonomous Weapons and abs/1709.02349v2 and DeepNeural Network Policies,” arXiv preprint server, archives/2017/02/security_and_th.html a Vexing Foe: ISISDrones,” New York Times, September 23, and Defending against Poisoning of AnomalyDetectors,” Internet Measurement, pp. 1-14, https://people.eecs.berkeley. Insecure--and OftenUnpatchable,”Insecure--and Wired,https://www.wired. For-Social-Engineering-Automated-E2E-Spear-Phishing-On- Black Hat conference, https://www.blackhat.com/docs/us-16/ Deep Reinforcement Learning Chatbot,” https://arxiv.org/ Michalski, V., Nguyen, Pineau, A., J., and Bengio, Y. 2017. “A Rajeshwar, S., de Brebisson, Sotelo, A., J., Suhubdy, D., Machine Learning Systems with Formal Mathematics,” arXiv world/middleeast/isis-drones-pentagon-experiments.html Risk,” Center for a New American Security,” http:// warontherocks.com/2015/03/counter-swarm-a-guide-to- Robotic Swarms,” War on the Rocks blog,https:// Proceedings of the 9thACM SIGCOMM Conference on Rubinstein, B., Nelson,B., Huang,L., Lau,S.,Joseph, A., Rao, things-and-thats-a-huge-problem/ the Future of War. New York, NY: W.W. Norton. (forthcoming) materials/us-16-Seymour-Tully-Weaponizing-Data-Science- preprint server,https://arxiv.org/abs/1706.08605 https://arxiv.org/abs/1702.00020 References “Membership inference attacks against machinelearning networksneural“Mastering with deep the gameofGo tree and T. 2016. “Summoning Demons: The Pursuit of Exploitable A., A., Chen, Y., Lillicrap, T., Hui,F., Sifre, L., van den Driessche, Šrndic, N. and Laskov, P. 2014. “Practical Evasion of a of Go without HumanKnowledge,” Nature 550:354-359, drones economist.com/technology-quarterly/2017-06-08/civilian- of SocialMedia.Princeton: Princeton University Press. van den Driessche, G., Schrittweiser, J., Antonoglu, I., Shu, K., Wang, S., Sliva, A., Tang, J., and Liu,H.2017. “Fake Shokri, R., Stronati, M., and Shmatikov, V. 2016. Silver, D., Schrittweiser, J., Simonyan, K.,Antonoglu, I., Silver16.pdf search”, Nature 529 http://web.iitd.ac.in/~sumeet/ pp484-9 Silver Maddison, D., Huang,A., C., Sifre,Guez, A., L., Shehadeh, K.K.,1999. The Wassenaar Arrangement and Standage, T. 2017. “Taking flight,” The Economist, http://www. Sunstein, C. 2017. #Republic: DividedDemocracy in the Age Solomon, B. 2017. “Witnessing an ISISDrone Attack,” Stoica, I.,Song,D., Popa, R., Patterson, D., Mahoney, M., Stocky, T. Facebook post. May 10, 2016. Stevens, R., Suciu, O., Ruef, Hong,S., A., Hicks, M.,Dumitras, Singer, P. 2009. Wired for War: The Robotics Revolution and arxiv.org/abs/1610.05820 agz_unformatted_nature.pdf October 19, 2017, https://deepmind.com/documents/119/ G., Graepel, T., and Hassabis, D. 2017. “Mastering the Game Goldberg, K., Ghodsi, A., Culler, D., and Abbeel,P. 2017. “A Conflict in the 21st Century, London: Penguin Press. arXiv preprint, https://arxiv.org/abs/1708.01967 Int’l L. Rev.,Int’l L. 15,p.271. IEEE ComputerSociety. Huang, A., Hubert, Huang, A., Guez, A., T., Baker, Lai,M.,Bolton, L., M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. 2016. Nham, J., Kalchbrenner, N.,Sutskever, I.,Lillicrap, T., Leach, Paneershelvam, V., Lanctot, M.,Dieleman,S., Grewe, D., Encryption Exports: AnIneffective Export Control Regime New YorkNew https://www.nytimes.com/video/world/ Times, Pubs/TechRpts/2017/EECS-2017-159.html No. UCB/EECS-2017-159, http://www2.eecs.berkeley.edu/ Berkeley View of Systems Challenges for AI,” Technical Report Hellerstein,M., Jordan, J.,A., Gonzalez,R., Katz, J., Joseph, Processing Systems 2016, Reliable MachineLearning in the Bugs inMachineLearning,” Proceedings of Neural Information Learning-Based Classifier: A CaseStudy.” InProceedings of News Detection on SocialMedia:AData MiningPerspective,” Wild workshop,https://arxiv.org/abs/1701.04739 that Compromises United States Economic Interests. Am.U. the 2014 IEEESymposium on Security and Privacy, pp. 197-211. models,” CoRR, vol. abs/1610.05820, 2016. Available at http:// middleeast/100000005040770/isis-drone-attack-mosul.html posts/10100853082337958?pnref=story https://www.facebook.com/tstocky/ The Malicious Use of Artificial Intelligence

p.74 Jaderberg, M.,Silver, D. and Kavukcuoglu, K.,2017. “Feudal Task Force on Supply Cyber Chain,” http://www.acq.osd.mil/ Turing, 1949. A. “Checking a large routine,” report of a Technologies. Cambridge: MITPress. Tucker, J. (ed.). 2012. Innovation, DualUse, and Security: Timm, T. 2013. “Prominent SecurityResearchers, Academics, Thies, J., et al. 2016. “Face2Face: Real-time Face Capture The Telegraph, 2016. “Drone hits BritishAirways plane as it Tambe, M.2011. Security and Game Theory: Algorithms, Vincent, J. 2016. “The UK government is crashing drones into Vezhnevets, A.S., Osindero, S., Schaul, T., Heess, N., Verma, I.“Editorial Expression of Concern and Correction.” Vanian, J. 2017. “Drone Registrations are StillSoaring,” Aaron’s Law,” Electronic Frontier Foundation https://www.eff. Vision and Pattern Recognition 2016, www.graphics.stanford. of neural networks.” arXiv preprint server, https://arxiv.org/ crash-airplane-test-uk-dangers org/content/111/24/8788.full.pdf drones-registrations-soaring-faa/ dsb/reports/2010s/DSBCyberSupplyChain_ExecSummary_ conference on HighSpeedAutomatic Calculating Machines, org/deeplinks/2013/08/letter edu/~niessner/papers/2016/1facetoface/thies2016face.pdf 2013. Relocating “MakingEthics Explicit: Ethics to the Core of 2016. “Verifiable ASICs,” Security andPrivacy (SP) 2016, U.S. Defense ScienceBoard, DSB Task Force on Cyber Szegedy, C., Zaremba, W., Sutskever, I.,Bruna, J., Erhan,D., Ahn, M., J.,Sunderland, Carson, C.,Kastenberg, and W. Supply Chain, 2017. “Report of the Defense ScienceBoard airplanes tohappens,” seewhat The Verge,2016, October18, and Lawyers Demand Congress Reform the CFAA and Support ReenactmentRGB Videos,”and of Proceedings ofComputer Cambridge University Press. abs/1312.6199 Goodfellow, I. and Fergus, R., 2013. “Intriguing properties PNAS. Vol 111, No. 24. 10779. July 22, 2014. http://www.pnas. Fortune, 6,2017,January http://fortune.com/2017/01/06/ Distribution_A.PDF Managing the Risks of Emerging Biological and Chemical Deployed Systems, and Lessons Learned. Cambridge: Engineering Education,” 2013 ASEE Annual Conference. Wahby, R., Howald, M., Garg, S., shelat, a., and Walfish, M. turingarchive.org/browse.php/b/8 pp.67-9, corrected version availablehttp://www. onlineat british-airways-plane/ http://www.telegraph.co.uk/news/2016/04/17/drone-hits- prepares to land at Heathrow,” The Telegraph, April18,2016, http://ieeexplore.ieee.org/abstract/document/7546534/ https://www.theverge.com/2016/10/18/13314916/drone- preprint server,https://arxiv.org/abs/1703.01161 networks for hierarchical reinforcement learning,” arXiv References Journal of Conflict Resolution, 61.9 (2017): 1970-1991. Terrorism”. Texas A&M University Law Review, vol. 2,635-673, Times, July 15,2017. Yin, T. 2015. “Game of Drones: Defending Against Drone Yao, A.C. 1982. “Protocols for secure computations,” Yampolskiy, R. 2017. “AI Is the Future of Cybersecurity, for Anyone Ever Finding Out,” New Republic, June 1, 2014, http://ieeexplore.ieee.org/stamp/stamp.164. files.wordpress.com/2017/04/facebook-and-information- Zubiaga, A., Aker,Zubiaga, A., Bontcheva, A., K.,Liakata, M., and Procter, Zittrain, J. 2014. “Facebook Could Decide an Election Without Zeitzoff, T. 2017. “How SocialMediaIs Changing Conflict,” ox.ac.uk operations-v1.pdf documents/os-cwatts-033017.pdf org/2017/05/ai-is-the-future-of-cybersecurity-for-better-and- 2017, https://www.intelligence.senate.gov/sites/default/files/ 23rd AnnualSymposium on Foundations of Computer University ofPress, Chicago Chicago. soon patrol parking lots, offices, andmalls,” Digital Trends, solution-facebook-digital-gerrymandering Science (sfcs 1982), Chicago, USA,pp. 160- IL, for-worse, May 8,2017. Oxford:Project ComputationalPropaganda, on comprop.oii. Operations and Facebook,” Facebook, https://fbnewsroomus. Cybersecurity, April27, 2017, https://www.rand.org/pubs/ abs/1704.00656 MeasuresInfluence and Campaigns,” statement prepared for Need for CognitiveSecurity,” testimonypresented before Media: ASurvey,” arXiv preprint server, https://arxiv.org/ R. 2017. “Detection and Resolution of Rumours inSocial Better and for Worse,” Harvard Business Review, https://hbr. British regulatory policy for genetic engineering, 1972-1982. jsp?tp=&arnumber=4568388&isnumber=4568364 Wu, T. 2017. “Please Prove You’re Not a Robot,” New York Worldwide: Executive Summary.” Working Paper2017.11. Woolley, S. and Howard, P. 2017. “Computational Propaganda Wiggers, K.2017. “Meet robotsthe 400-pound that will Weedon, J., Nuland, W., and Stamos, 2017. A. “Information Watts, C. 2017. “Disinformation: APrimerinRussian Active Waltzmann, R. 2017. “The Weaponization of Information: The Wright, S. 1994. Molecularpolitics: developing American and the U.S. Senate Select Committee on Intelligence, March 30, testimonies/CT473.html the Senate ArmedServices Committee, Subcommittee on https://www.digitaltrends.com/cool-tech/knightscope-robots- https://newrepublic.com/article/117878/information-fiduciary- https://ssrn.com/abstract=2944426 interview/ p.75 Details Workshop Appendix A: on cybersecurity, AI, and robotics from relevant experts in these On February 19, the event began with background presentations Event Structure The workshop was heldunder Chatham Houserules. backgrounds, andanalyzed a varietyrisks ofrelated misuse. AI to camefrom wide variety a institutional anddisciplinary of Leverhulme Centre for the Future of Intelligence (CFI). Participants Kingdom. The workshop was co-organizedbyFHI, CSER, and the “Bad Actor Risks Intelligence” inArtificial in Oxford, United (CSER) co-chairedExistentialworkshopRisk a Study of entitled AvinShahar Instituteand (FHI) Humanity of theCentre for the On February 19 and 20, 2017, Miles Brundage of the Future of Summary

p.76 Appendix A: WorkshopDetails report andcirculatedit among allofthe attendees workshop atthe Avinauthorstopic.the Brundage and on wrote et al. adraft ofthe at theworkshop, as wellprior and subsequentresearchas by the This document isbased inlarge part on notes from the discussions ReportProcess Writing focus thesubsequentreportprocess. writing votedmeasures on which useful seemed and tractable,in order to agreed upon the need for a research agenda to beproduced, and possible prevention and mitigation measures. The group present the workshopmet to nextdiscuss steps prioritization and the of 20,February On a subset participantsof the from thefirst day of possiblescenarios,then discussed defenses. and participantssessions: first discussed securitydomainsand underexploredrisks. The afternoon featured two sets breakoutof fields. A particular focus of thepresentations was onhighlighting TimHwang, Google Katja Hofmann, Microsoft Research UniversityRichardHarknett, ofOxford/University ofCincinnati Dylan Hadfield-Menell,UCBerkeley and Center Ulrike Franke, University of Oxford Carrick Flynn,Future of HumanityInstitute Ben Garfinkel, Future of HumanityInstitute Peter Eckersley, Electronic Frontier Foundation Eric Drexler, Future of HumanityInstitute Allan Dafoe, Yale University Rebecca Crootof, Yale Law School Owen Cotton-Barratt, Future of HumanityInstitute Guy Collyer, OrganizationReduction forBiorisk Global Jack Clark,OpenAI Joanna Bryson, University of University Bath/Princeton Center Miles Brundage, Future of HumanityInstitute Shahar Avin, Centre for the Study of Existential Risk Amanda Askell, Centre for Effective Altruism Armstrong,Stuart Future of HumanityInstitute Ross Anderson, University of Cambridge Amodei, OpenAI Dario List ofWorkshop Participants werenotto able capture perspectives. alloftheir workshopparticipants forinvaluable their contributions, evenif we wellas asadditionaldomainexperts.We are gratefulto allofthe for Human-compatible AI for Information Technology Policy

p.77 Appendix A: WorkshopDetails Yueh-HsuanWeng, Tohoku University Roman Yampolskiy, University of Louisville Andrew Trask,OxfordUniversityof Helen Toner, Open Philanthropy Project Jaan Tallinn, Centre forStudy Existentialof the Risk Eden Shochat, Aleph VC Paul Scharre, Center for a New American Security Heather Roff, University of Oxford/Arizona State University Michael Page, Centre for Effective Altruism Toby Ord, Future of HumanityInstitute hÉigeartaigh,Seán Ó Centre forStudy Existentialof the Risk Jan Leike, DeepMind/Future of HumanityInstitute Ben Laurie, DeepMind Victoria Krakovna, DeepMind/Future of Life Institute Eva Ignatuschtschenko, University of Oxford /New America Foundation

p.78 Research for Further Questions Appendix B: This appendixgives additionalcommentarytopicsrelated on to or conclusive. content as a jumping-off point for researchers interested inmaking In each more case, three high-levelwe whichoneor the flag of Landscape threat factors (introduced in the the progress in these areas; the below isnot intended to be exhaustive initial questions anddirectionsinvestigationfor oneachtopic. Interventions Recommendations ) the research) the area aims to address. Weinclude this section of the main report, section main ofalong with some the and Priority Research Areas GeneralImplications for the Threat

described in described 3 2 1 Shehadeh, 1999 Tucker, ed.2012;Harris,2016 Allen andChan,2017 p.79 Appendix B: QuestionsforFurther Research • • • • •

The apparentnaturetechnologiesraisesAI dual-use ofthe Allen and Chan Allen and Arrangement AI might be similar those that have beenhistorically applied to Dual UseAnalogies and Case Studies ensurepositive outcomes ends). Examples include chemicals potentially useful for chemical possible areaOne oftheory, practice,history be exploredto and of regulatingof cryptographicnetworksecurity algorithmsand cryptography. pathwe shouldavoid,be a maywellThis least orat for biologicalweapons,for cryptography,nucleartechnologies. and forinsightsis theset of technologiesprominent with concerns following questions: and hardlaws(e.g.and export controls) developed overmany years to and military aims (or, more generally, to both beneficial andharmful around dual use- technologies that can beused for both peaceful and precautionary lessons about poorregulation that should as wellharm, suggest thatas the default controlmeasures for we should learn bothwelearn constructive should solutionsfrompast successes, weapons orexplosives,potentiallyusefulbiological engineering When consulting the history ofthe governingWhen consulting technologies,use dual there is a rich tapestry of soft norms(e.g. pre-publication review) tools through export controlmeasures such as theWassenaar take verycautiously. termsrunning ongeneral-purposehardware, of termsbeing in of potential insights for AI dual-use policymaking. In these cases, be avoided.relevantA examplethe difficultieslatteris ofthe rangeprotectlegitimatetotheir ability ofin applications, and immaterialtermshaving objects(algorithms),in of a verywide What lessons can belearned from challenges and failures in Arethere exemplarywerewhich dual-useconcerns in cases if any,unique challenges, What pose asadual-use AI does fromnorms What other dual-usedomainsareAI? applicableto most appropriatethe is levelWhat of analysis andgovernance effectively addressed? of dual-usecharacteristicstechnologies(e.g.AI of field asa the applying controlmeasures to technologies? dual-use whole, individual algorithms, hardware,individual algorithms, whole, software, data)? technology? . The similarities between AI and cryptography, in exploredseveral ofthese casestudiestheir and . 4 3 2 1 though seee.g.Andersonetal.,2017 Knight, 2017 Papernot etal.,2016b Bass, 2017 p.80 Appendix B: QuestionsforFurther Research • •

“blue team”“blue responding to these attacks. These exercises explore There areseveral regarding“red openquestionsuse ofteam” the (with some limitationspreventto(withsome lasting damage),with anoptional A common tool in cybersecurity and military practice isred Red Teaming Origins ProjectOrigins this yearearlier Cyber Grand Challenge but acrossrangeGrand Challenge of wider Cyber aattacks(e.g. literature,this document,in andelsewherethe earlierin are Cyber GrandCyber Challenge. direction strategies for mitigating malicioususes of AI: In the caseIn of cyber attacks,many ofthe concernsdiscussed another recentanother effort aimedatthisgoalwas alsoconductedbythe and defense,highlightwe andadversariallearning.While machine practices.and Two subsets AI security ofdomain seem the attackssystemsthe against practices and ofthe organization aremany theoretical papers showing the vulnerabilities machine of Defenses competition Likewise,the casein ofthere adversariallearning,while machine what an actual attack might looklike in order to ultimately better they work inpractice. these subsetsbecause they seemrelevantespecially to security, “redteam”teaming- a composed ofsecurity experts and/or the CleverHansthe librarybenchmarks ofmodels are this stepin anda testing of real-world AI systems has only just begun useful in the AI/cybersecurityin the useful domain, analogous toDARPA the understandimprove and thesecurity of the organization’s systems requiredto carry outcertain attacks anddefenses,howwell and memory attacks), in order to better understand the skill levels hypothetical. Conducting deliberate red team exercises might be beneficial. Inbeneficial. addition to this report andthe associated workshop, red teaming AI technologiesofmore broadly seems generally toparticularly amenable such exercises: AI-enabled cyber offense members ofthe organization deliberatelyplans andcarries out red teaming effort, as is the NIPS2017 Adversarial Attacks and including social engineering,and vulnerabilityincluding exploitationbeyond learning systems to attack, thesystematic and ongoing stress- Is itpossible to detect most serious vulnerabilities through “red What lessons can belearned from the history to date of “red team” exercises,surfacethe areais for or attackbroad?too team” exercises? , creatingfoundationthe for adistributedsource open , which is moreis , which analogous toDARPA the . . Efforts like 3 2 1 Neema, 2017 e.g. Selsametal.,2017; Fisher, 2014 Turing, 1949;BaierandKatoen,2008 p.81 Appendix B: QuestionsforFurther Research

• • • • • • •

AI systems,AI required, analysisfurther is but the challenges about Formal Verification very complex systems are amenable to formalproofs that they or elements thereof, are amenable to formal verification. At the decades forAI verification,formal given thecomplexity modern some of a givensystem: researchand topicthe on continues apace Formal verification of software systems been studied has for workshop there was substantial skepticismprospects about the will operate compilerand CompCert the intended,including as the seL4 microkernelthe seL4 might beinterested in the following properties being verified for “redteam” exercises? What arethe challenges andopportunities of extending“red Are theremechanisms to sharelessons from“red team” Are theremechanisms topromote uptake the lessonsof from What sorts of skills arerequired AI systems, toundermine and responsiblebe for conductingsuch exercises,should Who and that its goals will be remainfacebe constantthe itsin of adversariesthat goalswill itsinternalthatprocesses in fact attain the goals specified for exercises with other organizationsbe susceptiblemay to that difficult to specify indifficult advance,to specify andtherefore difficult to verify), similar attacks?Howto avoid disclosure of attackmethodsto systems,how informand shouldthese findings thethreat model skills overlaprequiredskills the with to develop anddeployAI attempts to change them, what is the whatdistribution of those skills? To what extent do these the systemnoting the (though existence of thespecification testing exercises? teaming” (or related practices like tabletop exercises) to AI used inred teaming exercises (and other AI security analysis)? bad actors? how incentivisedcould they tobe do so? problem,i.e. that desiredpropertiessystemsAI of are often issues in the physical and political domains? What can be learned forphysical the domain fromphysical penetration . Inrecent years, ithasbeen shown that even some . An open question is whether AI systems, . Inparticular, we

4 3 2 1 et al.,2017 real worlduse—seee.g.Beurdouche though therearesomeinstancesof Katz etal.,2017 Blanchet, 2017 Harrison, 2010;Wahby,R.2016 p.82 Appendix B: QuestionsforFurther Research •

The notion of being able to provetobeing able notion ofThe thatsystem a behaves as As discussed above, despitesuccessesthe of contemporary Additionally, inrecent years formal verification hasbeen applied to Verifying AIFunctionality Verifying Security Verifying Hardware Responsible “AI Disclosure 0-Day” vulnerabilities, such as inducing misclassificationinducing vulnerabilities, viaadversarialsuch as verification someaspects of AI systems,of image suchas of the input spaceof the on verification of deepneural networks provided amethod classifiers, still is feasible even behaviorverification the of of development process. It should benoted that much of this work example of adeveloper-focusedtool that allowsprogrammers to certaintypes of attacks. The JavaScriptprover CryptoVerif domains limiteddomains theoretical foundations for their mayoperation, it increasingGiventhe complexitysystems,AI of some in and systems due to thestate space explosionproblem.Nonetheless, seems particularly amenable toparticularly amenable verification,seems methods as formal security protocolssecurity provideto robust guarantees ofsafety against formal methodsformal are up to arbitrary difficult scale to complex for them. However, itmay be feasible to use formal methods to apply formalmethods to their code to check correctnessin the ML algorithms also have vulnerabilities. These includeML-specific tofor check the existence of adversarial examplesregionsin the wholesystemprohibitivelyis complex.For example, work machine learning algorithms, ithasbeen shown time and again that been limited have been widely adopted in the hardware industry for decades be prohibitivelybe expensive, orevenpractically theoretically or intendedis anattractive intelligence.for However, one artificial largelyis still theoreticalreal and in the adoption worldhas sofar improve thesecurity of components of these systems.Hardware impossible,provideto anend-to-end verification framework that its ability to be deceived with adversarial inputs isbounded tosome extent. .

.

is an

.

3 2 1 e.g. AblonandBogart,2017 Laskov, 2014 Rubinstein etal.,2009;Šrndicand Carlini etal.,2016 al., 2016;Evtimovet2017; Szegedy etal.,2013;Papernot p.83 Appendix B: QuestionsforFurther Research • • • • • •

Thereis currently agreatinterest deal of amongcyber-security (2010)for survey. a remaintoML algorithmsalso open traditional vulnerabilities that have not beenmadepubliclyknown (and thus overflowmemory vulnerabilities,such as (Stevens et al., 2016). defenders have zero days to prepare for an attack makinguse of examples questionsresearch: for further such vulnerabilities responsibly to affected parties (such as those Should there be a normin the AI community for how to disclose In the cybersecurityIn community,“0-days” aresoftware applications)? broadThis question givestorise additional affected parties before publishing widely about them, in order to presentat theremore seemtobe questions than answers. who developedforthem the algorithms,orareusing commercial them). It is common practicethem).Itis common to disclose these vulnerabilities to researchersunderstandingin thesecurity ML systems,of though providepatchfor be developed. anopportunity a to What is the equivalent of “patching” for AI systems, and how If such a norm wereIf such a appropriatebroadin terms,be whoshould Which empirical findings inAI would beuseful ininforming whatsafety-criticalIn contextssystems areAI currently Should AI systems (both existing and future) bepresumed As AI technologies becomeincreasingly integrated into vulnerabilities privately isunnecessary? provenuntilvulnerable secure, to anextentnewthat disclosing changing attacks anddefenses? of possible defense measures be weighed in a world of rapidly discussed in cybersecuritydiscussed analyses should trade-offs (e.g. between resource demands, accuracy and robustness to noise) and prioritization amongst the variety and potentially acted upon? an appropriate disclosurepolicy(analogous to theway that and communities? aroundresponsible disclosure extendtechnologiesAI to be given before publication, and what mechanisms should notified in case a vulnerability is found, how muchnotice should being used? historical trends in 0-day discoverieshistorical0-day trendsin andexploitationrates are productsplatforms, and willthe existingnorm security institutions createtoprocessed ensurerecommendationis a or via poisoning the trainingthe poisoning data or via )? ; see Barrenosee et; al.

1 see e.g.Kesarwanietal.,2017 p.84 Appendix B: QuestionsforFurther Research • • • • • • •

AI-Specific ExploitBounties Toresponsiblenorm of disclosure complementthe of (cashbounties)to anyonewho detects responsibly and discloses vulnerabilities (discussed above), which relies on social incentives Security Tools development anddeployment,thattheysuch arelessto amenable componentssystems and integrated AI componentswith during capabilities(testing, fuzzing,anomalydetection, etc.), could specific vulnerabilities, somequestions arise: In thesameway software development and deployment tools attack? These could include: could attack? These a vulnerability in their products. With the emergence of new AI- and goodwill, somesoftware vendors offer incentives financial we start envisioning tools to testimprove and thesecurity AI of have evolvedincreasinginclude an to array ofsecurity-related Is there scopeto offerbountiesby third(e.g.parties weShould expect, orencourage,AI vendors to offerbounties Are existing vulnerability bounties likely to extend to AI Automatic suggestions for improving modelrobustness (see e.g. Automatic detection of attemptsremote at model extraction or Toolsfor analysing classification errors Automatic generation of adversarial data vendorstounable areunwilling offerfor or them, examplein government, NGO, or philanthropic source) in cases where open-sourceprojects in academia? or for AI-specific for exploits? Koh and Liang(2017) for related ideas) technologies? the case of popular machine learningframeworks machine popular the case of developed as remote vulnerabilityscanning 2 1 Anderson, 2008 U.S. DefenseScienceBoard,2017 p.85 Appendix B: QuestionsforFurther Research (such that itisprohibitively difficult to physically examine the (in general orof(in level) acertaintype orcapabilitytightly be could At theworkshop we exploredpotential the value of adding security Secure Hardware distribute securehardware forAI-specific applications it than could potentiallycould be valuable sothat outsiders areunable to cryptography, tamper-proof hardware hasbeen developed in the cybersecurityconcern context, where thereis fear that drive research in secure hardware more generally, not just for the engineering and, ultimately, be adopted by hardware providers. It Other desirable secure hardware features includehardware- havecould harmfulpolitical effects. economic, social or controlled,if the capabilities AI systemsof such harmful wouldbe createhardware secureAI that wouldprevent copying a trainedAI generic (commercial GPUs), butincreasingly, AI(and specifically fasterenabling execution offacilitating existingmodels, and more specialization could make itmuchmore feasible to develop and sabotage cyber-physicalsystems specific hardware, which could thenbeused to inform hardware fully specialized (e.g. Tensor Processing Units (TPUs)). This features such as tamper it evidenceclear(making that tampering features to AI-specifichardware. For example, itmay bepossible to actorswith control overintroducehardware-may supply chain a a featurebe desirablecould sothat thetotalnumber AI systemsof Finally,note that for other security-relevant domains such as Hardware innovation has accelerated the pace of innovation in workings of the chip in orderworkingsto ofthe chip defeatit). Tamper-proofhardware widely used. tobe would develop securehardware generic be toit andcause has occurred when it has occurred)layout andobscurity ofit has occurred designwhen based vulnerabilitiesin ordertomore surveil effectively or produce;hardware supply chain vulnerabilities are currently a manufacturers toundermine thesecurity hardwareof the they hardware runningAI systems, as a response measure to changes maybe the casepotentialalso that securitythreats fromAI will be considered is developing a reference model for secure AI- machine learning)systems machine hardware arerun on trained and that rapid iteration of possible models.In some cases, this hardware is machine learning, by allowing more complex models to be trained, model off a chip without the originalcopymodel offwithout achip first beingdeleted. Such in the cyber threatlandscape.Note, however, potential the for systemsAI large-scale diffusionofsuch if a wronghands, or the in is semi-specialized (e.g. graphics processing units (GPUs)) or levelrestrictions access research andaudits.One trajectory to . , with 2 1 as suggestedbyStoicaetal.,2017 Anderson, 2008 p.86 Appendix B: QuestionsforFurther Research • • • • • • • •

Therethis domain: aremanyin openquestions Pre-Publication RiskAssessment in Technical Areas ofSpecial Concern developed for AIpurposes. systemworkingsAI inner of an the discern from external emission; considerations are sufficiently widespread in computer security secureprocessors tend to cost significantly more insecure than safe and beneficial way by hard-coding certain software properties thatstolenhardwareso toused duplicateso AI; and be cannot an By pre-publication risk assessment we mean analyzing the that organizations can crediblyto commit operatingsystem a in a to what extent, to publishit.Suchnorms are already widespread processors rather thanfullyworking exploits are Indeed, such oftenpublished. becamewidely available, basiswhether, that anddecidingon and particular risks (or lack thereof) of a particular capability ifit in a if tamperedchip that, with,wouldbreak down.However, in the computerin security community, where e.g.proofs of concept Would changes in the risklandscape (as surveyed above) if any,What, are security requirementsspecific the AI of Can secure processors bemade affordable, or could policy Howsecure could AI context enclavesimplementedin an be Could/should AI-specific secure processors be developed? How applicable are existingsecureprocessor designsto the measures,Whatif any, are availableto ensure compliance What set of measures (e.g. reference implementation) would ofpremium? acost distribution of vendorsincentives andcompeting such ascost, encourage adoption ofsecurehardware? security? systems,in differentin general and domainsof application? with hardware safety requirements given the international provide incentivesufficient major for a overhaul hardware of mechanisms be devisedmechanisms toincentivizeuse even their in theface protection AI systemsof from tampering? potentialimplications offorlegal auditability? surveillance and and, to our knowledge, have not specifically been

? 3 2 1 on nextpage Lyrebird, 2017 see e.g.NDSS,2018 p.87 Appendix B: QuestionsforFurther Research (as inbiotechnology, for example), this may not be a significant AsRethinking inthe described Openness section report,of the vendorsnegotiatefortoperiod oftime adiscovered a vulnerability credit for their work. Some AI-relatedcreditSome forwork. their discoveries,the casein of as concern. If, however, restricted publication becomes common, as on legitimateon applications. of malicioususe, but also slows down research and places barriers computationalrequirements there arefor another actorrecreateto Generallyless speaking, the one shares,higher theskill and the part of(or acrucialdataingestionbe componentthat could Openness isnot a binary variable: today, many groups will conferences considered. If the number of restricted publications is very small publicationon would affectbenefits these be should carefully is the hand, caseotheras with other technologies withsignificant given target speaker(asreportedly will be available soon as a specifying the hyperparameters the specifying to ittoget work effectively, or security research, for example in white hat penetration testing. section) and disinformationpolitical securitysection).(see Onthe fromservice the companyLyrebird action byactionimagine coordinating vendors.AI, onecanthe case In of a givenlevel of capabilityis shared: withwhatreduces this risk the applications, likeapplications, automated(see digitalsecurity spearphishing For anexamplepotentially of a abusablecapabilitywhere full while allowing the researcherswhile allowingthe retainto priority claimsandgain will revealwill detailsresearch ofnot givebut detailsparticular onone tothe commercialinterests of vendorsneeds of security the and to bepatched before the vulnerability ispublished.In addition institutions wouldneedtobe then developed tobalance the tutorials/tips on it togetting workpractice, wellin there are various transformation)pipeline. Onthespectrum fromrough idea,a to that they arehighlighted as criteria for submission toprestigious there arebenefits level clear the to of openness currently prevalent users, suchschemes oftenprotect also researchers fromlegal measures,means ofsecure or deployment,be developed, can in cybersecuritymechanisms researchers allow andaffected needs of allaffectedForparties. example, responsible disclosure possible points, and perhaps there are multiple axes (see pseudocode,sourcetowith modelalong trained codeand a learningwithout algorithm sourcethe machine publish codeof a potential for malicioususe, there could be value in openness for be publicationmaydeemed toorisky, voice synthesis for a institutions that will withhold publicationuntilinstitutions that willwithhold appropriate safety the casein of vulnerability disclosurein cybersecurity research, in machine learningfield. asa machine in The extent which restrictionsto . ) isripe for potential criminal Figure 3 ). ). p.88 Appendix B: QuestionsforFurther Research Decreasingrequirement skill Vague description of achievement Figure 3.Aschematicillustrationoftherelationshipbetween openness aboutanAIcapabilityandtheskillrequired to reproducethatcapability. Pseudocode Increasing openness trained models, Source code, tutorials

p.89 Appendix B: QuestionsforFurther Research • • • • • • •

“Responsible Disclosure”. AI0-day Another potentialAnother model foruse ofopennessis the what we call Central Access LicensingModels explicitly prohibit malicious use, allowing clear legalrecourse. certain capabilitiesinacentrallocation,such asacollection of usersmodel, aretothis able In central access licensing. access Some valuable questions for futureresearch related topre- adversarial examples in the wild, may be subsumed under existing analysis and imagerecognition, can placelimits on the malicious industry forAI-basedadopted services such in as sentiment the speed of use can beimposed, potentially preventing some to the use of the capabilities. This model, which isincreasingly use of the underlyingAI technologies. For example, limitations on notshared, termsis and andconditionsapply underlying code publication research assessment include: responsiblein disclosurewebelowmechanisms, as discuss remotely accessiblesecure, interlinked data centers,the while large-scale harmfullarge-scale applications, terms and andconditions can Whatbe mayheuristics sort appropriateof for weighing the possibleit toIs say,in advance high confidence,and with what What can belearned from pre-publication risk assessment What sorts of pre-publication research assessment would AI How does the community adopt such a model in the absencein of model How doesthecommunityadoptsuch a Can we say anything fine-grained yet generalizable aboutthe Howincorporated cansuchassessment be into decision- capabilitiesfrom (code, a givenpseudocode,type etc.) of shared information? sorts of capabilitiesfor areripe abuse? normsaroundseen asconflicting with openness? such analysis into publications)? regulation? mechanisms in otherscientific/technological mechanisms domains? researchersbe willingto consider? To what extent wouldbe this making (e.g.making informing one’s opennessincorporating choices, or potentially-abusableprosup andconsof capabilities? opening levels ofskill andcomputationalresources required recreateto 4 3 2 1 see e.g.Ghodsietal.,2017 Gu etal.,2017 Dwork andNaor,1993 Bastani etal.,2017 p.90 Appendix B: QuestionsforFurther Research • • • • • •

(though see Laurie and Clayton, 2004Clayton,Laurie and for acriticismofthis see (though (though also from well-intentionedresearchers). Note thoughthat Additionally,toproposalssimilarly early for, in effect, an Centralisedprovides access analternativepublicationtothat of powerof biggest to threatbe the fromAI technologies;note, organizations,byincluding those blessingacting withthe of the of organizations may heighten potential for malicioususe at those Some initial researchinitial Some related questionsthat arise to acentral spam approach,Liu and Camp,2006and for further discussion; the allowsuniversal accessto acertain capability,keepingthe while access licensingmodel: as the introduction of “backdoors” into machinelearning systems behavior,addition tomonopolistic there aremore risks subtle such Finally,note that the concentrationparticular setin a AI services of workshop attendees consideredrisksthese from concentration the underlying technology.the that users may beunaware of underlying technologicalbreakthroughsunderlying awayfrombad actors malicious userisks, rather than systemic threats (seeScope). In however,reportin this wehave that decided to focus on direct relevant organizationthreats.byinsider well as some as Indeed, bitcoin may lead to advances in this area). large-scaleuse AI services, suchthatplaced of on the attacksbe black box model extraction increasedinterest in crypto-economics following thesuccess of informationprocessing“tax” in order to onemails disincentivize like automated spear phishing could bemadeless economical Is thereIs enoughoftechnologythat a such actors gap without might objectto Who acentralisedwhatmodel and on access proposalIs the technologically,politicallylegally feasible?and How can a user determine whether a service providerHowuser determine canservice whetheris a a How effectivelyprovider canservice a determineAI whether mightwant one Whatsorts ofservices onlyavailable ona grounds? access cannot developtechnologiesindependently? the uses aremalicious? malicious per-use basis? , centralized AI infrastructures, centralizedAI better enableconstraintsto ? maybad allow actors to gainaccessto .

1 Shokri etal.,2016 p.91 Appendix B: QuestionsforFurther Research • • • • • •

Sharing Regimes that Favor Safety and Security even larger number of “eyes” are on the problem), while reducing meetorganizations certainthat criteria,individuals such as or one might imagine an arrangement where some particularly cyber domain —Information Sharing and Analysis Centers (ISACs) One possible approach for reducing security risks of AIis to Such an approach might be valuableSuch anapproachformight facilitating collaborative straightforwardlyto automatedhacking) areshared with only pointsserve as of concentrationknowledgesharing, giving of selectivelyshare certaininformation capability anddatawith fractionbenefit sourceof anopen ofthe approach (where an analysis ofsafety issues,thus gettingsome and security and mutually agreedhasoversightagencythatthe grouphas and and Information Sharing and Analysis Organizations (ISAOs) — where companiesshareinformation aboutcyber attacks amongst them advantages overkinds of actors. other AI, the caseIn of largethemselves.Antivirustech and companies themselves trusted parties. A somewhat analogous approach isusedin the random inspection bymembersinspectionrandom other ofthe group, third-party ora having establishedsafety routines,security and oragreeingto powerfulhazardous or capabilities(e.g. themselveslend onesthat inspection powersinspection overthem. (extracting thetraining dataFredrikson frommodel; the (How) can cloud providers(How) can cloud vetsafetythe AI security of or What wouldbethe associated trade-offs limits?of such Are providerscloud computing sufficiently flexible in their How usefulHow arelimitsthe amountfrequency on or of queries How effective can blackbox model extraction bein What arepotentialrisks anddownsidesto centralised access, et al., 2015)et al., andotherforms ofmembership attacksuch as different contexts? e.g. in aggravating political security risks? or would this intervention bemost applicable to preventing systemsinternalinspecting their without workings,ifsuch services to allow the experimentation required by researchers, to models asacountermeasuretoinversion model against potentiallyharmful dissemination trainedAI systems? information isprivate? in thetraining data inference(ascertaining whether certain datais contained )?

1 Burton etal.,201 p.92 Appendix B: QuestionsforFurther Research • • • • • • • •

There has recently been discussion of the rolerecentlybeen discussionThere ofthe ofhas ethics education for Future Develope Security, capabilities would bepublished. Several questionsthe above ariseabout proposals: riskssome associatedIf,with diffusion. basedsuch analysis,it on and fosteringpreparedness tomake decisions about when access licensing model. Note that this mechanismhaspartial overlap with pre-publication potential and pitfalls of AI.Educational efforts might bebeneficial risk assessmenttechnicalin areas ofspecial concern andcentral is concluded that there is no harm in further in further diffusion, thenthe harm is concluded thatno thereis in highlighting the risks of malicious applications to AIresearchers, in AI Should there be a limited-sharingstage AIdevelopments,therebe a forShould all What types informationof be sharedmight amongst such tobe applied Whatsorts anorganization ofmight criteria or What have been the benefits andlimitations of existing ISACs What arepotentialrisks anddownsidestotype of this Areincentivesthere be createdparticular anywhich can be Howestablished between cansufficient trust groups such informationWhat limited types should sharing apply to: code, of researchof on)? or should capabilities be evaluated individually, and if the latter sharing regime? a group? ISAOsand AI? andareusefulto elementsmodels ofsuch that would make this sort of collaboration more likely (for that this kind of coordination is seen as mutuallybeneficial? basis? whatthen on particularly sensitiveinformation? research papers, informal notes? individual in orderindividual to ascertain theirtrustworthiness to deal with instance,the creation ofshared a clusterto test kind acertain , in light of ongoing public and privatepublic and light of discussion ongoing ofthe in , Ethics, and SocialImpact Education rs

3 2 1 Future ofLifeInstitute,2017 IEEE StandardsAssociation,2017 Sunderland etal.,2013 p.93 Appendix B: QuestionsforFurther Research • • • • • •

Anotherway multi- of actingbe onethical concernscould Ethics Statements and Standards career development andeventualsuggesting decision-making, impactsthe on ofsuch educationalresearchers’ effortsAI on order to mitigate such risks. As yet there isnolong-term research deploying AI systems.deployingAI Two examplesprocesses ofsuch are development anddeploymentsystems,AI of be which could signed on to bytosigned on research companies, organizations andothers stakeholder conversation to develop ethical standards for the Intelligence and Autonomous Systems technologies should be open, and how they should be designed, in the Asilomar AIPrinciples the IEEE Global Initiative for Ethical Considerations inArtificial possible areasresearch:for What ought to be included in such acurriculum:ethicalin included Whatbe oughtto How canmost ethicstobe designedsoaseffectively education practicesWhatbest are forthe policy ethics and educationfor What institutional frameworksinstitutional What are appropriatefor ensuring most effectivebe wouldWho providing at such acurriculum? iteratedbe Howsuch acurriculum could overstatethe time as occursin other domains intereststhe with engage AI developers, andconcernsof rather especially around mitigating security risks? developed standardsinclude statements reporting about owninternal capacity to teachAI specific ethics? security advances?AI and of science and engineering in generalthat arescience andengineering AI, applicableto Should ethics educatorsfromphilosophy andother disciplines and accountability? than being seen as merelyboxtickedbe toburdenseen asa being off ora than technological ‘greenwash’? For instance, should community thatstatements standards and concerningethics arefully unrelatedto one’spractical sometimes decision-making,as be brought in or isitbetter for the community to develop its principlesmethodologies, and/or theories? implementedin order to ensure that they aremoremere than . Several. remain questions open: ? and the development and of

p.94 Appendix B: QuestionsforFurther Research • • • • • • •

As noted in previousAsnotedin sections, there are substantialrisks security Norms, Framings and SocialIncentives cases AI can beused to enhance rather than diminish security. This exploitingrisks. such At thesametime, there are also substantial associated with AI, and in some casesin oneactorassociatedAI, and from couldgain with upsides to progress inAIresearch and development, and inmany raises the likequestions the following: AI development, AI safety and riskmanagement play inboth What processesWhat are appropriaterevisingfor updating ethics and What roleWhat dodiversenormative cultures across suchas fields processesWhatbe allowed should to governthe emergence What arelearned, analogouscaseslessonsbe from canwhich upsidesAI developmentHowway ofsuch a canthe in framedbe Are standards and statementsbest waykind the of this to foster researchShould companies and organizationshave statement a flexible and can incorporate best practice whilst retaining their consider themselves‘outsiders’ tobe morepeople consider debatesthatAI andensuring about creativity withinthesector? effective norms, and to avoid this normative culture beingused orderbothto ensure the creation ofstrong, enforceable and harmfuldiscourage exploitation? on ethics,taken either directlyfrom oneofthese communal sense of permanence andobjectivity.sense of statements and standardsin order to ensure that theyremain some alternatives? be encouraged?howso, canthis standards or developed inhouse for their particular situation? If allowing for a diverse range of perspectives to inform public and implementation of a normative culture for beneficial AIin about in a zero-sum manner was governed in a way that toas beneficial galvanize developmentsmutually focus on and wherethoughthavetechnologyused and a that could been themselves to be‘insiders’ in such debates, and fewer people to preserve rigid and/or biasednorms that hamper diversity and benefited all? industry-wide conversationWhatthe ethicsAI? are about of 3 2 1 leakage on theprobabilityofinformation 2006) referringtostrongguarantees a conceptfirstdevelopedin(Dwork, Shokri etal.,2016 Fredrikson etal.,2015 p.95 Appendix B: QuestionsforFurther Research • • • •

Technologically GuaranteedPrivacy Ji et al. (2014) surveyed methods for providing differential privacy differentialprivacy-guaranteeing securemulti- algorithmsand effectsperformance.Generally, on differentiallyprivate algorithms differentially private machine learning algorithms combine their differential privacy inneural networks. Suchmethods have security domains (e.g. automated spear phishing,personalised Several of thethreats withinthe digitalpolitical security and a model in such a waymodel informationa that fromunderlying data the set accesstotraining the data, anattackersome casesin query can attack Many machinelearning models are currently being developed Differential privacy We highlight two technologies as potentially relevant here: training data with noise to maintain privacy while minimizing to break anonymity in the underlying dataset of a machinelearning both technologies: party computation. Thereremainregarding openquestions privacy,the contextin systems.AI be applicable may of also which researchtechnological on tools for guaranteeinguser data propaganda)rely onattackers gainingaccessprivateto been reported by, for example, Abadi et al. (2016). In general, model that hasbeen deployed for publicuse via a modelinversion (see centralbyAPIs access companiesin for commercialuse information about individuals. In addition to proceduraltoIn addition information individuals. and about in machinelearning systems is revealed. legal measures to ensure individuals’ privacy, there isincreasing licensing above). Without precautions itispossible for individuals Can algorithmic privacy be combined with AI technologies, What lessons can belearned by efforts at technologically legalor (financial, educational, other) could mechanisms What What areif any,trade-offs,the implementingfor algorithmic viability ofservices? either in general or in specific domains? in in general or either guaranteed privacy (such as Apple’s use of differential privacy)? systems?encourageAI privacythe adoption ofin algorithmic privacy,termsin e.g.terms offinancial performancein or of or membershipinference or attack , though theynotthough address, do . That is, evenwithout is, . That

5 4 3 2 1 Garfinkel, forthcoming Dowlin etal.,2016;Trask,2017; e.g. Rouhanietal.(2017) OpenMined, 2017 Lindell andPinkas,2009 Yao, 1982 p.96 Appendix B: QuestionsforFurther Research To the extentAI systems that play activeroles surveillance, for As an important practical application, MPCprotocols make it At the same time, the use of MPC protocols remains limited by the An activesource open developmentis (OpenMined) effort of a vote,individual votes withoutsharingtheir with one another. party’seach input toprivate thefunction classifications. one company may develop machinelearning models that can make currentlyto aiming developplatformto a users allowto sell others simple MPCprotocol allows users to jointly compute the outcome Secure multi-party computation (MPC)refers to protocols that Secure ComputationMulti-Party still able toprovide its service. this companysend personalmedical data, copies oftheir theymay fact that, in many cases,facttheyin increasethat, can overhead associated In addition, MPC opens upnew opportunities for privacy- allow multipleparties to jointly compute functions, while keeping and so privacy may become a concern if the teams developing MPC could also help to enable privacy-preservingtohelp enable MPC couldalso surveillance MPC without significantly compromising its privacy the (often sensitive) data that isbeingused to make the relevant to operatesystemssuch toneeding collect oraccesswithout to increase individualprivacy. Inparticular, MPCmakes itpossible processthis in doesthecompanypoint gainanyAtno the output. right tothe learning systems trainmachine on their datausing users’toneeding accesssome cases,this data. data,in without possessesA technology them. companylearn fromcould similarly records by engaging in an MPCprotocol with the hospital that researchersmedical train could system a onconfidential patient possible tolearning systems trainmachine on sensitive data models are not incentivized to keep their datasets private. knowledge about the individual’s medical data; nevertheless, itis predictions based on health data. Ifindividuals do not want to preservingweb applications andcloudcomputation.For example, havemachine-learning also beenproposed individuals on the basisweb oftheir activity,the individuals on used be MPC can instancebyrecognizing facesin videos or flaggingsuspicious in particular an MPCprotocol where only the individualreceives protocolinsteadMPC the company,with in an optto engage and lose some performancelose some compared tonon-private their equivalents, . Anumber of other frameworks for privacy-preserving . For instance, one . . For example,.

. 2 1 Hilbert andLopez,2011 Hilbert andLopez,2011 p.97 Appendix B: QuestionsforFurther Research • • • • •

AI development community.However, broaderthereis a space

Exploring Legal and Regulatory Interventions Monitoring Resources ongoing public effortongoing to do so,best withthe availableinformation can be carried out by researchers and practitioners within the enforceable limitations on how hardware could beused. Questions capabilities. Additionally, having such monitoring inplace would distribution of computing resources could be to better understand of other potentially dangerous technologies, most notably the One type of measure that might help to predict and/or prevent Such monitoring regimes are well-established in the context for considerationfurther include: for the purpose of implementing nuclear andchemicalweaponimplementing purpose of for the agreements. An obvious example of an input that might bepossible Much ofthe discussion abovefocusesinterventions on that with acomputationbymultiple orders magnitude. of means This the likelythe distribution of offensive anddefensiveAI/cybersecurity to monitor is computing hardware. While efforts have beenmade that MPCisbest-suited for relatively simple computations or for use cases where increased privacy would be especially valuable. be valuableif strongermeasures were tobe employed, e.g. havingbenefit public, semi-public, or to a database theglobalof monitoring materials production offissile andchemical facilities misuse AI technologyof wouldbetomonitorinputs toAI systems. in the past toin the surveyresources computing likely withheld due to commercial or state secrecy.possible One Would other AIinputs bebetter suited to monitoring than Arethere drawbacksto(e.g. such aneffort in encouraging information?such with be done What could Are differentmorelesstractable domains or monitor,to more or How feasiblebe tomonitor wouldit global computing video game consoles be considered, in light of their largelightshare oftheir be considered,in video gameconsoles computing resources? or less important for AI capabilities, than others (e.g. should wasteful “racing” to have the most computing power)? resources? in totalin computing butlimited current role inAI)? , there isnomajor 2 1 see e.g.Farquharetal.,2017 Calo, 2011;Cooper,2013 p.98 Appendix B: QuestionsforFurther Research • • • • • • • • •

could be counterproductive, and that itisimportant that the considered. We note that ill-considered government interventions of possible interventions, includinglegal ones, that should be proper scopefor governmentintervention AI security arise; wein be carefullynumber of questionsthe A concerning analyzed. implications policyinterventions of any thisareaspecific in should list some initial examples here: How suitable would existingbe playingatinstitutions this Which government departments,marketplace actors orother thereIs responsibility aclear chainofpreventing for security-AI Are datapoisoning andadversarial example attacks aimed at Whatpros arethe andconsof governmentrequiringpolicies Should governmentshold developers, corporations, or preparedHow does e.g.USgovernmentthe much howfeel, and Are liabilityregimes adequate? Do they provide the right Arerelevant actorsspeakingto each other, and coordinating field? disrupting AI systemsdisrupting penalties subject tolegal as thesame defenses against adversarial examples andotherforms of externalities explicitlymake them exempt fromliability such others liable for the malicioususe of AI technologies (or, communities be? sufficiently, especially across legal,cultural political, and appetite would therebe for focused offices/channels designed approaches might be considered for pricingAI security-related what wouldinteractions the withthe academicindustry and traditional forms of hacking? Ifnot, should they be(and how the use of privacy-preserving machinelearning systems or increaseto awareness andexpertise? malicious use? measures? related problems? role, and how much will itrequire the establishment of new incentivesfor various actorsto take competent defensive institutions would ideally have what responsibilities, and in orderto effectively operatesuch anevolvingin technical and institutions founded on novel principles or innovative structures linguistic barriers? ? )? What)? other 1 Wu, 2017 p.99 Appendix B: QuestionsforFurther Research • • • • •

Achieving this islikely to require both a high degree of trust AI community,intervenewellto asseeking its as behalf. own on AI community should beinforming legal and policy interventions, contrast, different sectorsresponding reactively to the different community and those within other institutions, includingpolicy Ideally discussionsAI safety about and securityfrom withinthe Itunlikely seems interventions that AI development within the a suitablea to channel facilitate proactive collaborationin developing policylegal and willingnessbe a there amongst and should also institutions, willworklegal welland overlong termunless the Recommendations #1 and #2. technical communities alike. These considerationsmotivated our some degreethereis of coordinationbetweenthese groups. kinds pressuresof that they each face at different times seems norms, ethics educationstandards, and laws;in policies and betweenthe different groupsinvolved the governancein AI and of institutions to devolveresponsibility some forAIsafety to the likely to result in clumsy, ineffective responses from the policy and “Blade Runnerlaw” What kind of process can beused when developing policies What should the AI security community’s “publicpolicy model” How desirable is it that community norms, ethicalstandards,that it community How desirableis Should thererequirementbe a fornon-humansystems internationalShould agreementsbe consideredtools as to example, over thetelephone) toidentify themselves (a as such operatinginteracting(for onlineorotherwise humans with if so)?dealt with can legalbutrelated tactics like search engine optimization be safety /bad actor and highuncertainty /low uncertainty risks)? and lawsand to govern adynamically evolvingunpredictable and and governments? what shouldthescope ofpolicy thatbe, how and should to different kinds of risk(e.g. near term,term/long technical responsibilitybe distributed acrossindividuals, organizations, howis, shouldwe- that be aim to affect governmentpolicy, public policieslaws public and all sayhowmuch thesamething and research anddevelopment environment? incentivizesecurity? collaborationAI on is to be gained frombe gained tois different levels of governancerespondto ) toincrease political security? The Malicious Use of Artificial Intelligence: February 2018 Forecasting, Prevention, and Mitigation

Future of Humanity Institute

University of Oxford

Centre for the Study of Existential Risk

University of Cambridge

Center for a New American Security

Electronic Frontier Foundation

OpenAI The Malicious Use of Artificial Intelligence p.101