Future of University Centre for University of Center for a Electronic OpenAI Humanity of Oxford the Study of Cambridge New American Frontier Institute Existential Security Foundation Risk The Malicious Use February 2018 of Artificial Intelligence: Forecasting, Prevention, and Mitigation The Malicious Use of Artificial Intelligence: February 2018 Forecasting, Prevention, and Mitigation Miles Brundage Shahar Avin Jack Clark Helen Toner Peter Eckersley Ben Garfinkel Allan Dafoe Paul Scharre Thomas Zeitzoff Bobby Filar Hyrum Anderson Heather Roff Gregory C. Allen Jacob Steinhardt Carrick Flynn Seán Ó hÉigeartaigh Simon Beard Haydn Belfield Sebastian Farquhar Clare Lyle Rebecca Crootof Owain Evans Michael Page Joanna Bryson Roman Yampolskiy Dario Amodei 1 Corresponding author 9 American University 18 Centre for the Study of miles.brundage@philosophy. Existential Risk, University ox.ac.uk of Cambridge Future of Humanity Institute, 10 Endgame University of Oxford; Arizona State University 19 Future of Humanity 11 Endgame Institute, University of 2 Corresponding author, Oxford [email protected] University of Oxford/ Centre for the Study of 12 Arizona State University/New Existential Risk, University 20 Future of Humanity America Foundation of Cambridge Institute, University of Oxford 13 Center for a New American 3 OpenAI Security 21 Information Society Project, Yale University Open Philanthropy Project 4 14 Stanford University 22 Future of Humanity Electronic Frontier Institute, University of 5 Future of Humanity Foundation 15 Oxford Institute, University of Oxford 6 Future of Humanity 23 OpenAI Institute, University of Centre for the Study Oxford 16 of Existential Risk and 24 University of Bath Centre for the Future of 7 Future of Humanity Intelligence, University of Institute, University of Cambridge 25 University of Louisville Oxford; Yale University 17 Centre for the Study of 26 OpenAI 8 Center for a New American Existential Risk, University Security of Cambridge Authors are listed Design Direction in order of contribution by Sankalp Bhatnagar and Talia Cotton Executive Summary Artificial intelligence and machine learning capabilities are growing at an unprecedented rate. These technologies have many widely beneficial applications, ranging from machine translation to medical image analysis. Countless more such applications are being developed and can be expected over the long term. Less attention has historically been paid to the ways in which artificial intelligence can be used maliciously. This report surveys the landscape of potential security threats from malicious uses of artificial intelligence technologies, and proposes ways to better forecast, prevent, and mitigate these threats. We analyze, but do not conclusively resolve, the question of what the long-term equilibrium between attackers and defenders will be. We focus instead on what sorts of attacks we are p.3 likely to see soon if adequate defenses are not developed. In response to the changing threat landscape we make four high-level recommendations: 1. Policymakers should collaborate closely with technical researchers to investigate, prevent, and mitigate potential malicious uses of AI. 2. Researchers and engineers in artificial intelligence should take the dual-use nature of their work seriously, allowing misuse- related considerations to influence research priorities and norms, and proactively reaching out to relevant actors when harmful applications are foreseeable. 3. Best practices should be identified in research areas with more The Malicious Use of Artificial Intelligence mature methods for addressing dual-use concerns, such as computer security, and imported where applicable to the case of AI. 4. Actively seek to expand the range of stakeholders and domain experts involved in discussions of these challenges. Executive Summary p.4 As AI capabilities become more powerful and widespread, we expect the growing use of AI systems to lead to the following changes in the landscape of threats: • Expansion of existing threats. The costs of attacks may be lowered by the scalable use of AI systems to complete tasks that would ordinarily require human labor, intelligence and expertise. A natural effect would be to expand the set of actors who can carry out particular attacks, the rate at which they can carry out these attacks, and the set of potential targets. • Introduction of new threats. New attacks may arise through the use of AI systems to complete tasks that would be otherwise impractical for humans. In addition, malicious actors may The Malicious Use of Artificial Intelligence exploit the vulnerabilities of AI systems deployed by defenders. • Change to the typical character of threats. We believe there is reason to expect attacks enabled by the growing use of AI to be especially effective, finely targeted, difficult to attribute, and likely to exploit vulnerabilities in AI systems. Executive Summary p.5 We structure our analysis by separately considering three security domains, and illustrate possible changes to threats within these domains through representative examples: • Digital security. The use of AI to automate tasks involved in carrying out cyberattacks will alleviate the existing tradeoff between the scale and efficacy of attacks. This may expand the threat associated with labor-intensive cyberattacks (such as spear phishing). We also expect novel attacks that exploit human vulnerabilities (e.g. through the use of speech synthesis for impersonation), existing software vulnerabilities (e.g. through automated hacking), or the vulnerabilities of AI systems (e.g. through adversarial examples and data poisoning). The Malicious Use of Artificial Intelligence • Physical security. The use of AI to automate tasks involved in carrying out attacks with drones and other physical systems (e.g. through the deployment of autonomous weapons systems) may expand the threats associated with these attacks. We also expect novel attacks that subvert cyber- physical systems (e.g. causing autonomous vehicles to crash) or involve physical systems that it would be infeasible to direct remotely (e.g. a swarm of thousands of micro-drones). • Political security. The use of AI to automate tasks involved in surveillance (e.g. analysing mass-collected data), persuasion (e.g. creating targeted propaganda), and deception (e.g. manipulating videos) may expand threats associated with privacy invasion and social manipulation. We also expect novel attacks that take advantage of an improved capacity to analyse human behaviors, moods, and beliefs on the basis of available Executive Summary data. These concerns are most significant in the context of authoritarian states, but may also undermine the ability of democracies to sustain truthful public debates. p.6 In addition to the high-level recommendations listed above, we also propose the exploration of several open questions and potential interventions within four priority research areas: • Learning from and with the cybersecurity community. At the intersection of cybersecurity and AI attacks, we highlight the need to explore and potentially implement red teaming, formal verification, responsible disclosure of AI vulnerabilities, security tools, and secure hardware. • Exploring different openness models. As the dual-use nature of AI and ML becomes apparent, we highlight the need to reimagine norms and institutions around the openness of research, starting with pre-publication risk assessment in The Malicious Use of Artificial Intelligence technical areas of special concern, central access licensing models, sharing regimes that favor safety and security, and other lessons from other dual-use technologies. • Promoting a culture of responsibility. AI researchers and the organisations that employ them are in a unique position to shape the security landscape of the AI-enabled world. We highlight the importance of education, ethical statements and standards, framings, norms, and expectations. • Developing technological and policy solutions. In addition to the above, we survey a range of promising technologies, as well as policy interventions, that could help build a safer future with AI. High-level areas for further research include privacy protection, coordinated use of AI for public-good security, monitoring of AI-relevant resources, and other legislative and regulatory responses. Executive Summary The proposed interventions require attention and action not just from AI researchers and companies but also from legislators, civil servants, regulators, security researchers and educators. The challenge is daunting and the stakes are high. p.7 Contents Executive Summary 3 Introduction 9 Scope 10 Related Literature 10 General Framework 02 for AI and Security Threats 12 AI Capabilities 12 Security-Relevant Properties of AI 16 General Implications 18 Scenarios 23 Digital Security 24 Physical Security 27 Political Security 28 Security Domains 30 03 Digital Security 31 Physical Security 37 Political Security 43 Interventions 50 04 Recommendations 51 Priority Areas for Further Research 52 Strategic Analysis 58 05 Factors Affecting the Equilibrium of AI and Security 59 Overall Assessment 61 Conclusion 64 Acknowledgements 66 References 67 Appendix A 75 Appendix B 78 Introduction AI refers to the use of digital 1 technology to create systems that are capable of performing tasks commonly thought to require intelligence. Machine learning is variously characterized as either a sub- Artificial intelligence (AI) and machine learning (ML) have field of AI or a separate