CEPS Task Force Report Artificial Intelligence and Cybersecurity Technology, Governance and Policy Challenges R a p p o r t e u r s L o r e n z o P u p i l l o S t e f a n o F a n t i n A f o n s o F e r r e i r a C a r o l i n a P o l i t o Artificial Intelligence and Cybersecurity Technology, Governance and Policy Challenges Final Report of a CEPS Task Force Rapporteurs: Lorenzo Pupillo Stefano Fantin Afonso Ferreira Carolina Polito Centre for European Policy Studies (CEPS) Brussels May 2021 The Centre for European Policy Studies (CEPS) is an independent policy research institute based in Brussels. Its mission is to produce sound analytical research leading to constructive solutions to the challenges facing Europe today. Lorenzo Pupillo is CEPS Associate Senior Research Fellow and Head of the Cybersecurity@CEPS Initiative. Stefano Fantin is Legal Researcher at Center for IT and IP Law, KU Leuven. Afonso Ferreira is Directeur of Research at CNRS. Carolina Polito is CEPS Research Assistant at GRID unit, Cybersecurity@CEPS Initiative. ISBN 978-94-6138-785-1 © Copyright 2021, CEPS All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means – electronic, mechanical, photocopying, recording or otherwise – without the prior permission of the Centre for European Policy Studies. CEPS Place du Congrès 1, B-1000 Brussels Tel: 32 (0) 2 229.39.11 e-mail: [email protected] internet: www.ceps.eu Contents Preface ............................................................................................................................................................ 1 Executive Summary ........................................................................................................................................ 2 Policy recommendations from the Task Force ......................................................................................... 3 AI for cybersecurity .................................................................................................................................... 3 Cybersecurity for AI ................................................................................................................................... 4 PART I. INTRODUCTION .................................................................................................................................. 7 1. Introduction ........................................................................................................................................... 8 2. Where does Europe stand on the AI and cybersecurity interplay discussion? .................................. 9 3. Some definitions.................................................................................................................................. 11 4. AI for cybersecurity and cybersecurity for AI .................................................................................... 12 PART II. ARTIFICIAL INTELLIGENCE IN CYBERSECURITY .............................................................................. 14 1. Introduction ......................................................................................................................................... 15 2. AI systems’ support to cybersecurity ................................................................................................. 15 2.1 System robustness .......................................................................................................................... 17 2.2 System resilience ............................................................................................................................ 19 2.3 System response ............................................................................................................................. 20 2.4 Major techniques in the use of AI for system robustness, resilience, and response ................. 21 3. AI malicious uses ................................................................................................................................. 26 3.1 Expansion of existing threats ......................................................................................................... 27 3.1.1 Characteristics of AI-powered attacks .................................................................................. 29 3.2 Introduction of new threats ........................................................................................................... 30 3.2.1 Deepfakes ............................................................................................................................... 30 3.2.2 Breaking CAPTCHAs ............................................................................................................... 35 3.2.3 Swarming attacks .................................................................................................................. 36 3.3 Changes to the typical character of threats and new forms of vulnerabilities on AI systems ... 36 4. Ethical considerations related to AI in cybersecurity ........................................................................ 40 5. Asymmetries in the interplay of AI and cybersecurity ...................................................................... 42 5.1 Asymmetry of cognition ................................................................................................................. 42 5.2 Asymmetry in AI ethical standards development ......................................................................... 43 5.3 Offence/defence asymmetry ......................................................................................................... 43 6. Trustworthy versus reliable AI ............................................................................................................ 44 7. Cybersecurity risks associated with anthropomorphising AI ............................................................ 47 7.1 Deanthropomorphising and demystifying AI ................................................................................ 49 8. Weaponisation and the offence versus defence debate .................................................................. 50 PART III. CYBERSECURITY FOR ARTIFICIAL INTELLIGENCE .......................................................................... 55 1. Introduction ......................................................................................................................................... 56 2. Machine learning systems do indeed have a larger attack surface ................................................. 58 3. A high-level view of the threat landscape.......................................................................................... 59 3.1 Input attacks ................................................................................................................................... 59 3.2 Poisoning attacks ............................................................................................................................ 61 4. An AI threat model .............................................................................................................................. 62 4.1 Role of human operators ............................................................................................................... 64 5. Safety and security of open, autonomous, AI-based IT infrastructure, and its runtime evolution .... 65 6. Addressing the insecurity of the network as it relates to AI ............................................................. 69 7. An example of a secure development life cycle for AI systems ........................................................ 70 PART IV. POLICY ISSUES AND RECOMMENDATIONS .................................................................................. 76 1. Introduction ......................................................................................................................................... 77 2. Current and future AI laws: accountability, auditability, and regulatory enforcement .................. 77 3. Existing legal frameworks: EU cybersecurity ..................................................................................... 79 4. Major policy issues .............................................................................................................................. 81 4.1 Delegation of control ...................................................................................................................... 81 4.2 Openness of research ..................................................................................................................... 82 4.3 Risk-assessment policies and suitability testing ............................................................................ 85 4.4 Oversight ......................................................................................................................................... 87 4.5 Privacy and data governance ........................................................................................................ 88 4.5.1 Application of GDPR in securing AI and in using AI for cybersecurity ................................. 89 5. Develop and deploy reliable AI ........................................................................................................... 95 6. The role of AI standards activity and cybersecurity .......................................................................... 96 7. Additional policy issues ....................................................................................................................