European network of legal experts in gender equality and non-discrimination Algorithmic discrimination in Europe Challenges and opportunities for gender equality and non-discrimination law Including summaries in English, French and German Justice and Consumers EUROPEAN COMMISSION Directorate-General for Justice and Consumers Directorate D — Equality and Union citizenship Unit D.1 Non-discrimination and Roma coordination Unit D.2 Gender Equality European Commission B-1049 Brussels EUROPEAN COMMISSION Algorithmic discrimination in Europe: Challenges and opportunities for gender equality and non-discrimination law A special report Authors Janneke Gerards (Utrecht University) and Raphaële Xenidis (University of Edinburgh and University of Copenhagen) 2020 Directorate-General for Justice and Consumers 2021 The authors would like to warmly thank the whole coordination team of the European Network of Legal Experts in gender equality and non-discrimination at Utrecht University and in particular Birte Böök, Franka van Hoof, Alexandra Timmer and Linda Senden, the executive team at Human European Consultancy, in particular Yvonne van Leeuwen-Lohde and Marcel Zwamborn, the 31 national experts in gender equality law and the national experts in non-discrimination law who collaborated on this report for their invaluable help, as well as Florianne Peters van Neijenhof and Pita Klaassen for their excellent research support. The drafting of this report was concluded in October 2020. Europe Direct is a service to help you find answers to your questions about the European Union. Freephone number (*): 00 800 6 7 8 9 10 11 (*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you). LEGAL NOTICE This document has been prepared for the European Commission; however, it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. More information on the European Union is available on the Internet (http://www.europa.eu). Luxembourg: Publications Office of the European Union, 2021 ISBN 978-92-76-20746-7 doi:10.2838/544956 Catalogue number DS-02-20-549-EN-N © European Union, 2021 Contents EXECUTIVE SUMMARY 7 RÉSUMÉ 13 ZUSAMMENFASSUNG 20 GENERAL INTRODUCTION 27 Subject, context and scope of the report 27 Methodology 29 Structure 29 1 WHAT IS ALGORITHMIC DISCRIMINATION AND WHAT IS NEW ABOUT IT? 31 1.1 Introduction 31 1.2 Types of algorithms 32 1.2.1 Rule-based algorithms 32 1.2.2 Machine-learning algorithms 33 1.2.3 Deep learning 36 1.2.4 Enabling technologies and combining algorithms: AI 36 1.3 Stages of algorithmic decision making and uses of algorithms 37 1.3.1 Planning stage 38 1.3.2 Development stage 39 1.3.3 Decision-making and use stage 39 1.4 Algorithmic characteristics and challenges 40 1.4.1 The human factor and the stereotyping and cognitive bias challenge 41 1.4.2 The data challenge 42 1.4.3 The correlation and proxies challenge 44 1.4.4 The transparency and explainability challenge 45 1.4.5 The scale and speed challenge 46 1.4.6 The responsibility challenge 46 1.5 Terminology and interactions between gender equality and non-discrimination law and data protection law 47 1.5.1 Terminology: ‘bias’ and ‘fairness’ versus ‘discrimination’ and ‘equality’ 47 1.5.2 Interactions between non-discrimination law and data protection 48 1.6 Conclusion 50 2 CHALLENGES TO THE EU GENDER EQUALITY AND NON-DISCRIMINATION LEGAL FRAMEWORK 53 2.1 The scope of EU gender equality and non-discrimination law in light of the problem of algorithmic discrimination 53 2.1.1 The legal framework 54 2.1.2 Equal pay, employment and self-employment 55 2.1.3 Goods and services: problematic gaps in the material scope 58 2.2 Protected grounds and algorithmic discrimination 62 2.2.1 Algorithmic gender-based classification 62 2.2.2 Correlations and proxies 63 2.2.3 New forms and grounds of discrimination 64 2.2.4 The CJEU’s interpretation of the grounds listed in Article 21 of the Charter of Fundamental Rights 65 2.2.5 Algorithmic granularity and intersectionality 65 2.2.6 The dynamic nature of algorithmic categorisations 66 2.3 The types of discrimination defined in EU law 67 2.3.1 Direct discrimination: an uneasy fit with algorithmic discrimination 67 3 2.3.2 Indirect discrimination: a better conceptual fit with a wide pool of potential justifications 70 2.4 Questions of proof, responsibility and liability 73 2.5 Conclusion 75 3 CHALLENGES FOR THE EUROPEAN STATES IN RELATION TO ALGORITHMIC DISCRIMINATION 78 3.1 Examples of the use of algorithms in European countries 78 3.1.1 Introduction 78 3.1.2 Examples of the use of algorithms in the public sector 78 3.1.3 Examples of use of algorithms in the private sector 83 3.2 Problems related to algorithmic decision-making 85 3.2.1 Biases in data 85 3.2.2 Discriminatory effects 86 3.2.3 Transparency problems and lack of information 87 3.2.4 Detecting algorithmic discrimination 87 3.2.5 Responsibility issues 88 3.2.6 A gender digital gap in European countries 88 3.3 Awareness of risks of algorithmic discrimination in European countries 90 3.3.1 Public discussions on the impact of algorithms on gender equality and non-discrimination 90 3.3.2 Scientific discussions on the impact of algorithms on gender equality and non-discrimination 98 3.4 Legal responses to algorithmic discrimination in the European countries 107 3.4.1 Legislative instruments 107 3.4.2 (Semi-)judicial application and enforcement of legislation 113 3.5 Conclusion 117 4 ENFORCING ALGORITHMIC EQUALITY: SOLUTIONS AND OPPORTUNITIES FOR GENDER EQUALITY AND NON-DISCRIMINATION 121 4.1 Introduction 121 4.2 Benefits and opportunities of algorithmic decision making 121 4.3 Tackling algorithmic discrimination: a review of national good practice in European countries 124 4.3.1 Monitoring algorithmic discrimination: examples of good practices and opportunities 124 4.3.2 Addressing algorithmic discrimination: examples of good practices and opportunities 125 4.3.3 The diversity question in relevant professional and educational communities 133 4.4 Potential solutions and tools to prevent and remedy algorithmic discrimination: a tridimensional approach 140 4.4.1 Introduction 140 4.4.2 Legal solutions 141 4.4.3 Knowledge-based solutions 146 4.4.4 Technology-based solutions 148 4.5 Conclusion: PROTECT – proposal for an integrated approach to algorithmic discrimination 150 GENERAL CONCLUSIONS 152 BIBLIOGRAPHY 153 ANNEX – QUESTIONNAIRE ALGORITHMIC DISCRIMINATION IN EUROPE: CHALLENGES AND OPPORTUNITIES FOR GENDER EQUALITY AND NON-DISCRIMINATION LAW 183 4 Members of the European network of legal experts in gender equality and non-discrimination Management team General coordinator Marcel Zwamborn Human European Consultancy Specialist coordinator Linda Senden Utrecht University gender equality law Content coordinator Alexandra Timmer Utrecht University gender equality law Specialist coordinator Isabelle Chopin Migration Policy Group non-discrimination law Project managers Ivette Groenendijk Human European Consultancy Yvonne van Leeuwen-Lohde Content managers Franka van Hoof Utrecht University gender equality law Birte Böök Content manager Catharina Germaine Migration Policy Group non-discrimination law Senior experts Senior expert on gender equality law Susanne Burri Senior expert on age Elaine Dewhurst Senior expert on sexual orientation/trans/intersex people Peter Dunne Senior expert on racial or ethnic origin Lilla Farkas Senior expert on EU and human rights law Christopher McCrudden Senior expert on social security Frans Pennings Senior expert on religion or belief Isabelle Rorive Senior expert on EU law, CJEU case law, sex, gender identity Christa Tobler and gender expression in relation to trans and intersex people Senior expert on disability Lisa Waddington 5 National experts Non-discrimination Gender Albania Irma Baraku Entela Baci Austria Dieter Schindlauer Martina Thomasberger Belgium Emmanuelle Bribosia Nathalie Wuiame Bulgaria Margarita Ilieva Genoveva Tisheva Croatia Ines Bojić Adrijana Martinović Cyprus Corina Demetriou Vera Pavlou Czechia Jakub Tomšej Kristina Koldinská Denmark Pia Justesen Natalie Videbaek Munkholm Estonia Vadim Poleshchuk Anu Laas Finland Rainer Hiltunen Kevät Nousiainen France Sophie Latraverse Marie Mercat-Bruns Germany Matthias Mahlmann Ulrike Lembke Greece Athanasios Theodoridis Panagiota Petroglou Hungary András Kádár Lídia Hermina Balogh Iceland Gudrun D. Gudmundsdottir Herdís Thorgeirsdóttir Ireland Judy Walsh Frances Meenan Italy Chiara Favilli Simonetta Renga Latvia Anhelita Kamenska Kristīne Dupate Liechtenstein Patricia Hornich Nicole Mathé Lithuania Birutė Sabatauskaitė Tomas Davulis Luxembourg Tania Hoffmann Nicole Kerschen Malta Tonio Ellul Romina Bartolo Montenegro Maja Kostić-Mandić Vesna Simovic-Zvicer Netherlands Karin de Vries Marlies Vegter North Macedonia Biljana Kotevska Biljana Kotevska Norway Lene Løvdal Marte Bauge Poland Łukasz Bojarski Eleonora Zielinska and Anna Cybulko Portugal Dulce Lopes and Joana Vicente Maria do Rosário Palma Ramalho Romania Romanița Iordache lustina Ionescu Serbia Ivana Krstić Davinic Ivana Krstić Davinic Slovakia Vanda Durbáková Zuzana Magurová Slovenia NeŽa KogovŠek Šalamon Tanja Koderman Sever Spain Lorenzo Cachón Dolores Morondo Taramundi Sweden Paul Lappalainen Jenny Julén Votinius Turkey Ulaş Karan Kadriye Bakirci United Kingdom Lucy Vickers Rachel Horton 6 Executive summary In recent years, media stories and scholarly articles about algorithmic discrimination