Use cases: Impermissible AI and fundamental rights breaches This briefing has been compiled to assist policymakers in the context of the EU’s regulation on artificial intelligence. It outlines several cases studies across Europe where artificial intelligence is being used in a way that compromises EU law and fundamental rights, and therefore requires a legal prohibition or ban. ugust 2"!" #rave concerns remain as to how artificial intelligence $ I% will impact people& communities and society as a whole. 'ome I systems have the ability to exacerbate surveillance and intrusion into our personal lives& reflect and reinforce some of the deepest societal inequalities& fundamentally alter the delivery of public and essential services& vastly undermine vital data protection legislation& suppress freedoms of expression and assembly& and disrupt the democratic process itself. In European )igital *ights’ explainer& E)*i details some of the implications of I on fundamental rights. In E)*i’s recommendations for a fundamental rights+based regulation on artificial intelligence& E)*i outlines the need for clear legal limits on the uses of AI& legal criteria for& democratic oversight& and the need for a prohibition on impermissable uses of AI E)*i recommends the European Commission draw red+lines for AI, in particular in these areas- • Indiscriminate biometric surveillance and biometric capture and processing in public spaces& including public facial recognition. • use of AI to determine access to or delivery of essential public services !such as social security, policing, migration control"# • uses of I which purport to identify& analyse and assess emotion& mood& behaviour& and sensitive identity traits (such as race& disability% in the delivery of essential services. • predictive policing# • use of AI systems at the border or in testing on marginalised groups, such as undocumented migrants; • autonomous lethal weapons and other uses which identify targets for lethal force $such as law and immigration enforcement%. European )igital *ights / 0! *ue 1elliard& 0"2" 1ruxelles& 1elgium / Tel. 34! ! !54 !6 5" / www.edri.org • general purpose scoring of citi7ens or residents& otherwise referred to as unitary scoring or mass+scale citi7en scoring. 8urther information on E)*i’s recommendation to the ,ommission on fundamental rights+based regulation on artificial intelligence- https-99edri.org/wp+content/uploads9!"!"9":9 I_E)*i*ecommendations.pdf <ore information on biometric processing and capture in publicly accessible spaces can be found in E)*i’s position paper on 1iometric <ass 'urveillance& where E)*i calls for an immediate and indefinite ban on biometric mass surveillance. European )igital *ights / 0! *ue 1elliard& 0"2" 1ruxelles& 1elgium / Tel. 34! ! !54 !6 5" / www.edri.org Table of Contents I. Use of AI to determine access to or delivery of essential public services.................................................4 )E=< *>- Gladaxe system...................................................................................................................................................4 U=ITE) KI=#)?<- Harm Assessment Risk Tool (A@ RT’%.......................................................................................2 )E=< *>- TvBrspor research proCect.............................................................................................................................2 =ETHE*D =)'- SyRI Case.....................................................................................................................................................2 E?D =)- Random Allocation of Cases.............................................................................................................................6 'E IN- B?',? and Bono Social de Electricidad..........................................................................................................: #E*< =F- S,@U8 system...................................................................................................................................................: U'TRIA- Employment Agency A<'..................................................................................................................................: II. Uses of AI to identify& analyse and assess emotion& mood and sensitive identity traits in the delivery of essential services................................................................................................................................................5 8* =,E- Behaviour prediction in Marseille...................................................................................................................5 8* =,E- Emotion recognition programs (several%....................................................................................................G 8I=D =)- Digital<inds.............................................................................................................................................................G #E*< =F- Affective Computing.........................................................................................................................................G IT DF- Redditometro...................................................................................................................................................................H III. Predictive policing...............................................................................................................................................................0" =ETHE*D =)'- Pro>id 1!+'I...............................................................................................................................................0" )E=< *>- Gladaxe system..................................................................................................................................................0" IT DF- Key,rime..........................................................................................................................................................................0" 'IITZE*D =)- Precobs& Dyrias and RO'.......................................................................................................................00 U=ITE) KI=#)?< Gangs Violence Matrix.....................................................................................................................00 =ETHE*D =)'- Crime Anticipation System................................................................................................................0! IT DF- Video surveillance and the prediction of ‘abnormal behaviour’............................................................04 U=ITE) KI=#)?<- Offender Group Reconviction Scale........................................................................................04 1ED#IU<- Zennevallei.............................................................................................................................................................02 1ED#IU<- iEolice.......................................................................................................................................................................02 U=ITE) KI=#)?<- National Data Analytics Solution (=) '%.............................................................................06 U=ITE) KI=#)?<- Origins Software...............................................................................................................................06 IK. Use of AI systems at the border& in migration control or in testing on marginalised groups........0: EU*?EE- Common Identity Repository (,IR) and Visa Information System (KI'%.....................................0: EU*?EE- Military Drones at borders................................................................................................................................0: EU*?EE- Frontex scales up AI uses in border control.............................................................................................05 @U=# *F& GREE,E- i1order,trl.........................................................................................................................................05 U=ITE) KI=#)?<- U> Home Office Visa Algorithms..............................................................................................0G 'D?KE=IA- B?*)ER AI............................................................................................................................................................0H #REE,E- SEIRIT ProCect.........................................................................................................................................................!" K. Biometric Surveillance......................................................................................................................................................!" IT DF- S RI facial recognition.............................................................................................................................................!" #REE,E- ‘'mart’ policing.......................................................................................................................................................!0 European )igital *ights / 0! *ue 1elliard& 0"2" 1ruxelles& 1elgium / Tel. 34! ! !54 !6 5" / www.edri.org U>- Covert facial recognition..............................................................................................................................................!! 'E*1IA& U# =) - Government surveillance..............................................................................................................!!