European Data Protection Supervisor

Issue 1 | July 2019

Ever since Alan Turing published his paper Computing Machinery and Intelligence in 1950, computer scientists have tried to get machines to mimic human behaviour and make them as intelligent or as smart as human beings, by having them play imitation games.

Turing raised the question: Can machines think? He suggested that something “resembling thinking” could be achieved if we provide the machine with the best sense organs that money can buy, and then teach it to understand and speak English. This is the main reason why we call modern machines with some imitation capacity smart devices.

Today, a new generation of speaking devices interact with us in human-like ways to execute simple tasks and answer questions, and not only in English. How is this possible?

I. What is a ? What is a virtual ?

A smart speaker is a speaker with and provides a spoken response. a built-in microphone that allows users to interact with other smart Users say a wake-up expression devices or internet services using to activate the , their voice. such as Alexa, or OK . Unless turned off by users, smart speaker The brain that makes the smart microphones are switched on speaker smart is the virtual continuously in order to detect assistant. A virtual assistant is a wake-up expressions. To indicate software service that takes the voice to users when the smart speaker of the user as its input, identifies is recording voice data, the device a command or question, interacts switches on a light. when necessary with other services

TechDispatch Issue 1 | July 2019 1 The current generation of smart speakers typically have built-in functionalities to search for information on the internet, play music, make phone calls and manage lists. They may also allow for new functionalities and extensions, such as controlling a smart door lock, by installing third-party developed software, through Alexa’s skills or Google Home’s actions, for example.

Most popular smart speaker and virtual assistants

Vendor Device Virtual Assistant

Amazon Echo, Echo plus and Dot Alexa Google Google Home Alibaba AliGenie XL Genie AI

Baidu Raven H and R, Xiaodu Zijia DuerOS Xiaomi Mi AI speaker Xiao AI Apple Homepod

There are other smart speaker vendors, but all their devices rely on the virtual assistants listed above, or on ’s .

The number of smart speakers sold in the USA until January 2019 was 66.4 million. The smart speaker market is also growing in China, Europe and worldwide. Most smart speaker owners have more than one speaker and install them in different rooms in their home.

TechDispatch Issue 1 | July 2019 2 3 II. What are the data protection issues?

Until now, user interaction with most technology was mainly visual and tactile. Smart speakers and virtual assistants use voice as the main means of interaction. In our preliminary understanding, this poses several data protection challenges to most of the existing smart speakers and virtual assistants.

Lack of transparency Lack of an appropriate consent Repurposing of data management mechanism Ensuring that all users are By analysing data collected clearly informed about Smart speakers process the via smart speakers, it is the data controllers and data of voices recorded by possible to know or infer user processors involved in the their microphones, whether interests, schedules, driving processing, the types of data these voices are one of routes or habits. All this processed – such as the user’s the intended users or not. data could be used, without voice, location and usage There is no mechanism yet people knowing, for profiling history - or the purposes of to prevent a smart speaker and to provide unsolicited the processing, is very difficult. from processing the data of a personalised services, such as For example, once a smart specific individual. advertising or modified search speaker is up and running, results. it is unlikely that users other When it comes to processing than the person who installed children’s data, there seems the device will have read the to be no way of ensuring written notices that came with that the person with parental Security of personal data the device. responsibility has provided Current smart speaker and their consent. Parental virtual assistant designs do Without transparency, controls are available to a not offer yet proper access personal data collected by certain degree but in their control to personal data. smart speakers could be current form they are not user Authentication by voice misused for purposes that go friendly. More alarmingly, recognition is optional and far beyond user expectations. there are already reports that does not allow for control of For example, newly patented voice recordings of children access to personal data by all technologies aim to infer are being used to create services. If it is not using voice health status and emotional voiceprints that could be recognition, access control states from a user’ and used to identify them when relies on an optional PIN adapt the services provided detected by other devices in number that a user has to say. accordingly. other locations. It is possible to manipulate a smart speaker remotely Excessive data retention Processing without consent via signal broadcasting and compromise the speaker It is not easy for users of via radio or TV. Most smart Smart speakers can mistakenly smart speakers to find out speakers blindly trust detect a spoken expression as how long their data will be their local networks. Any their wake-up expression and stored. Users are generally left compromised device in the therefore process personal with the task of deleting any same network could change data without user consent. data the virtual assistant has the settings of the smart processed and kept. In some speaker or install malware on it The wake-up expression can cases, it might not even be without the user’s knowledge be changed and an individual possible to delete all of the or agreement. who is not aware of this data stored. Depending on the change could accidentally virtual assistant, the deletion switch on the smart speaker. function might only partially delete the data, deleting only voice data, for example.

TechDispatch Issue 1 | July 2019 3 Potential for mass surveillance Data protection by design and by default

If working as currently As is evident from the previous examples, the current designed, smart speakers design and behaviour of most smart speakers and virtual locally record for a few assistants does not fully comply with the principles of data seconds and do not send any protection by design and data protection by default. When information to the speech developing, designing, selecting and using applications, recognition cloud service services and products that are based on the processing until the wake-up expression of personal data or process personal data to fulfil their is detected. However, smart task, producers of the products, services and applications speakers could be the target should take into account the right to data protection from of attacks and access requests the outset. from law enforcement agencies. They could access The design of a smart speaker should address data local voice recordings before protection issues, in particular to make sure users are they are deleted, turning informed about the processing of their personal data, to smart speakers into a enable them to manage their consent and to demonstrate massive surveillance system accountability so that users are fully aware of what happens installed by the victims of this to their personal data. surveillance.

TechDispatch Issue 1 | July 2019 4 5 III. References and related reading

Adam Wright, ‘Worldwide Smart Home Device Forecast • Worldwide Smart Home D evice F orecast Update, 2018– Update,2022: 2018–2022: CY 4Q18 CY 4Q18’ [2018]

Amazon• IEEE CWebomputer Services magazine Inc. ‘Alexa vol. 5 Privacy0 issue 9and (Sept Data 2017) Handling Hyunji Overview’ [2018] Can I Trust You? Hyunji Chung, Michaela Iorga, Jeffrey Voas and Sangjin Lee, ‘Alexa,• Symantec. Can I Trust Candid You? ’ Wu[2017]eest ( Nov 2017): A guide t o the security of voice-activated smart speakers. Candid Wueest (Symantec) ‘A guide to the security of voice- activated• Veton smartKëpus speakerska and G‘ [2017]amal B ohouta (Jan 2018): Next- generation of virtual personal assistants.

Veton• 2018 Këpuska I EEE 8 thand A nnualGamal C omputingBohouta, ‘andNext-generation C ommunication of virtualWo personalrkshop a ndassistants C onference’ [2018] (CCWC). Hyunji Chung and Sangjin Lee (Feb 2018): Intelligent Virtual Assistant knows Hyunjiyour Chung life. and Sangjin Lee, ‘Intelligent Virtual Assistant knows your life’ [2018] • Wake forest journal of business and intellectual property law. WhitneyAlexa, L. t ransmitHosey, c ‘lientAlexa, data transmit to amazon: client ethical data considerations to amazon: ethicalfor attorneysconsiderations looking foforrw attorneysard to virtual looking assistant forwards. to virtual assistants’ [2018] • 2018 29th Irish Signals and Systems Conference (ISSC) (Jun 2018). Eoghan Furey and Juanita Blue: She Knows Too Much Eoghan– Vo iceFurey Command and Juanita Devices Blue, and Pr ‘Sheivac y.Knows Too Much – Voice Command Devices and Privacy’ [2018] • 2018 IEEE Global Communications Conference (GLOBECOM) Xuejing(Dec Yuan2018) Xetuejing al, ‘All Yuan Your et al:Alexa All Y ourAre A lexaBelong Are Btoelong Us: toA RemoteUs: A Voice Remote Control Voice AttackControl against Attack against Echo’ [2018]Echo.

Martin Courtney, ‘Careless talk costs privacy’ [2017]

TechDispatchTTechFlechFlashash Issue Issue 1 |1 March | July 2019 5 This publication is a brief report produced by the Information Technology Policy Unit of the European Data Protection Supervisor (EDPS). It aims to provide a factual description of an emerging technology and discuss its possible impacts on privacy and the protection of personal data. The contents of this publication do not imply a policy position of the EDPS.

Author of this issue: Xabier Lareo Editor: Thomas Zerdick Contact: [email protected]

HTML ISBN 978-92-9242-423-7 ISSN 2599-932X https://data.europa.eu/doi/10.2804/004275 QT-AD-19-001-EN-Q PDF ISBN 978-92-9242-424-4 ISSN 2599-932X https://data.europa.eu/doi/10.2804/755512 QT-AD-19-001-EN-N

© European Union, 2019. Except otherwise noted, the reuse of this document is authorised under a Creative Commons Attribution 4.0 International License (CC BY 4.0). This means that reuse is allowed provided appropriate credit is given and any changes made are indicated. For any use or reproduction of photos or other material that is not owned by the European Union, permission must be sought directly from the copyright holders.

To subscribe or unsubscribe to this publication, please send an email to [email protected]

For more information on how the EDPS processes your personal data, please refer to our data protection notice.

TechDispatch Issue 1 | July 2019