CHI 2020 Paper CHI 2020, April 25–30, 2020, Honolulu, HI, USA Empathy Is All You Need: How a Conversational Agent Should Respond to Verbal Abuse Hyojin Chin Lebogang Wame Molefi Mun Yong Yi Graduate School of Graduate School of Graduate School of Knowledge Service Knowledge Service Knowledge Service Engineering, KAIST Engineering, KAIST Engineering, KAIST Daejeon, Republic of Korea Daejeon, Republic of Korea Daejeon, Republic of Korea [email protected] lebo.molefi@kaist.ac.kr [email protected] ABSTRACT term "conversational agent" as "the form of emergent dialogue With the popularity of AI-infused systems, conversational system" increasingly embedded in personal technologies and agents (CAs) are becoming essential in diverse areas, offer- devices. When using the term conversational agent, people ing new functionality and convenience, but simultaneously, might refer to a chatbot, virtual assistant, interface agent, em- suffering misuse and verbal abuse. We examine whether con- bodied conversational agent, or avatar [6, 41]. On the other versational agents’ response styles under varying abuse types hand, Intelligent Personal Assistant (IPA) refers to commercial influence those emotions found to mitigate peoples’ aggressive services such as Siri, Cortana, Alexa, Google Assistant, and behaviors, involving three verbal abuse types (Insult, Threat, Bixby, and the main functions of them are to retrieve informa- Swearing) and three response styles (Avoidance, Empathy, tion such as weather or music, and send notifications about Counterattacking). Ninety-eight participants were assigned schedules such as meeting schedules for the day [33]. These to one of the abuse type conditions, interacted with the three types of artificial intelligent (AI) solutions are widely available spoken (voice-based) CAs in turn, and reported their feelings on various devices including smartphones, wearable devices, about guiltiness, anger, and shame after each session. The and smart speakers. There are 57.8 million smart speaker users results show that the agent’s response style has a significant in the U.S. market and worldwide spending on smart speakers effect on user emotions. Participants were less angry and more is forecast to be nearly $3.52 billion in 2021 [1, 3]. guilty with the empathy agent than the other two agents. Fur- CAs offer new functionality and convenience, but at the same thermore, we investigated the current status of commercial time, they continuously fall victim to verbal abuse from their CAs’ responses to verbal abuse. Our study findings have direct users [10, 74]. Empirical studies indicate that 10∼44% of inter- implications for the design of conversational agents. actions with CAs reflect abusive language, including sexually- explicit expressions [11, 71]. The abuse of CAs by humans Author Keywords is currently not considered a serious problem because AI sys- Conversational Agent; Virtual Assistant; Intelligent Personal tems are not thought to be capable of feeling emotionally hurt Assistant; Smart Speaker; Verbal Abuse; Agent Abuse or offended like humans when verbally abused [10]. Verbal abuse of agents is considered pervasive in both text-based and CCS Concepts voice-based CAs [10, 11, 21, 71]. However, there is growing •Computing methodologies ! Intelligent agents; •Human- evidence that, if not mitigated, this type of behavior directed centered computing ! Personal digital assistants; User stud- towards a system can transfer to real-life social relationships. ies; Laboratory experiments; By allowing users to verbally abuse CAs without restraint, abusers’ actions can unintentionally be reinforced as normal INTRODUCTION or acceptable. For example, prior research showed that reg- Conversational Agents (CAs) such as chatbots and smart ular involvement with simulated violence such as abuse of speakers are integrated into everyday life and are widely used robot does in fact desensitize users to violent activities in real- in education, business, and public-service, often assuming hu- life [73]. The prevalence of cyberbullying was found to over- man roles such as tutors and secretaries [19, 24, 74]. There are lap with real-life bullying [55]. There have also been reports various ways of naming those machines that people can ‘talk about children learning bad manners from smart speakers [21]. to’, including conversational agents and intelligent personal as- Therefore, verbal abuse of conversational agents by their users sistants [50]. In their study, Luger and Sellen [41] defined the should be discouraged and effectively handled. Permission to make digital or hard copies of all or part of this work for personal or The issue of verbal abuse of conversational agents by their classroom use is granted without fee provided that copies are not made or distributed users has been addressed in the field of Human-Computer Inter- for profit or commercial advantage and that copies bear this notice and the full citation action only from limited perspectives. Veletsianos et al. [71] in- on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or vestigated the discourses between a female CA and 59 teenage republish, to post on servers or to redistribute to lists, requires prior specific permission students to find that users readily misuse and abuse the agent and/or a fee. Request permissions from [email protected]. while regarding the agent subordinate and inferior. De Angeli, CHI ’20, April 25–30, 2020, Honolulu, HI, USA. Copyright is held by the owner/author(s). Publication rights licensed to ACM. and Brahnam [22] also discovered that verbal abuse towards ACM ISBN 978-1-4503-6708-0/20/04 ...$15.00. http://dx.doi.org/10.1145/3313831.3376461 Paper 334 Page 1 CHI 2020 Paper CHI 2020, April 25–30, 2020, Honolulu, HI, USA a chatbot was pervasive through an analysis of conversation on the things that were done onto others rather than self [69]. log data between 146 users and Jabberwacky, a chatbot that In short, shame focuses on the global self; guilt emphasizes won the 2005 Loebner prize. In their research, Brahnam [10] specific behaviors [64]. examined how the agent responded to users’ verbal abuse and sexual harassment. However, as far as we know, there is no ex- Grasmick and Bursik [31] measured direct effects of present isting research on the response style of a conversational agent perception of self-imposed shame on inclinations to violate with the aim of mitigating verbal abuse by users. Moreover, the law to find that, for the three offenses of tax cheating, theft, most prior research examined text-based interactions [10, 11, and drunk driving, shame had a strong deterrent effect. The 18, 22]. The outcomes of human and computer interaction are inclination to feel shame was also associated with the indi- known to differ between voice and text interactions [53, 57]. rect expression of hostility and anger arousal [65]. In their Verbal abuse of conversational agents needs to be studied in study, Tangney et al. [66], using Anger Response Inventory, spoken, voice-based environments. found a negative correlation between guilt-proneness and ver- bal aggression in independent samples of all age groups. They In this paper, we seek to answer how a conversational agent suggested that shame and guilt were very helpful when inter- should respond to verbal abuse by its user. Our primary goal vening with individuals who displayed aggressive or antisocial is to understand what kind of response style has more positive behaviors. These study findings suggest that guilt and shame effects on those emotions found to mitigate users’ aggressive are crucial precursors to behavioral change. behaviors as well as on users’ evaluations of the agents. We manipulate voice-based conversational agents’ response styles According to Computer-Are-Social-Actor (CASA) paradigm, and users’ abuse types, and trace their effects on users’ emo- people react socially to a computer and their interactions with tions and evaluations, in an effort to understand the complex a computer correspond to the ways people naturally interact triadic relationships among verbal abuse types, response styles, with each other [54]. People were distressed by the robots’ and user reactions. More specifically, we examine: response when they abused robots [63]. Similarly, when a user verbally mistreats an agent, the user can also feel shame and • how the three different styles of responses (i.e., avoidance, guilt the same way he or she would if the user mistreated empathy, and counterattacking) made by the agents to users’ another human. verbal abuse would influence the intensity of users’ moral emotions of shame and guilt, as well as the users’ percep- tions of the agents’ capability (e.g., likability, intelligence), Agent Response Style and Many attempts have been made to identify the response styles that recipients of verbal aggression can use. Coping refers to • how the three different types of verbal abuse (i.e., insult, cognitive and behavioral efforts to control, tolerate, or reduce threaten, and swear) users employ would influence the in- demands created in a stressful situation [26]. Coping is a tensity of users’ moral emotions of shame and guilt, as well process that seeks to deal with or reconcile the situation. A as the users’ perceptions of the agents’ capability. response style emphasizes what the recipient can actually do in a stressful situation to effectively cope with [34]. We addressed these research questions in a laboratory exper- imental study (Study 2). Before the experimental study, we Most of coping tactics are aimed at reducing the emotional dis- investigated the responses of selected Intelligent Personal As- tress associated with or caused by customer or by superior in sistants (IPAs) to understand the current status of the extant the work place [28, 34]. The typical way of coping with verbal agents in handling verbal abuse. Specifically, we examined abuse that service workers take on by themselves is avoid- how each of the response styles we study was employed by ance or counterattacking [7, 28, 38].