Modeling Empathy in Embodied Conversational Agents
Total Page:16
File Type:pdf, Size:1020Kb
Modeling Empathy in Embodied Conversational Agents by Özge Nilay Yalçın M.Sc., Middle East Technical University, 2014 B.Eng., Istanbul Technical University, 2010 Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in the School of Interactive Arts and Technology Faculty of Communication, Art and Technology c Özge Nilay Yalçın 2019 SIMON FRASER UNIVERSITY Summer 2019 Copyright in this work rests with the author. Please ensure that any reproduction or re-use is done in accordance with the relevant national copyright legislation. Approval Name: Özge Nilay Yalçın Degree: Doctor of Philosophy Title: Modeling Empathy in Embodied Conversational Agents Examining Committee: Chair: Marek Hatala Professor Steve DiPaola Senior Supervisor Professor Fred Popowich Supervisor Professor School of Computing Science Alissa N. Antle Internal Examiner Assistant Professor Jonathan Gratch External Examiner Professor Institute for Creative Technologies University of Southern California Date Defended: August 28, 2019 ii Ethics Statement iii iii Powered by TCPDF (www.tcpdf.org) Abstract Embodied conversational agents (ECAs) are designed with the goal of achieving natural and effortless interactions with humans by displaying the same communication channels we use in our daily interactions (e.g. gestures, gaze, facial expressions, verbal behaviors). With advances in computational power, these agents are increasingly equipped with social and emotional capabilities to improve interaction with the users. Recently, research efforts are focused on modeling empathy, which is a human trait that allows us to share and understand each other’s feelings. The emerging field of computational empathy aims to equip artificial agents with empathic behavior, which has shown great promise in enhancing the human-agent interaction. However, two issues arise in this research endeavor. Firstly, even though a variety of disciplines have extensively examined empathic behavior, there is minimal discussion on how that knowledge can be translated into computational empathy research. Second, modeling and implementing a complex behavior such as empathy poses a great challenge on fluent and automated integration of these behaviors to achieve real-time and multi-modal interaction with ECAs. This thesis aims to model and implement empathy in embodied conversational agents while focusing on both of these issues. To achieve this goal, an extensive literature review of the definitions and models of empathy from various disciplines is provided. Building upon this background knowledge, a model of empathy is presented that is suitable for interactive virtual agents, which includes three hi- erarchical layers of behavioral capabilities: emotional communication competence, emotion regulation and cognitive mechanisms. This dissertation further provides suggestions on how to evaluate perceived empathy of such a system, as there are no agreed-upon standards or best-practices in this novel field on evaluation metrics. Following the establishment of these theoretical foundations, levels of empathic behavior were implemented into an ECA with real-time spoken conversation capabilities that include synchronized gestural and emotional behavior. Evaluations of this system, which is called M-PATH, showed that the proposed levels of behavioral capabilities resulted in an increase in the perception of empathy as well as the perceived usefulness, human-likeness and believability of the agent. This disserta- tion further demonstrates that implementing empathic behaviors in artificial agents would not only improve our interaction but can also enhance our understanding of empathy by providing us with a controlled environment to implement and test our theories. iv Keywords: empathy; affective computing; embodied conversational agents; human-computer interaction v Dedication ... to me. vi Acknowledgements I would like to express my appreciation to the people that made this work possible by supporting me in various ways in the last four years. First of all, I would like to thank my senior supervisor Prof. Steve DiPaola for adopting me at a point where I was thinking of giving up on my PhD adventure in Canada and restoring my faith in academia again. His vision and enthusiasm has always inspired me to develop my ideas and pushed me further than I thought I could go. Also my supervisor Prof. Fred Popowich for his guidance, encouragement, support, and his hard questions which made me think about my research from different perspectives. I would also like to thank my friends and lab-mates at SIAT and iVizLab. Thank you Michael, Ulysses, Procheta, and our undergrads Ioana and Marie, for their help during the project. And my partner in academic misery, Burcu. Your dark humor and friendship have been priceless. I hope life will throw us and Burçin into each other’s direction much more often. My little ball of happiness, Aynur, I am glad that I decided to be roommates with a crazy person. Let’s do more projects together and pretend we are mature. Last but not least, my family. Anniş and babiş. I love you. Thank you for encouraging me in all the steps I took in my life. It was your unconditional trust that led me believe in myself and love that lifted my spirits up. Abiş, I am so lucky that my best friend and mentor is my brother. I miss your company and our laughs the most. And Sergey. The best thing about these four years was meeting you. You made everything about this journey amazingly better. I can’t wait to discover more in life with you. Finally, I would like to acknowledge the financial support. This work was partially sup- ported by the Natural Sciences and Engineering Research Council of Canada (NSERC) [RGPIN-2019-06767] and the Social Sciences and Humanities Research Council of Canada (SSHRC) [435-2017-0625]. Additional funding was provided by Ebco Eppich Graduate Scholarship in Intelligent Systems, School of Interactive Arts and Technology in Simon Fraser University (SFU). Additionally, the presentation of the papers included in this thesis in respective conferences were supported by the travel grants from SFU Graduate Stu- dent Society (GSS), School of Interactive Arts and Technology (SIAT) and Association of Computing Machinery (ACM). vii Contents Approval ii Ethics Statement iii Abstract iv Dedication vi Acknowledgements vii Table of Contents viii List of Tables xii List of Figures xiii 1 Introduction 1 1.1 Background . 1 1.1.1 Empathy in Embodied Conversational Agents . 2 1.2 Research Overview . 4 1.2.1 Research Questions . 5 1.3 Contributions . 6 1.4 Thesis Structure . 11 2 Modeling Empathy: Building a Link Between Affective and Cognitive Processes 15 2.1 Abstract . 15 2.2 Introduction . 16 2.3 Theoretical Background of Empathy . 17 2.3.1 Definitions of Empathy . 18 2.3.2 Models of Empathy . 19 2.3.3 A Systematic Categorization of Empathy Models . 23 2.4 Computational Models of Empathy . 25 2.4.1 Communication Competence . 26 viii 2.4.2 Emotion Regulation . 27 2.4.3 Cognitive Mechanisms . 28 2.5 Modeling Empathy in Artificial Agents . 31 2.5.1 Theory-Driven Approaches . 32 2.5.2 Data-Driven Approaches . 34 2.6 Discussion . 35 2.6.1 Methodological Issues . 36 2.6.2 Evaluation of the Model . 36 2.7 Future Directions . 38 2.8 Conclusion . 39 3 A Computational Model of Empathy for Interactive Agents 40 3.1 Abstract . 40 3.2 Introduction . 41 3.3 Theoretical background on empathy . 42 3.4 A proposed model of empathy . 43 3.4.1 Communication competence . 44 3.4.2 Emotion regulation . 45 3.4.3 Cognitive mechanisms . 46 3.5 Empathy in conversational agents . 47 3.5.1 Affective computing . 48 3.5.2 Social computing . 49 3.5.3 Context in conversational systems . 50 3.6 Conclusion . 51 4 Evaluating Empathy in Artificial Agents 52 4.1 Abstract . 52 4.2 Introduction . 53 4.3 Evaluation of Empathy in Humans . 54 4.3.1 Evaluating Global Empathy . 56 4.3.2 Evaluating Components of Empathy . 58 4.4 Evaluation of Empathy in Interactive Agents . 59 4.4.1 System-Level Evaluation . 60 4.4.2 Feature-Level Evaluation . 62 4.5 Concluding Remarks . 64 5 Empathic Listener Framework for Embodied Conversational Agents 66 5.1 Abstract . 66 5.2 Introduction . 66 5.3 Related Work . 67 ix 5.4 Agent and the Context . 70 5.5 Empathy Framework . 71 5.5.1 Perceptual Module . 72 5.5.2 Behavior Controller . 75 5.5.3 Behavior Generation . 79 5.6 Conclusion and Future Work . 80 5.7 Acknowledgements . 80 6 Automated Affective Gesture Generation for Embodied Conversational Agents 81 6.1 Abstract . 81 6.2 Introduction . 82 6.3 Co-Speech Gestures . 83 6.4 Related Work on Automated Gesture Generation . 84 6.5 Our Automated System . 86 6.5.1 Gesture Repertoire of the Agent . 87 6.5.2 Natural Language Processing . 92 6.5.3 Rule-Based Gesture Selection . 93 6.5.4 Emotional Dynamics Post-Processing . 94 6.6 Conclusion . 96 7 Levels of Emotional Contagion in an Embodied Conversational Agent: Implementation and Evaluation 97 7.1 Abstract . 97 7.2 Introduction . 98 7.3 Agent Behavior . 99 7.3.1 Perceptual Module . 100 7.3.2 Behavior Controller . 100 7.3.3 Behavior Generation . 101 7.4 Method . 101 7.4.1 Participants . 101 7.4.2 Procedure . 102 7.4.3 Experiment Conditions . 103 7.4.4 Study 1 . 105 7.4.5 Study 2 and Study 3 . 106 7.5 Discussion . ..