Expression of Emotion in Virtual Crowds: Investigating Emotion Contagion and Perception of Emotional Behaviour in Crowd Simulation MIGUEL RAMOS CARRETERO Master Thesis at CSC Supervisor: Christopher Peters Examiner: Olle Bälter Abstract Emotional behaviour in the context of crowd simulation is a topic that is gaining particular interest in the area of artificial intelligence. Recent efforts in this domain have looked for the modelling of emotional emergence and social interaction inside a crowd of virtual agents, but further investigation is still needed in aspects such as simulation of emotional awareness and emotion contagion. Also, in relation to perception of emotions, many questions remain about perception of emotional behaviour in the context of virtual crowds. This thesis investigates the current state-of-the-art of emotional characters in virtual crowds and presents the im- plementation of a computational model able to generate expressive full-body motion behaviour and emotion conta- gion in a crowd of virtual agents. Also, as a second part of the thesis, this project presents a perceptual study in which the perception of emotional behaviour is investigated in the context of virtual crowds. The results of this thesis reveal some interesting findings in relation to the perception and modelling of virtual crowds, including some relevant effects in relation to the influence of emotional crowd behaviour in viewers, specially when virtual crowds are not the main focus of a particular scene. These results aim to contribute for the further development of this interdisciplinary area of computer graphics, artificial intelligence and psychology. Referat Emotionellt Beteende i Simulerade Folkmassor Emotionellt beteende i simulerade folkmassor är ett ämne med ökande intresse, inom området för artificiell in- telligens. Nya studier har tittat på modellen för social in- teraktion inuti en grupp av virtuella agenter, men fortsatt utredning behövs fortfarande inom aspekter så som sim- ulation av emotionell medvetenhet och emotionell smitta. Också, när det gäller synen på känslor, kvarstår många frå- gor kring synen på känslomässigt beteende i samband med virtuella folkmassor. Denna studie undersöker de nuvarande "state-of-the- art" emotionella egenskaperna i virtuella folksamlingar och presenterar implementationen av en datormodell som kan generera smittsamma känslor i en grupp av virtuella agen- ter. Också, när det gäller synen på känslor, kvarstår många frågor kring synen på känslomässigt beteende i samband med virtuella folksamlingar. Som en andra del av denna avhandlingen presenteras, i detta projekt, en perceptuell studie där uppfattningen av emotionella beteenden under- söks i samband med virtuella folksamlingar. Resumen Simulación de emociones en multitudes virtuales La simulación de emociones en multitudes virtuales es un área de investigación con gran interés en el campo de la inteligencia artificial. Estudios recientes han tratado de desarrollar personajes virtuales con comportamiento emo- cional y social, pero aún son pocos los estudios que han intentado simular conciencia emocional y sensibilidad al contagio de emociones. Por otro lado, aún existen ciertos interrogantes en relación con la percepción de emociones en el ámbito de multitudes virtuales. Este proyecto es el resultado de una investigación en el área de simulación de emociones para multitudes vir- tuales, e incluye la descripción detallada de un modelo com- putacional capaz de simular multitudes virtuales con con- ciencia emocional y sensibilidad al contagio de emociones. Esta investigación también incluye un estudio de percep- ción centrado en el comportamiento emocional en multi- tudes virtuales. Los resultados de este proyecto incluyen interesantes hallazgos con respecto a la percepción de com- portamiento social y pretenden contribuir al desarrollo in- terdisciplinar de los campos de la informática gráfica, la inteligencia artificial y la psicología. Acknowledgements I would like to thank all the people who helped me with the development of this master thesis: Thanks to Veronica Ginman, Yoann Gueguen and Cédric Morin for their work on the graphics scene. Thanks to Hongjie Li and Anxiao Chen for their assistance with the eye-tracker set-up. Thanks to Nadia Berthouze and her team at UCL for the use of the UCL Inter- action Centre (UCLIC) Affective Body Position and Motion Database. Thanks to Adam Qureshi for his help with the development of the perceptual study and the analysis of the results. Thanks to my supervisor Christopher Peters for his invaluable help, his guidance and his contagious enthusiasm. Thanks to my family and friends around the world for being there whenever I needed it. Finally, special thanks to my mother, my father, my sister and my brother for their support and love day by day. Thank you all! Publications The following papers are included as part of the work of this thesis: Ramos, M., Peters, C., Qureshi, A. Modelling Emotional Behaviour in Virtual Crowds through Expressive Body Movements and Emotion Contagion. Presented in SIGRAD 2014. Ramos, M., Qureshi, A., Peters, C. Evaluating the Perception of Group Emotion from Full Body Movements in the Context of Virtual Crowds. Presented in the ACM Symposium on Applied Perception 2014. Contents 1 Introduction 1 1.1. Aims of the Research . 2 1.1.1. Main Goals . 2 1.1.2. Research Question . 2 1.2. General Background . 3 1.3. Significance . 3 1.3.1. Industry . 3 1.3.2. Academia . 4 1.3.3. Societal . 4 1.4. Interdisciplinary Aspects . 4 1.5. Limitations . 5 1.6. Report Overview . 6 2 Foundations and Theory 7 2.1. Virtual Crowds and Multi-Agent Systems . 7 2.1.1. Multi-Agent Systems . 7 2.1.2. Virtual Crowds Based in Multi-Agent Systems . 8 2.1.3. A* Path-Finding Algorithm for Crowd Navigation . 9 2.2. Emotional Behaviour and Emotion Contagion . 9 2.2.1. Expressive Body Movements . 10 2.2.2. Animation of Expressive Behaviour . 10 2.2.3. Finite State Machines for Animation of Emotional Behaviour 10 2.2.4. Emotional Awareness and Emotion Contagion in Crowds . 11 2.3. Perception of Emotional Expressive Behaviour . 12 2.3.1. Perceptual Experiments . 12 2.3.2. Perceptual Studies with Virtual Behaviour . 12 2.4. Theory Summary . 12 3 Methodology 13 3.1. Management Methods . 13 3.1.1. Specifications and Milestones . 13 3.1.2. Work Tracking . 13 3.1.3. Prototypes . 14 3.2. Implementation Methods . 14 3.2.1. Computational Modelling . 14 3.2.2. Off-The-Shelf Components and Software . 14 3.3. Research Methods . 15 3.3.1. Research for Computational Modelling . 15 3.3.2. Research for Experimentation . 16 3.4. Methodology Summary . 16 4 Implementation 17 4.1. Graphic Models . 17 4.1.1. Character Model . 17 4.1.2. Scenario Model . 18 4.2. Animation . 18 4.2.1. Annotated Affective Data Corpus . 18 4.2.2. Selection of Emotional Animation . 19 4.2.3. FSM for Individual Emotional Behaviour . 19 4.2.4. Steering Paths for Walking Characters . 20 4.3. Emotional Model . 21 4.3.1. Internal State for Emotional Characters . 21 4.3.2. Algorithm of the Emotional Model . 22 4.4. Architecture of the Simulation . 24 4.5. Implementation Summary . 27 5 Evaluation 29 5.1. Controlled Scenario Simulations . 29 5.1.1. Scenario 1: Strong Contagion . 30 5.1.2. Scenario 2: Contagion by Steps . 31 5.2. Perceptual Study . 32 5.2.1. Definition of the Study . 32 5.2.2. Stimuli Composition . 33 5.2.3. Design and Set-Up . 33 5.2.4. Experiments . 34 5.2.5. Analysis of the Data . 36 5.2.6. Discussion . 39 5.3. Evaluation Summary . 39 6 Conclusions 41 6.1. General Findings . 41 6.2. Outcomes . 42 6.3. Reflections . 43 6.3.1. Gained Experience . 43 6.3.2. Difficulties . 43 6.3.3. External Reviews . 44 6.4. Final Summary and Future Work . 44 A Prototypes 47 A.1. First prototype . 47 A.2. Second prototype . 47 A.3. Third prototype . 48 B Participant Sheets 49 B.1. Ethical Clearance . 49 B.2. Instructions of the Experiment . 51 C Critic from Reviewers 53 C.1. Commentaries from SIGRAD 2014 . 53 C.2. Commentaries from the ACM Symposium of Applied Perception 2014 54 D Colour Figures 55 Bibliography 59 Chapter 1 Introduction Expression of emotion in virtual characters is a challenging topic in the area of artificial intelligence (AI) and, recently, it has gained strong interest in the context of virtual crowds. The search for the generation of more believable artificial behaviour has always been an important issue in both industrial and scientific activities related to entertainment and virtual simulation and, nowadays, the advances in computer science and engineering allow for the creation of hundreds of virtual characters and complex digital worlds filled with life [41] (see Figure 1.1). Also, recent advances in this area have opened new paths of research in the modelling of virtual emotion for artificial characters. However, some issues still remain in relation to the emotional expressive behaviour in context of multiple characters and the way they interact with each other. Although the current state-of-the-art of virtual characters is broad and there has been lots of research in relation to crowd behaviour, the simulation of emotional awareness and emotion contagion between artificial characters are two important aspects in virtual crowds that still need further investigation [41]. The development of perceptual experiments for testing how viewers perceive simulations and artificial behaviour has also strong interest in many aspects of computer science, and usually the results of these experiments help for the search of better computational models and more satisfactory human-computer interaction. In relation to crowd behaviour, modelling better computational models for virtual crowds has become a matter of great interest not only in the areas of special effects for film and video-games, but also in other specific domains such as crisis-training- simulation or psychology in relation to perception of social behaviour.