Expression of Emotion in Virtual Crowds: Investigating Emotion Contagion and Perception of Emotional Behaviour in Crowd Simulation

MIGUEL RAMOS CARRETERO

Master Thesis at CSC Supervisor: Christopher Peters Examiner: Olle Bälter

Abstract

Emotional behaviour in the context of crowd simulation is a topic that is gaining particular interest in the area of artificial intelligence. Recent efforts in this domain have looked for the modelling of emotional emergence and social interaction inside a crowd of virtual agents, but further investigation is still needed in aspects such as simulation of emotional awareness and emotion contagion. Also, in relation to perception of emotions, many questions remain about perception of emotional behaviour in the context of virtual crowds. This thesis investigates the current state-of-the-art of emotional characters in virtual crowds and presents the im- plementation of a computational model able to generate expressive full-body motion behaviour and emotion conta- gion in a crowd of virtual agents. Also, as a second part of the thesis, this project presents a perceptual study in which the perception of emotional behaviour is investigated in the context of virtual crowds. The results of this thesis reveal some interesting findings in relation to the perception and modelling of virtual crowds, including some relevant effects in relation to the influence of emotional crowd behaviour in viewers, specially when virtual crowds are not the main focus of a particular scene. These results aim to contribute for the further development of this interdisciplinary area of computer graphics, artificial intelligence and psychology.

Referat

Emotionellt Beteende i Simulerade Folkmassor

Emotionellt beteende i simulerade folkmassor är ett ämne med ökande intresse, inom området för artificiell in- telligens. Nya studier har tittat på modellen för social in- teraktion inuti en grupp av virtuella agenter, men fortsatt utredning behövs fortfarande inom aspekter så som sim- ulation av emotionell medvetenhet och emotionell smitta. Också, när det gäller synen på känslor, kvarstår många frå- gor kring synen på känslomässigt beteende i samband med virtuella folkmassor. Denna studie undersöker de nuvarande "state-of-the- art" emotionella egenskaperna i virtuella folksamlingar och presenterar implementationen av en datormodell som kan generera smittsamma känslor i en grupp av virtuella agen- ter. Också, när det gäller synen på känslor, kvarstår många frågor kring synen på känslomässigt beteende i samband med virtuella folksamlingar. Som en andra del av denna avhandlingen presenteras, i detta projekt, en perceptuell studie där uppfattningen av emotionella beteenden under- söks i samband med virtuella folksamlingar.

Resumen

Simulación de emociones en multitudes virtuales

La simulación de emociones en multitudes virtuales es un área de investigación con gran interés en el campo de la inteligencia artificial. Estudios recientes han tratado de desarrollar personajes virtuales con comportamiento emo- cional y social, pero aún son pocos los estudios que han intentado simular conciencia emocional y sensibilidad al contagio de emociones. Por otro lado, aún existen ciertos interrogantes en relación con la percepción de emociones en el ámbito de multitudes virtuales. Este proyecto es el resultado de una investigación en el área de simulación de emociones para multitudes vir- tuales, e incluye la descripción detallada de un modelo com- putacional capaz de simular multitudes virtuales con con- ciencia emocional y sensibilidad al contagio de emociones. Esta investigación también incluye un estudio de percep- ción centrado en el comportamiento emocional en multi- tudes virtuales. Los resultados de este proyecto incluyen interesantes hallazgos con respecto a la percepción de com- portamiento social y pretenden contribuir al desarrollo in- terdisciplinar de los campos de la informática gráfica, la inteligencia artificial y la psicología.

Acknowledgements

I would like to thank all the people who helped me with the development of this master thesis:

Thanks to Veronica Ginman, Yoann Gueguen and Cédric Morin for their work on the graphics scene.

Thanks to Hongjie Li and Anxiao Chen for their assistance with the eye-tracker set-up.

Thanks to Nadia Berthouze and her team at UCL for the use of the UCL Inter- action Centre (UCLIC) Affective Body Position and Motion Database.

Thanks to Adam Qureshi for his help with the development of the perceptual study and the analysis of the results.

Thanks to my supervisor Christopher Peters for his invaluable help, his guidance and his contagious enthusiasm.

Thanks to my family and friends around the world for being there whenever I needed it.

Finally, special thanks to my mother, my father, my sister and my brother for their support and love day by day.

Thank you all!

Publications

The following papers are included as part of the work of this thesis:

Ramos, M., Peters, C., Qureshi, A. Modelling Emotional Behaviour in Virtual Crowds through Expressive Body Movements and Emotion Contagion. Presented in SIGRAD 2014.

Ramos, M., Qureshi, A., Peters, C. Evaluating the Perception of Group Emotion from Full Body Movements in the Context of Virtual Crowds. Presented in the ACM Symposium on Applied Perception 2014.

Contents

1 Introduction 1 1.1. Aims of the Research ...... 2 1.1.1. Main Goals ...... 2 1.1.2. Research Question ...... 2 1.2. General Background ...... 3 1.3. Significance ...... 3 1.3.1. Industry ...... 3 1.3.2. Academia ...... 4 1.3.3. Societal ...... 4 1.4. Interdisciplinary Aspects ...... 4 1.5. Limitations ...... 5 1.6. Report Overview ...... 6

2 Foundations and Theory 7 2.1. Virtual Crowds and Multi-Agent Systems ...... 7 2.1.1. Multi-Agent Systems ...... 7 2.1.2. Virtual Crowds Based in Multi-Agent Systems ...... 8 2.1.3. A* Path-Finding Algorithm for Crowd Navigation ...... 9 2.2. Emotional Behaviour and Emotion Contagion ...... 9 2.2.1. Expressive Body Movements ...... 10 2.2.2. Animation of Expressive Behaviour ...... 10 2.2.3. Finite State Machines for Animation of Emotional Behaviour 10 2.2.4. Emotional Awareness and Emotion Contagion in Crowds . . 11 2.3. Perception of Emotional Expressive Behaviour ...... 12 2.3.1. Perceptual Experiments ...... 12 2.3.2. Perceptual Studies with Virtual Behaviour ...... 12 2.4. Theory Summary ...... 12

3 Methodology 13 3.1. Management Methods ...... 13 3.1.1. Specifications and Milestones ...... 13 3.1.2. Work Tracking ...... 13 3.1.3. Prototypes ...... 14 3.2. Implementation Methods ...... 14 3.2.1. Computational Modelling ...... 14 3.2.2. Off-The-Shelf Components and Software ...... 14 3.3. Research Methods ...... 15 3.3.1. Research for Computational Modelling ...... 15 3.3.2. Research for Experimentation ...... 16 3.4. Methodology Summary ...... 16

4 Implementation 17 4.1. Graphic Models ...... 17 4.1.1. Character Model ...... 17 4.1.2. Scenario Model ...... 18 4.2. Animation ...... 18 4.2.1. Annotated Affective Data Corpus ...... 18 4.2.2. Selection of Emotional Animation ...... 19 4.2.3. FSM for Individual Emotional Behaviour ...... 19 4.2.4. Steering Paths for Walking Characters ...... 20 4.3. Emotional Model ...... 21 4.3.1. Internal State for Emotional Characters ...... 21 4.3.2. Algorithm of the Emotional Model ...... 22 4.4. Architecture of the Simulation ...... 24 4.5. Implementation Summary ...... 27

5 Evaluation 29 5.1. Controlled Scenario Simulations ...... 29 5.1.1. Scenario 1: Strong Contagion ...... 30 5.1.2. Scenario 2: Contagion by Steps ...... 31 5.2. Perceptual Study ...... 32 5.2.1. Definition of the Study ...... 32 5.2.2. Stimuli Composition ...... 33 5.2.3. Design and Set-Up ...... 33 5.2.4. Experiments ...... 34 5.2.5. Analysis of the Data ...... 36 5.2.6. Discussion ...... 39 5.3. Evaluation Summary ...... 39

6 Conclusions 41 6.1. General Findings ...... 41 6.2. Outcomes ...... 42 6.3. Reflections ...... 43 6.3.1. Gained Experience ...... 43 6.3.2. Difficulties ...... 43 6.3.3. External Reviews ...... 44 6.4. Final Summary and Future Work ...... 44 A Prototypes 47 A.1. First prototype ...... 47 A.2. Second prototype ...... 47 A.3. Third prototype ...... 48

B Participant Sheets 49 B.1. Ethical Clearance ...... 49 B.2. Instructions of the Experiment ...... 51

C Critic from Reviewers 53 C.1. Commentaries from SIGRAD 2014 ...... 53 C.2. Commentaries from the ACM Symposium of Applied Perception 2014 54

D Colour Figures 55

Bibliography 59

Chapter 1

Introduction

Expression of emotion in virtual characters is a challenging topic in the area of artificial intelligence (AI) and, recently, it has gained strong interest in the context of virtual crowds. The search for the generation of more believable artificial behaviour has always been an important issue in both industrial and scientific activities related to entertainment and virtual simulation and, nowadays, the advances in computer science and engineering allow for the creation of hundreds of virtual characters and complex digital worlds filled with life [41] (see Figure 1.1). Also, recent advances in this area have opened new paths of research in the modelling of virtual emotion for artificial characters. However, some issues still remain in relation to the emotional expressive behaviour in context of multiple characters and the way they interact with each other. Although the current state-of-the-art of virtual characters is broad and there has been lots of research in relation to crowd behaviour, the simulation of emotional awareness and emotion contagion between artificial characters are two important aspects in virtual crowds that still need further investigation [41]. The development of perceptual experiments for testing how viewers perceive simulations and artificial behaviour has also strong interest in many aspects of computer science, and usually the results of these experiments help for the search of better computational models and more satisfactory human-computer interaction. In relation to crowd behaviour, modelling better computational models for virtual crowds has become a matter of great interest not only in the areas of special effects for film and video-games, but also in other specific domains such as crisis-training- simulation or psychology in relation to perception of social behaviour. Specially for the latter, many questions still remain about how people perceive expression of emotion in a context of multiple individuals and, particularly, there are few studies dealing with the investigation of emotional effects of virtual crowd behaviour. The purpose of this research is to try to shed more light to all these matters and to contribute to this area of computer graphics related to emotional behaviour in virtual crowds.

1 CHAPTER 1. INTRODUCTION

Figure 1.1. These pictures display examples of virtual crowds from Monster University (Pixar, 2013), The Lord of the Rings (New Line Cinema, 2001), and Assassin’s Creed (Ubisoft, 2007).

1.1. Aims of the Research

In general terms, this thesis deals with the investigation of the current state-of- the-art of emotional behaviour for virtual crowds, particularly in relation to emotion contagion between virtual characters. Also, this work intends to investigate some perceptual aspects related with emotional virtual crowds, particularly when they are part of the context of a certain virtual scene.

1.1.1. Main Goals

This thesis aims to accomplish the following goals:

To develop a computational model capable of generating a virtual crowd in which the characters are both able to convey different emotional behaviour and susceptible to be affected by other characters’ emotions (See Chapter 4).

To design a perceptual study with the purpose to evaluate the computational model as well as to investigate certain aspects about perception of virtual emotional behaviour (See Chapter 5).

1.1.2. Research Question

In addition to the goals stated, this work deals with a study in relation to the effects of social context on the perception of emotions of a virtual scene. Specif- ically, this research aims to find how the effects of an emotional virtual crowd in a background context influence the perception of emotions of a virtual scene (See Chapter 5). As a problem statement, the research question is outlined as follow:

How is the emotional behaviour of virtual crowds perceived in virtual scenes?

2 1.2. GENERAL BACKGROUND

1.2. General Background

The simulation of groups of virtual agents has been widely investigated since the capabilities of computer graphics technologies allowed for the generation of multiple artificial individuals, and one of the most significant works in this area can be attributed to Reynolds and his ’boids’ model [37] in 1987. From this early stage, later works have been continuously improving the modelling of virtual agents in relation to more believable crowd behaviour [31], autonomous agents [39], collision avoidance [18] or impressions of personality (OCEAN) [10]. In relation to perception of emotion behaviour from virtual characters, there has been recent research regarding the perception of crowds [14] and body language [32]. More recent studies have also been studying the generation [19] and mapping [3] of expressive motions between artificial agents and real people. Social awareness and psychology of human crowds has been also a challenging topic in relation to the understanding of these phenomena, and there has been a large investigation of the emotion contagion effect and its significance in human behaviour [20]. Specially for virtual characters and crowds, recent studies have developed some generic computational models in order to simulate the emotional awareness of virtual characters and the effects of emotion contagion [27, 35, 38, 6, 2]. Nevertheless, further investigation is needed in this area, since the modelling of virtual emotions is still a recent area of research and the current state-of-the-art is still rather basic and far from being mature [29].

1.3. Significance

Virtual crowds and emotion behaviour for virtual characters is a topic with increasing interest in AI and computer graphics. The following sections present an overview of some of the most significant contributions of this research in the industry and the academia as well as in relation with some societal aspects.

1.3.1. Industry Believability in virtual crowds, that is, the portrayal of realistic behaviour in virtual characters, is a matter of great importance in several areas of computer graphics industry. Activities in the domain of entertainment, particularly in films and video-games, required an increasingly necessity of recurring crowds digitally generated to fill the context of the stories. Famous examples such as the film tril- ogy The Lord of The Ring or games like the saga Assassin’s Creed use a large amount of virtual characters in order to represent crowded cities and big armies. Thus, in order to achieve a better immersion of the viewers into these virtual worlds, it is of great importance to reach more believability in the behaviour of their vir- tual characters, particularly when dealing with emotional behaviour. As stated by Lasseter [26], when bringing artificial characters to life, one of the most important things to bear in mind is the personality and the simulation of a thought process,

3 CHAPTER 1. INTRODUCTION that is, the simulation of thinking to justify each motion and each posture of the character. Likewise, this thought process should be as well emotionally portrayed in the behaviour of characters. On the other hand, understanding the effects of social context (the aim of the research question, see Section 1.1.2) on the perception of emotion from a virtual crowd is of great importance, particularly in a domain where the artificial behaviour of multiple agents may alter the perception of other main characters which intends to robustly convey certain emotions. It is of great significance to recognise those situations in a particular scene in which the perception of emotions is context sensi- tive, so then animators, directors and storytellers can take extra care when dealing with these situations.

1.3.2. Academia In the academia domain, the implementation of computational models dealing with emotional behaviour of multiple characters has been of great significance in the study of social behaviour, particularly for cases of emergency evacuation or panic simulations. Hence, the seek for more realistic models regarding virtual crowd be- haviour seems to be of great interest in educational areas such as crisis-management and training-simulations [27]. Also, regarding emotion contagion, there have been many studies that have proved, theoretically, the importance of this contagion effect in the way humans and other species behave [20, 36], which gives more importance to the investigation of this effect in the area of AI as well as to the development of better computational models capable of simulating this phenomenon.

1.3.3. Societal Regarding the study of social context in relation to the perception of emotional behaviour of virtual crowds, understanding the human mind has been the most important task in the area of psychology. Thus, while implementing AI models to simulate particular processes of the mind, certain paths of knowledge about the complexity of the human mind can be as well discovered, helping humanity to gather more knowledge about how humans behave and the way emotions are perceived. Also, economical and ecological aspects can possibly be as well related with the aim of this thesis, and behaviour of crowds in relation to specific contexts such as urban environments and sustainable cities may bring up also interesting research questions, since these contextual environments are altered and modified to a large extent by the behaviour of human beings and their interaction with such environments.

1.4. Interdisciplinary Aspects

Although the main area of this thesis is computer graphics, the research domain deals also with several aspects of AI and psychology, mainly in relation to artificial

4 1.5. LIMITATIONS agents, emotion contagion and perception of emotional behaviour. Due to these interdisciplinary aspects, it is of great importance to clarify certain terms that may not be common for readers without a background in these areas. The following list presents a definition of some of the most important key terms of this thesis (more information about the foundations and theory of this project is explained in Chapter 2):

Emotion: For this research, this term is defined as the particular state of mind from which an individual experience a certain mood. While an emotion represents a state of mind in general (i.e. happiness), mood refers to the physiological and psychological aspect of that state (in the case of happiness, the mood would be well-being) [12].

Virtual character: An animated artificial agent represented by a visual model, usually with humanoid appearance. A large number of virtual characters considered together constitute a virtual crowd [34].

Emotional behaviour: All the movements, gestures and positions made by an individual from which it is possible to infer or deduce a certain emotion [8] (see Section 2.2.1 for more details).

Emotion contagion: The tendency to catch another person’s emotions. It is the first step of empathy, this complex cognitive process that consists in evaluating the situation from another person’s perspective [20] (see Section 2.2.4 for more details).

Affective appraisal: Human process from which the evaluation of events from the environment has an emotional significance with respect to the individual goals and decisions [23].

1.5. Limitations

The current state-of-the-art in the domain of virtual crowds and emotional be- haviour is enormously large, so it is important to define some limitations in this research, since the scope of this thesis cannot cover every single aspect of this area. Hence, this research focus on the perception of emotional behaviour of virtual char- acters in relation to full-body movements, not considering other details such as face expressions or finger motion. Particularly, the emotions that are handled are two: happiness and sadness, both taken from the list of Ekman basic emotions [11] (see Section 4.2.2). In relation with the computational model presented, the implemen- tation focuses on the simulation of emotion contagion in virtual crowds, but it does not consider high-quality rendering strategies for virtual agents nor other aspects of crowds such as character variety nor advanced motion planning.

5 CHAPTER 1. INTRODUCTION

1.6. Report Overview

Chapter 1 has introduced the aim of this thesis, including an overview to the main areas of related research, its potential contributions and the boundaries in with the project has been carried out. The next chapters of this report are organized as follow: Chapter 2 presents the theory of this project, describing the foundations on which this research is based; Chapter 3 details the methodology followed in this project in terms of manage- ment, implementation and research; Chapter 4 describes the implementation of the computational model for virtual crowds, including a detailed description of the al- gorithm for the simulation of the emotion contagion effect; Chapter 5 presents the evaluation of the computational model, first, through a controlled simulation, and second, through a perceptual study; finally, Chapter 6 concludes this report with some comments about the findings of this research as well as some reflections and a final discussion of potential future work in related areas.

6 Chapter 2

Foundations and Theory

Chapter 2 details the foundations of the work and the theory in which this thesis lays in order to understand better the basis of this research. The main areas of this research can be categorized in three main domains: virtual crowds, emotional behaviour and perception of emotions.

2.1. Virtual Crowds and Multi-Agent Systems

The first part of this research deals with multi-agent systems and the theory from which virtual crowds are created.

2.1.1. Multi-Agent Systems

Although it is not the intention of this thesis to go in deep detail in this matter, it is important to clarify the definition of this term, since it is the basis of the computational model of this project (see Chapter 4 for more details). To define what a multi-agent system is, we need to specify first what an agent is. Although there are several definitions for this term, the one that approximates better to the purpose of this thesis is the one coined by Jennings et al. [22], who proposes that an agent is a computer-based system, situated in an environment that acts autonomously and flexibly to attain the objectives for which it was created. For the matter of this research, we will consider an agent as a virtual character represented by a graphic model which interacts with other virtual characters in the context of a virtual scenario. Thus, a multi-agent system can be defined as an organized group of individual agents able to communicate and interact with each other following a set of rules [22]. The communication takes place through the exchange of messages using certain protocols of coordination and cooperation. In the context of this research, the multi-agent system will be a virtual crowd composed of characters represented by individual agents (See Figure 2.1).

7 CHAPTER 2. FOUNDATIONS AND THEORY

Perceptors ENVIRONMENT

Agent Y

Agent X Scenario Agent Z

. . .

Actuators

Figure 2.1. Simplified diagram of a multi-agent system. The agent X perceives information from the environment (the scenario and the other agents) and generate responses according to this.

2.1.2. Virtual Crowds Based in Multi-Agent Systems

Multi-agent systems have great importance in crowd simulation and they con- tribute significantly to the functionality, individuality and autonomy of the indi- viduals of the virtual characters of a crowd, as discussed by Pelechano et al. [34]. For many computational models for virtual crowd generation, it is in these systems in which rely the implementation of the AI and the interaction between virtual characters. There has been some significant milestones based in the area of multi-agent systems applied to crowd simulation. In academia, the pioneering work of a multi- agent system for flock behaviour could be considered Reynolds’ ’boids’ model [37]. In the industry, some of the most famous and successful software for virtual crowds are Massive SW1 and Golaem Crowd2, two simulation systems based on multi-agent approaches that have been extensively used for crowd generation for famous films such as the trilogy The Lord of The Rings and in digital animation such as Pixar’s Monsters University. Others approaches apart from multi-agent systems have also attempted to sim- ulate crowd behaviour with methods based on particle systems, rule-based simula- tions and social forces [34]. Since these approaches consider the crowd as a whole, they are very effective with general behaviour. However, these approaches ignore the individuality of each member of the crowd, which make them less effective when dealing with characteristics related with personality such as emotional behaviour. For these reasons, it seems that multi-agent systems are more feasible for the seek- ing of individual behaviour in virtual characters of crowds. This is also the main reason why this research has taken this approach (see Chapter 4).

1Massive SW: http://www.massivesoftware.com/ 2Golaem Crowd: http://www.golaem.com/

8 2.2. EMOTIONAL BEHAVIOUR AND EMOTION CONTAGION

Figure 2.2. This is an example of how the A* path-finding algorithm works for virtual agents. Supposing that the red rectangle is an obstacle, if the blue agent wants to reach the yellow agent, the A* path-finding algorithm will calculate the shortest path on the graph mesh of the scenario.

2.1.3. A* Path-Finding Algorithm for Crowd Navigation

An important aspect to take into account in crowd behaviour is the steering navigation for walking characters. The most basic behaviour for a character from a virtual crowd is to be able to find a path between an origin point and a destiny point avoiding any obstacle that the scenario could have in-between such as walls, buildings or roads where the pedestrian is not able to pass. To implement this behaviour, there are already several methods for video-games which deal with this problem. One of the most common is the A* path-finding algorithm, which was used in the computational model of this thesis. The A* path-finding algorithm is based on a graph mesh, and it basically calcu- lates the optimal path between two single nodes. For the heuristics of path-finding, the optimal path is usually the shortest path. This method follows the same core than the Dijkstra search algorithm [9], with some minor changes in the heuris- tics. Figure 2.2 shows an example of how this algorithm works for virtual agents. Additional features such weighting the connections between nodes allows for more realism when calculating paths in environments with non-physical obstacles such as gardens or roads. Although this approach is not the most realistic for steering behaviour, since it always considers the shortest path, it is not the purpose of this thesis to go further in this matter, but the reader can find more information about more complex techniques of path-finding in the literature of this research [42, 34].

2.2. Emotional Behaviour and Emotion Contagion

The second area of this thesis concerns with the theory of emotional behaviour, the animation of emotional virtual characters and the effects of emotion contagion in relation to crowd behaviour.

9 CHAPTER 2. FOUNDATIONS AND THEORY

2.2.1. Expressive Body Movements

The term body language refers to certain body and face behaviours in humans and other species from which it is possible to infer intentions, thoughts or emo- tional states [8]. In relation to this, there have been many studies coping with the investigation of body movements and expression of emotions, both in facial expres- sions and in motion behaviour [11, 5, 8]. As stated in Section 1.5, this thesis focus on the latter, since dealing with facial expressions requires much more extensive investigation and that goes further beyond the scope and the limitations of this research. According to Marco de Mejier [8], the inference of emotions from body move- ments is based in several aspects of the action, mainly characterized by the move- ments of the trunk and the arms, the force, the velocity and the direction of the action. As this author discusses, certain postures and behaviours are strongly re- lated with positive mood, such as openness of arm arrangement and head position. Likewise, other cues can also be related with negative mood, such as drooping one’s head and closeness in the movements. Although it is difficult to categorize objec- tively every single gesture, position and movement in separate emotional categories, the results of several studies [5, 25, 8] indicate this general relationship between body movements and emotions, despite minor differences that can be found in relation to gender, age or culture [13].

2.2.2. Animation of Expressive Behaviour

In relation to the animation of emotional behaviour for virtual characters, there has been lots of research with the aim to map body movements from real actors into virtual characters. This process, known as motion tracking or motion capture (MoCap), has become the pioneer technology of animation for virtual characters in the entertainment industry as well as in academic research. Particularly for expres- sive body movements, there has been a recent research carried out at the University College of London [25], in which several acted and non-acted movements captured with MoCap technologies were studied and emotionally categorized according to the perception from people from three different cultures (Western, Middle-East and Far East cultures). As it is explained in Chapter 3, the aforementioned research and its MoCap library were the main references used for the animation of the virtual crowd of this thesis.

2.2.3. Finite State Machines for Animation of Emotional Behaviour

At this stage, it is important to clarify how the implementation of the animation is set for several virtual characters. Due to the large number of characters in a crowd, it is not feasible to use a traditional way of animation for each single character. In these cases, when dealing with AI for animation, a very common tool easy to use is finite state machines (FSM).

10 2.2. EMOTIONAL BEHAVIOUR AND EMOTION CONTAGION

Drink [Thirsty]

[Not Thirsty]

Awake [Hungry]

[Full] Eat

Figure 2.3. Example of a finite state machine for a virtual character. The conditions of ’thirsty’ and ’hungry’ change the state of the FSM and, therefore, the behaviour of the character.

[Tired] Drink [Thirsty] Look for [Not Thirsty] the bed

Awake [At the bed] [Hungry] [Rested] [Full] Eat Sleep

Figure 2.4. Example of a hierarchical finite state machine. In this case, the condition of ’tired’ changes the hierarchy and, depending on it, the behaviour of the character is controlled by a different FSM.

In animation, a FSM is usually represented with a directed graph in which the nodes (states) represent the different animation behaviours that a character can portray and the connections (transitions) depict the conditions to move from one behaviour to another [23]. Figure 2.3 illustrates an example of a FSM. Depending of the necessities of the simulation and the complexity of the be- haviour of the virtual characters, a FSM can become hierarchical, that is, a FSM which allow grouping states together in a hierarchical way [23], allowing with this several layers of behaviour. Figure 2.4 shows an example of a hierarchical FSM. In this research, the animation portrayed by the virtual characters of the crowd is controlled by hierarchical FSM. More details about this is explained in Chapter 4.

2.2.4. Emotional Awareness and Emotion Contagion in Crowds Emotion contagion, as defined by Hatfield et al. [20] is the tendency to automat- ically mimic and synchronize expressions, vocalizations, postures, and movements with those of another person’s and, consequently, to converge emotionally. In rela- tion with the body language, Hatfield also proposes that emotion contagion takes place between two or more individual through an unconscious mechanism of mimic and synchronization [20].

11 CHAPTER 2. FOUNDATIONS AND THEORY

The theory of emotion contagion has been deeply investigated, and the psy- chological process and effects from which this phenomenon arises has a strongly scientific base [20]. However, this effect is still missing or have a lack of realism in many of the current crowd simulations [41], although, as commented before in Section 1.2, recent studies have developed the first approaches to imitate this phe- nomenon in virtual crowds [35, 33].

2.3. Perception of Emotional Expressive Behaviour

The third part of this thesis presents an overview of perceptual studies in relation with how people perceive and infer emotions from virtual behaviour.

2.3.1. Perceptual Experiments Perceptual experiments deals with the investigation of the way humans respond and interpret certain stimuli directly related with perception (vision and hearing, mainly). With a concise and clear research question, the participants are asked to follow a certain task related with the aim of the problem that has to be solved. Then, the data gathered serve as the base for a post-analysis in order to find an answer for the aforementioned research question [7].

2.3.2. Perceptual Studies with Virtual Behaviour The investigation of how human perceive and infer emotions from others has been extensively investigated. According to Papelis et al. [33], there are mainly four channels to recognize emotions from other individual: face, voice, posture and body movement. Particularly for the latter, many studies have investigated how humans recognize emotions through body language, movements and gestures [24], as well as the perception of expressive body movements and emotions in virtual characters [5]. Also, recent studies have dealt directly with computational models for virtual characters and have focused their evaluation in perceptual experiments to help them recognize, evaluate and map expressive motions to artificial characters from motion capture technologies [4, 3, 19]. In a similar way, the computational model implemented in this thesis will be also evaluated through a perceptual experiment. More details about this are presented in Chapter 5.

2.4. Theory Summary

Chapter 2 has described the theory in which this project is based on, including a general overview to the main algorithms and technologies used in the computational model, as well as the foundations to understand the basic theory of body language, emotion contagion and design of perceptual studies. The next chapter introduces the methodology followed in this research.

12 Chapter 3

Methodology

Chapter 3 describes the methodology followed for the development of this thesis, including an overview of the methods used for management, implementation and research.

3.1. Management Methods

Due to the amount of work that is required in a master thesis, it was critical to follow certain methods of management and control to keep track of the work and to get auto-feedback of the progress during the development of the thesis. The next sections describe the management methods followed in this research. Additional comments and reflections about the progress of the work are detailed in Chapter 6.

3.1.1. Specifications and Milestones In an early stage of the development of the master thesis, and after the bound- aries of the research were settled down, the specifications of the project were es- tablished with the agreement of the supervisor and the examiner. In it, a set of activities was created in order to divide the work into individual workable parts. In addition to these activities, several milestones were defined in order to set individual sub-goals to achieve after each month of the project.

3.1.2. Work Tracking In order follow the progress of the specification agreement, several methods were used to keep the track of the work monthly, weekly and daily. To divide the progress of the work monthly, a time plan based on a Gantt Chart was created and updated every month with the progress of the research. In addition to this, the activities of each month were tracked in a tracing log. Certain number of hours were assigned to each activity, aiming to complete them in a monthly work-load of 120 solid hours (30 hours/week, 6 hours/day). Also, physical meetings were agreed with the supervisor every week and the important things were written in a meeting

13 CHAPTER 3. METHODOLOGY log. Lastly, individual notes of the progress of the research for each day were kept in a personal log.

3.1.3. Prototypes Based on an iterative-incremental development (See next Section 3.2), three prototypes were defined for the implementation of the computational model, each of them containing a software deliverable with incremental features (see Appendix A). In addition, a deadline week was defined for each of these prototypes in order to finish them and to keep track of the progress of the research.

3.2. Implementation Methods

Since a significant part of this research was focused on the development of a computational model for virtual crowds, it was essential to delineate certain imple- mentation methodologies and to decide the external software and the programming tools from which the model would be built up.

3.2.1. Computational Modelling The implementation methodology used for the development of the computa- tional model was based on an iterative-incremental development with basis in Agile Methodologies and Personal Software Development (Humphrey, W.). The full development of the computational model was divided in three iteration, each of them consisting of: initialization, analysis of the problem to solve, design- ing of a solution, coding of the algorithms, evaluation and final post-mortem. In each iteration new features were added incrementally to the computational model according to the three prototypes defined in Section 3.1.3. Since the computational model was aimed to be used in a perceptual experiment, the implementation of it was always guided by this fact. This required certain decisions when dealing with the visual aspects of the crowd such as the selection of a character model, the emotional animations, the colours of the scenario or the amount of characters, among others. All these characteristics are further explained in Chapter 4 and 5.

3.2.2. Off-The-Shelf Components and Software For the implementation of the computational model, several software tools were used in order to complete the development of the model graphically and internally. Since the project deals with the generation of an emotional virtual crowd in a virtual scenario, the model used to generate each of the individuals of the crowd was a free model of an androgynous mannequin downloaded from the online database Tur- boSquid1. In addition to this, the emotional animation was taken from two motion

1TurboSquid: www.turbosquid.com

14 3.3. RESEARCH METHODS capture libraries of acted emotions: the Carnegie-Mellon graphics Lab Motion Cap- ture Database 2 and the UCLIC Affective Posture and Body Motion Database [25]. These MoCap animations were mapped onto the mannequin model aforementioned in 3D Studio Max3. The virtual scenario used was based on a model developed from other students of the KTH as part of a course project. In relation to the behaviour, the crowd simulation was generated in the Unity Game Engine4, in which the steering behaviour was implemented with a free third- party plug-in based on the A* Pathfinding5 algorithm (See Chapter 4). The emo- tional animation of the characters were controlled with Mechanim, the Unity’s an- imation system based on hierarchical FSM (see Chapter 2 for more details about this). Finally, all the scripts containing the algorithms of the computational model and the emotional behaviour were coded in C# for Unity. Regarding the perceptual study used in the evaluation (see Chapter 5), an eye- tracker camera (Tobii X1 Light Eye Tracker) was used as part of the tools of the experiments, and the participant tests were designed with the software Tobii Stu- dio6. The sound of the scenes of the stimuli was taken from the open audio database Freesound 7.

3.3. Research Methods

To gather information about the state-of-the-art of the areas that were in close relation with the thesis, it was important to follow a research methodology to keep track of the important findings in each domain. For this purpose, six research logs were created, named: virtual crowds, emotional behaviour, emotion contagion, emotional model, perceptual experiments and a last additional category for general notes. During the first stage of the thesis there was an extensive preliminary research in the areas aforementioned, both in the form of articles and books. The results of this research is the Bibliography section that can be found at the end of this report.

3.3.1. Research for Computational Modelling

After reading the literature related with the different areas of this thesis, two recent research works were selected as the base for the implementation of the al- gorithms of the computational model. The one with strongest influence was the research of Pereira et al. [35], who proposes an implementation of a computa- tional model based in emotion contagion. The second research, which theory also contributes to this thesis, was a study by Papelis et al. [33], who focuses in the implementation of emotional behaviour of individuals using a cognitive approach.

2CMU MoCap: http://mocap.cs.cmu.edu/ 33D Studio Max: www.autodesk.com/products/autodesk-3ds-max/ 4Unity: www.unity3d.com 5A* Pathfinding: www.arongranberg.com/astar/ 6Tobii: www.tobii.com 7Freesound: http://www.freesound.org/

15 CHAPTER 3. METHODOLOGY

3.3.2. Research for Experimentation The perceptual study designed as part of the evaluation of the computational model 5, has its basis Cunningham’s tutorial [7], who proposes a method for design- ing perceptual experiments. Following this methodology, the perceptual study was developed in order to test the model and to find an answer for the research question stated in Chapter 1. The considerations taken into account while designing the experiment were as follow:

The concise definition of the experiment: the domain of the research, the main objectives of the study and, most important, the research question that the experiment is addressing.

The target audience: the participants that are going to take part in the ex- periment, which could be limited by age, gender, occupation, nationality, etc.

The stimuli creation: the perceptual stimuli that is going to be presented to the participants during the experiment.

The design of the task for the participants: the clear definition of what the participant should do in the experiment and how the data is going to be collected from them (free description, rating, forced choice, etc.).

The ethics aspects implied within the experiment: this includes the conditions of the experiment and the guarantee about the physical and psychological integrity of the participants. This research follows the ethical guidelines of KTH 8 (See Appendix B.1).

Two experiments were designed in this project, being the first a pre-study for testing and the second a main experiment which provided the main results of the research. More information about this is detailed in Chapter 5.

3.4. Methodology Summary

Chapter 3 has described the main methodologies followed in this research: first, the management methods to outline a work-plan and track the work; second, the implementation methods and the software engineering approach for the development of the computational model; and, third, the research methods to collect the relevant information in the areas of related work and to develop the perceptual study. Next chapter presents the details of the implementation of the computational model for virtual crowds and emotion contagion.

8KTH Ethical Policy: http://intra.kth.se/en/regelverk/policyer/etisk-policy-1.27141

16 Chapter 4

Implementation

Chapter 4 describes the implementation for the generation of a virtual crowd with expressive behaviour and emotion contagion. This includes the details regard- ing the graphical models used, the emotional animation, the steering tools of the crowd and the computational model developed for the control of emotions.

4.1. Graphic Models

To construct the virtual world, two graphic models were used to represent both the individuals and the scenario: an androgynous mannequin and a model of the frontal building of the KTH campus.

4.1.1. Character Model

As explained in Section 3.2.2, a model of an androgynous mannequin was used to represent graphically each of the agents of the crowd (see Figure 4.1). This model was chosen due to its simplicity, since this project is limited to full-body motion behaviour and the absence of a face and other details (such as finger motion) suits well the purpose of the research. The only modification made in the model was the addition of a small protuberance in the front of the face in order to simulate a nose, allowing to identify better the direction of the face.

Figure 4.1. Mannequin models representing virtual characters of the crowd.

17 CHAPTER 4. IMPLEMENTATION

Figure 4.2. KTH model representing the contextual scenario for the virtual crowd.

4.1.2. Scenario Model

To create a context in which the virtual crowd moves, a virtual scenario was used in order to set the steering paths (Section 4.2.4). The scenario chosen was a model of the main building of KTH (see Figure 4.2). The reasons why this model was chosen was, first, due to its availability (which saved time from the thesis, since the purpose was not focused in modelling an scenario) and, second, due to its suitability to set a context for the perceptual study (Chapter 5).

4.2. Animation

The animation of the virtual crowd was made using motion capture technologies (see Section 2.2.2). Since the androgynous mannequin model was already rigged (that is, with a virtual skeleton ready to be animated) the mapping of MoCap animation was very straight forward in 3D Studio Max, and only some tweaks were needed to keep the animation clean and believable.

4.2.1. Annotated Affective Data Corpus

Due to the interest of using emotional behaviour for the virtual crowd, the animations were selected from two annotated motion-capture libraries based on acted emotions for full-body motion and rated emotionally by different cultures (see Section 3.2.2). The animations were selected according to two types of agents in the crowd: standing characters for conversational groups and individual walking characters.

18 4.2. ANIMATION

4.2.2. Selection of Emotional Animation The emotions selected were focused in two of the six Ekman basic emotions [11]: happiness and sadness, in addition to a neutral emotional behaviour. Certain stud- ies have proven the facility of recognising these emotions [30], which was the main reason why they were selected. Other emotions such as anger and fear were con- sidered, but they were finally dismissed due to the limits of the research. To clarify these emotional terms, which may be confusing and open to subjective meanings, the following lines present the definitions used in this research for these words:

Happiness: An emotional state in which the individual is in a happy mood and feeling joy, pleasure or contentedness.

Sadness: An emotional state in which the individual is in a sad mood and feeling sorrow, despair or grief.

Neutral: The absence of a noticeable emotion state in an individual, that is, the state of being in a regular mood, not happy nor sad.

As explained in Section 3.2.2, the MoCap libraries used for the animation of the virtual characters were annotated, that is, each animation file included a description of the emotion or expression that was intended to convey. The selection of the emotional animation was done following these annotations. For the animation of walking characters, the tags from which the animations were chosen were happy walk, normal walk and sad walk, the three of them taken from the CMU MoCap database. For the animation of standing characters the selection required additional work, since the conversations were made from a loop or several animation clips. For neutral conversations, several animations with tags that did not express any emotion were taken from the CMU MoCap database. In the case of happy conversations, the neutral animations aforementioned were mixed with emotional animations taken from the UCLIC Affective database, making use of five additional clips tagged as joyful or happy. The same process was followed for sad conversations, taking five additional clips from the UCLIC Affective database tagged as sad or depressed.

4.2.3. FSM for Individual Emotional Behaviour To create the behaviour of each animation state, the animations were blended using hierarchical FSM in Mecanim (Unity’s animation system). Each of the three emotional states that the virtual characters are able to convey (happiness, neutral and sadness) is controlled by an individual FSM. In the second level of the hierarchy, the emotional state of each virtual character determines the FSM that controls its behaviour. Figure 4.3 and 4.4 show the hierarchical FSM for standing and walking characters, respectively. As detailed in Chapter 5, a pre-study allowed to confirm this emotional behaviour, which independently verified that the sad, happy and neutral expressions were perceived as such when mapped onto the mannequin.

19 CHAPTER 4. IMPLEMENTATION

SAD CONVERSATION NEUTRAL CONVERSATION HAPPY CONVERSATION

[Neutral] [Happy] Sad talk 1 Sad talk 2 Listening 1 Talking 2 Happy talk 1 Happy jump 2

Listening 2 Listening 1 Nodding Listening 1 Listening 1 Happy talk 3

Sad gesture Sad gesture Talking 1 Nodding Happy talk 2 Listening 2 [Sad] [Neutral]

Sad talk 2 Sad talk 1 Listening 2 Listening 1 Happy talk 3 Happy Jump 1

Figure 4.3. Hierarchical FSM for the standing characters of the crowd. Note that, at the lowest level, there are no conditions for moving from one state to the next, so the transitions blend the states automatically when the animation clip reach the end. Also, to reduce synchronized movements, the initial state is selected randomly for each character at the beginning of the simulation.

SAD WALKING NEUTRAL WALKING HAPPY WALKING [Neutral] [Happy]

Idle Idle Idle

[Path [Destiny [Path [Destiny [Path [Destiny calculated] reached] calculated] reached] calculated] reached] [Sad] [Neutral]

Sad walking Neutral walking Happy walking

Figure 4.4. Hierarchical FSM for the walking characters of the crowd. Random ve- locities are set at the beginning of the simulation to reduce synchronized movements.

4.2.4. Steering Paths for Walking Characters In the case of the walking characters, it was necessary to find a solution for implementing the steering behaviour, that is, the automatic calculation of feasible walking paths for each virtual agent during the crowd simulation. As commented in Section 3.2.2, a free plug-in for Unity based on the A* path-finding algorithm was used (see Section 2.1.3 for more details about the theory of this algorithm). The graph from which the algorithm calculates the routes for each character was based on the virtual scenario. A navigation mesh of 280x240 nodes was mapped onto the KTH model and, from it, just the side-walk pavements were considered as pedestrian areas in addition to three cross-walks (See Figure 4.5). A total of twenty nodes from the graph mesh were tagged as destiny points, and the virtual characters of the crowd were steered with a script to walk randomly from a destiny point to another (see Section 4.4). This allowed for automatic steering behaviour of the crowd and for the general impression of virtual agents walking around in the scenario. The velocity of each character was also controlled by the script, with some random variability to steer the crowd more heterogeneously.

20 4.3. EMOTIONAL MODEL

Figure 4.5. On the left, top view of the KTH scenario with the navigation mesh on it. The grey zones show the areas in which the virtual characters are able to walk. On the right, snapshot of the crowd simulation.

4.3. Emotional Model

After the virtual crowd was set in the virtual scenario with the basic behaviour implemented (individual behaviour and steering paths), a computational model (called here Emotional Model) was developed in order to make the agents behave emotionally and to implement the emotion contagion effect from which the charac- ters are aware of the mood of others agents, being also able to be affected by them (see Section 2.2.4 for more details about the theory). This emotional behaviour was integrated at the individual agent level. As commented in Section 3.3.1, the work described here builds upon two previous research works in the area of emotion contagion for virtual agents [35, 33].

4.3.1. Internal State for Emotional Characters

Given that the emotional behaviour was implemented as an individual level, the definition of an internal state for each agent was essential. Thus, each character of the crowd has several internal parameters which define its emotional state and other additional characteristics. The current mood of each agent was defined by an integer in a one-dimensional scale in the range [-1, 1] in order to numerically represent each of the three possible emotional states of the model, being -1 sad, 0 neutral and 1 happy. The emotional animations were blended according to this parameter using the FSM described in Section 4.2.3. In addition to that, a secondary parameter was defined to determine the susceptibility of each agent to contagion. This parameter was ranged in a float scale from 0.0 to 1.0. Higher values of this parameter imply more probabilities of being susceptible to catch others emotions. Thus, the emotional state of each agent was represented by the tuple , being e the current value of its mood and s the probability of being emotionally affected by others.

21 CHAPTER 4. IMPLEMENTATION

Inmune time lapse

NO

YES YES PERCEPTION Emotional APPRAISAL CONTAGION Susceptible MODULE agent seen MODULE MODULE

NO

Figure 4.6. Simplified diagram of the Emotional Model.

Apart from the emotional state, other additional parameters were defined for each character in order to adjust, individually, visual aspects such as model scale, animation velocity and emotional colouring. As explained in Section 4.4, all these parameters were adjustable at the beginning of each simulation through a contextual menu.

4.3.2. Algorithm of the Emotional Model

According to the theory of the emotion contagion effect (see Section 2.2.4), the changes in mood emerge from the emotional awareness between the different individuals of the crowd. Thus, to implement this phenomenon in the computational model, it was necessary to make each agent able to perceive, appraise and react to others emotions. This behaviour was implemented in the model through three separate parts: the Perception Module, the Appraisal Module and the Contagion Module. Figure 4.6 shows a simplified diagram of the algorithm. The next lines present the details of the functional aspects of each part:

Perception Module

To implement the Perception module, a field of view was defined for each agent in the Unity engine to represent what the character sees and its field of awareness of others emotions. This field of view was implemented through a ray-casting algo- rithm. To do so, each virtual character has a capsule attached to its model, which represent its area of influence. In addition to this, for each time t of the simulation, each character cast a finite ray in its frontal direction at check for collision with the capsule of other characters. The perception occurs when the ray of an emotional agent, i.e. character A, intersects with the capsule of another agent, i.e. character B. When this happens, character A receives the value e of the emotional state of character B.

22 4.3. EMOTIONAL MODEL

while not intersection do cast ray R; if ray R intersects with capsule of character B then take emotional state of B; go to Appraisal Module; end end Algorithm 1: Pseudo-code of the Perception Module

Appraisal Module When an agent A has perceived the emotion of another agent B, the next step of the algorithm is handled by the Appraisal module. This module is based in the affective appraisal theory briefly defined in Section 1.4. At this stage, if the emotional states of the agents differ, there is an evaluation to check the possibilities of emotional contagion. This evaluation is made using the susceptibility parameter s of the agent A. Through a random function based on an uniform distribution (random()), the model calculates a float number between 0.0 and 1.0 and compares it with the parameter s. Agent A will be affected by the mood of the other if random() < s. To prevent loops and make the behaviour of the crowd more believable, at this stage, and in spite of any possible contagion, agent A sets a time lapse t in which it will be immune to emotional contagion.

if emotional state A != emotional state B then start immune time lapse t; if random() < susceptibility A then go to Contagion Module; end end Algorithm 2: Pseudo-code of the Appraisal Module

Contagion Module In the last step, if the susceptibility was positive in the previous module, then contagion occurs, and the Contagion module handles the change of mood. In this module, two alternative approaches were implemented: strong contagion and con- tagion by steps. As explained in Section 4.4, the contagion approach can be selected before starting the simulation. In the first approach (strong contagion) the character that has been affected simply copies the value of the emotional state of the agent perceived, converging both emotionally. Thus, for example, if a happy character (1) is affected by a sad character (-1), the happy character will change its emotional state to sad (-1).

23 CHAPTER 4. IMPLEMENTATION

The second approach (contagion by steps), is less aggressive in terms of conta- gion, and in this case the emotional state of the character affected is moved one step down/up (-1/+1) in the one-dimensional mood scale, depending on the emotional state of the character perceived. In this case, if a happy character (1) is affected by a sad character (-1), the happy character will change its emotional state one step down, in this case, to neutral (0). Once the contagion is done, the algorithm loops back to the Perception Module when the immune time lapse t sets by the Appraisal Module is over.

if contagion by steps then if emotional state B > emotional state A then emotional state A + 1; else emotional state A - 1; end go to Perception Module when t = 0; else emotional state A = emotional state B; go to Perception Module when t = 0; end Algorithm 3: Pseudo-code of the Contagion Module

4.4. Architecture of the Simulation

As commented in Section 3.2.2, the implementation of the computational model was built up in the Unity Engine. The algorithms for steering paths, animation behaviour and emotion contagion were coded following an scripting methodology in C#. A total of six scripts were written in order to implement the different functionalities of the model (see Figure 4.8):

Spawner: This script runs at the beginning of the simulation to generate the characters of the crowd. Mainly, it sets the initial position of the walking characters and the groups and it sets the initial mood for each character. Walking controller: This is the main script for walking characters. It contains the algorithm to calculate the path from the current position of the character to a certain destiny point of the scenario. This script is based on the A* path- finding algorithm (see Section 2.1.3). Each walking agent of the simulation has its own instance of this script. Animation controller: This is the main script for the emotional behaviour. It contains the parameters regarding the emotional state of the character, as well as the emotion contagion algorithm. Each virtual agent (both walking and standing characters) has its own instance of this script.

24 4.4. ARCHITECTURE OF THE SIMULATION

Figure 4.7. This is an example of how the emotional model works. In image A, a sad agent (red) and a happy agent (yellow) are walking, no-one seeing each other. Then, in image B, the sad agent spots the other when the ray intersects the capsule of influence of the happy agent. After checking for susceptibility, the sad agent is affected, so contagion takes place. Image C shows the result of strong contagion, and in this case the sad agent changes its mood to happy. Likewise, image D shows the result of contagion by steps, changing the mood in this case to neutral (orange).

Emotional trigger: This script triggers an automatic spiral of emotion con- tagion from the centre of the scenario. This was mainly implemented for evaluation purposes, allowing the user to affect the crowd directly.

Simulation stats: This script just keeps the parameters that define the global state of the simulation.

Start Menu: To facilitate the adjustment of the parameters of the simulation from an user perspective, the final model was wrapped up with this script. This script presents a contextual menu at the beginning of the simulation, allowing the user to adjust several parameters such number of walking and standing characters, initial mood, type of contagion and emotional colouring, among others. The main purpose of this script was to facilitate the evaluation of the model an the generation of consistent scenes for the perceptual study.

25 CHAPTER 4. IMPLEMENTATION

startMenu.cs

contagionTrigger.cs spawner.cs simulationStats.cs

[0,n] [0,n]

Walking character Standing group

[2,8]

Standing character

walkingController.cs animationController.cs animationController.cs

Figure 4.8. Diagram of the architecture of the computational model. The script startMenu generates the parameters of the simulation (simulationStats) and launchs the spawner and the contagionTrigger. Then, the script spawner generates the walk- ing characters and the standing groups according to the parameters defined in sim- ulationStats. Each walking character has its own walkingController and animation- Controller. Likewise, each standing character has its own animationController.

Figure 4.9. First screen of the program, where the user can adjust several param- eters of the crowd simulation like the number of walking characters and the number of standing groups, among others (Note that the number of walking characters is independent from the number of standing groups).

26 4.5. IMPLEMENTATION SUMMARY

4.5. Implementation Summary

Chapter 4 has detailed the implementation of the computational model for the generation of virtual crowds and emotion contagion. The models and animations used have been presented, as well as the details of the emotion contagion algorithm and the general architecture of the final simulation program. Next chapter presents the two evaluations carried out for the computational model: the controlled scenario simulations and the perceptual study.

27

Chapter 5

Evaluation

Chapter 5 presents the evaluation of the computational model. This part of the project was performed in two different ways: the first one was based on the study of behaviour of a controlled simulation for a group of conversing characters; the second one was carried out through a perceptual study, which served to handle the research question of this thesis.

5.1. Controlled Scenario Simulations

To verify the emergence of the emotion contagion effect implemented in the com- putational model (Chapter 4), a series of simulations of the model were performed for a conversational group of virtual characters. The purpose of these simulations was to study the emotional spiral patterns from the behaviour of the characters, comparing them afterwards with expected patterns according to the parameters set for each case. These control scenario simulations were based in the work of Pereira et al. [35].

Figure 5.1. Examples of conversational groups for the controlled simulation. The characters were automatically coloured according to their current emotional state: yellow (happy), orange (neutral) and red (sad).

29 CHAPTER 5. EVALUATION

Two different scenarios were studied according to the two types of contagion presented in Section 4.3: strong contagion and contagion by steps. Each scenario was composed by a total of 8 emotional agents (see Figure 5.1) emulating a conversational group, and in a position such as each of them was aware of at least one of the other members of the group. For each scenario there was a total of 20 simulations, each of them running for a maximum of 60 seconds. To test the patterns and the spirals of emotion contagion, the metrics used were basically three: first, the emotional tendency, in percentage, of each of the three possible moods of the simulation; second, the number of simulations that ended up in a balanced mood, that is, with all the characters sharing the same mood once the simulation time was up; and, third, the average time to reach the aforementioned balanced mood. The conditions tested in each scenario were the initial emotional state of each agent (happy or sad) and the general susceptibility value.

5.1.1. Scenario 1: Strong Contagion

Table 5.1 shows the results of these simulations. In this case, the strong conta- gion was between happy and sad, so there were no neutral moods allowed. Here, it is interesting to highlight the consistency in the tendencies according to the initial emotional conditions. Thus, independently of the susceptibility param- eter, it seems clear that the number of happy and sad characters at the beginning of each simulation affects the tendency of the mood, with a higher propensity to be sad when more characters start with a sad mood (75%, 77.5% and 57.5%) in contrast with the opposite condition, in which the balanced mood tends to a happy mood (87.5%, 65% and 75%). Simulations with equal number of initial happy and sad characters keep the tendencies more equilibrated between the two moods.

Tag HC SC Susc HTend NTend STend No.BM AvT A1 2 6 25 25.0 0.0 75.0 9 31.80 A2 4 4 25 37.5 0.0 62.5 6 26.78 A3 6 2 25 87.5 0.0 12.5 10 20.65 B1 2 6 50 22.5 0.0 77.5 14 21.16 B2 4 4 50 55.0 0.0 45.0 12 24.26 B3 6 2 50 65.0 0.0 35.0 14 22.59 C1 2 6 75 42.5 0.0 57.5 18 13.95 C2 4 4 75 52.5 0.0 47.5 18 19.80 C3 6 2 75 75.0 0.0 25.0 20 17.07 Table 5.1. Results of the simulation for strong contagion. [HC: Initial happy char- acters; SC: Initial sad characters; Susc: susceptibility (%); HTend: Happy tendency (%); NTend: Neutral tendency (%); STend: Sad tendency (%); No.BM: Number of balanced mood; AvT: Average time (s)]

30 5.1. CONTROLLED SCENARIO SIMULATIONS

In relation to the number of simulations that ended up with a balanced mood, it is remarkable to highlight the direct relation between this and the susceptibility parameter. For high susceptibility values, the agents are more inclined to change their emotions, which ends up in a spiral of homogeneous emotion in the group. This has an inverse relation with the average time to reach balanced mood, which tends to decrease when the susceptibility has high values. This means that the balanced mood is reached faster when agents are more susceptible to others emotions. From these results it is possible to confirm the expected dynamics of the emotion contagion, and it is possible to infer some patterns according to the initial emo- tional conditions of the characters. Thus, it seems clear that predominant moods tend to pass on the minority of agent with other moods, so predominant happiness or sadness in the group will tend to end up with an overall happy or sad mood, respectively.

5.1.2. Scenario 2: Contagion by Steps

Table 5.2 shows the results of the second scenario of this controlled evaluation. In this case, the model approach tested was contagion by steps, so neutral moods were enabled in the simulations. For this scenario the tendencies were more heterogeneous, although it is possible observe a general consistency between the mood tendencies and the initial emotional conditions of the characters. However, it seems significant the fact that, for this type of contagion, neutral tendency has an strong repercussion in the results and, even for simulation conditions in which happy or sad are the predominant emotions of the group, neutral tendency keeps high values in most of the simulations (the cases of A1 (55%), B1(55%) and C1(65%) for predominant happy emotion; and B3(62.5%), A3 (40%), and C3 (82.5%) for predominant sad emotion).

Tag HC SC Susc HTend NTend STend No.BM AvT A1 2 6 25 0.0 55.0 45.0 8 39.52 A2 4 4 25 7.5 77.5 15.0 5 43.19 A3 6 2 25 50.0 40.0 10.0 7 45.19 B1 2 6 50 2.5 55.0 42.5 12 26.36 B2 4 4 50 25.0 65.0 10.0 12 27.62 B3 6 2 50 30.0 62.5 7.5 10 30.64 C1 2 6 75 5.0 65.0 30.0 19 20.61 C2 4 4 75 10.0 85.0 5.0 19 18.76 C3 6 2 75 17.5 82.5 0.0 18 20.07 Table 5.2. Results of the simulation for contagion by steps. [HC: Initial happy char- acters; SC: Initial sad characters; Susc: susceptibility (%); HTend: Happy tendency (%); NTend: Neutral tendency (%); STend: Sad tendency (%); No.BM: Number of balanced mood; AvT: Average time (s)]

31 CHAPTER 5. EVALUATION

Concerning the number of balanced moods and the susceptibility parameter, there seems to be the same relations found in the previous scenario analysed (Sec- tion 5.1.1): for higher susceptibility values, the number of balanced moods tend to increase and the average time tend to decrease. This emphasize the fact that, for groups in which agents are more willing to be affected by others emotions, the probability to end up in an homogeneous emotion is higher. These results show again the dynamics of the emotion contagion model and, particularly, it is remarkable to highlight the strong effect of the neutral mood when this is taken into account as part of the contagion approach. This emotional state tends to balance the final mood of the conversational groups to a middle point in which happy and sad no longer are predominant.

5.2. Perceptual Study

As a second part of the evaluation, a perceptual study was conducted to test the computational model from real human feedback and also with the aim to find an answer for the research question. This perceptual study consisted of two differ- ent experiments in which several participants were asked about certain aspects of crowded scenes. The next sections describe all the details regarding the design of these experiments (based on the notes presented in Section 3.3.2), as well as the analysis of the data gathered and the discussion of the results.

5.2.1. Definition of the Study As presented in Chapter 1, the problem statement of this research was outlined with the following question:

How is the emotional behaviour of virtual crowds perceived in virtual scenes?

With this research question in mind, the study was focused on studying how viewers perceive emotional behaviour in scenes in which the main context is a virtual crowd. Particularly, the study focused on how this context affects the perception of other elements of the scenes, such as foreground agents that are the main characters of the story. Two perceptual experiments were designed with basis in rating-task question- naires in which participants were asked to rate emotionally several scenes with virtual crowds. The first experiment is a pre-study with the main purpose to val- idate the emotional animation of the virtual characters; the second one, the main experiment, focus in perception of emotional behaviour in crowds. In both of them, an eye-tracker device was also used to record the gaze of the participants, which was useful to validate the correct performing of the tasks (see Section 5.2.5). The target audience of the experiment was not limited by age nor gender, and the only criteria asked to the participants was to be able to watch some videos and to be able to read and write.

32 5.2. PERCEPTUAL STUDY

Figure 5.2. Three examples of the final composition of the scenes, showing happy (left), neutral (middle) and sad (right) behaviour, according to the three emotional states implemented in the computational model.

5.2.2. Stimuli Composition In order to generate the stimuli for the experiments, several scenes with different crowd behaviour were composed with the computational model implemented in this research (Chapter 4). The scenes were composed in two layers: the foreground, which was formed by a group of three standing characters (the main characters of the scene); and the background, formed with the scenario model and a total of 200 characters portraying the contextual crowd, being half of them in small group formations and half of them walking. Small variabilities in scale and velocity were applied to the characters, although they were kept low (within the 0-5% range). The final scenes presented to the participants were rendered in the Unity Engine: six for the pre-study and thirty six for the main experiment (see Section 5.2.4). Before rendering the scenes, the colour effects were adjusted in order to keep them with low emotional influence. The efforts to control it were based in the research of Gao et al. [17], who suggest that lightness and chroma are the main factors influencing emotional perception in colours. Based on this finding, the colour histograms of the scenes where analysed and the lightness and the chroma were modified in order to keep them balanced in middle values. Also, for the virtual characters, the mannequins were coloured with a plain yellow texture instead of a wooden texture, since it has been suggested that yellow has a low emotional influence when its lightness and hue values are moderate [16]. Figure 5.2 shows three examples of final scenes.

5.2.3. Design and Set-Up Both experiments were held in a well lit room free of distractions. The partici- pants were placed 60 cm in front of a large screen (1920x1080) in which the video scenes were played using a desktop computer 1 (see Figure 5.3). A Tobii X1 Light Eye Tracker was used to record the eye-gaze. The chair where the participants were seated was fixed to the floor to maintain a fixed distance from the screen and reduce noise in the eye-tracked data. In addition to this, individual calibrations with the eye-tracker were carried out for each participant. 1Intel Core i7-4770K CPU @ 3.50GHz, 16 GB RAM, Nvidia GeForce GTX 780 GPU, OS Microsoft Windows 7

33 CHAPTER 5. EVALUATION

Figure 5.3. Example of a participant taking part of one of the experiments.

Before starting the experiments, each participant was encourage to read and sign a participant information sheet with some clarifications about the ethics of the experiment (see Appendix B). Also, to facilitate the immersion in the scenes, a contextual story was presented in a comic form with the three foreground characters as protagonists and the crowd as the context (see Appendix B). The intention with this was to focus the partici- pants in the foreground group for the main experiment. To help with the immersion, the scenes were played along with a soundtrack of a background crowd taken from the open audio database Freesound. As commented in Section 3.2.2, the final tests presented to the participants were built in Tobii Studio. In order to make the experiment straightforward, the partici- pants were guided with several explanations and a visual example at the beginning of each experiment. Then, each of the video scenes were presented in random order during 8 seconds, after which a black screen was shown while participants filled in the relevant question on the questionnaire. The participant then pressed the spacebar on the keyboard in order to see the next image.

5.2.4. Experiments The next lines present the details of the experiments that were conducted for this perceptual study:

Pre-Study Before running the main experiment, a pre-study was conducted in order to verify that the annotated animations from the two MoCap libraries used in the virtual crowd (see Section 4.2.2) remained valid in relation to emotional behaviour when mapped onto the androgynous mannequin model. This served both to test the implementation of the virtual crowd and to confirm the suitability of the model for the main experiment.

34 5.2. PERCEPTUAL STUDY

A total of twenty eight participants (22M:6F) took part of this experiment, in which they were asked to rate the emotion of the overall crowd (foreground and background together) on a five point Likert scale (rating from 1 = negative to 5 = positive, where 3 = neutral) as the portrayed emotion was varied in each trial between neutral, happy and sad. The diagram of conditions of this experiment are shown in Figure 5.4.

Main Experiment

The main experiment investigates how the emotional behaviour of the back- ground virtual crowd affects the emotional perception of viewers when they are focused on the three foreground characters of the scenes. The main purpose of this experiment was to find an answer for the research question of this thesis (see Section 5.2.1). A total of twenty participants (17M:3F) took part of this experiment, where they were asked to focus their attention in the foreground characters. Then, they were asked to rate the emotions of these characters while their expressed emotions were varied (neutral, happy, sad) as well as the expressed emotions of the background characters (neutral, happy, sad). The diagram of conditions of this experiment are shown in Figure 5.4. Ratings were made on a five point Likert scale (rating from 1 = negative to 5 = positive, where 3 = neutral).

Conditions Pre-Study

Foreground: Neutral Foreground: Happy Foreground: Sad Background: Neutral Background: Happy Background: Sad

x2 x2 x2

Conditions Main Experiment

Foreground: Happy Foreground: Happy Foreground: Neutral Foreground: Sad Foreground: Sad Background: Happy Background: Sad Background: Neutral Background: Happy Background: Sad

x4 x4 x4 x4 x4

Foreground: Happy Foreground: Neutral Foreground: Neutral Foreground: Sad Background: Neutral Background: Happy Background: Sad Background: Neutral

x4 x4 x4 x4

Figure 5.4. Conditions for the virtual scenes of the pre-study (top) and the main experiment (bottom).

35 CHAPTER 5. EVALUATION

Figure 5.5. This graph shows the results of the pre-study, suggesting that the scenes containing neutral, happy and sad expressive behaviours maintained these emotional qualities when mapped onto a virtual crowd of characters.

5.2.5. Analysis of the Data

Once the experiments were completed and the data were gathered from the participants, a statistical process (based in the analysis of the variance: ANOVA) was conducted in order to extract the results of the study, in addition to a general analysis of the eye-tracker data.

Results of the Pre-Study

Figure 5.5 shows the results of the pre-study. For this experiment, the analysis of the variance for the emotions expressed by all the characters (happy, neutral, sad) gave a strong statistical significance (p ≤ 0.01). In addition to this, the results show that happy (mean = 4.73, s.d. = 0.71) was rated significantly more positive than both neutral (m = 3.07, s.d. = 0.42) and sad (m = 1.73, s.d. = 0.50). Neutral was also rated significantly more positive than sad.

Results of the Main Experiment

Figure 5.6 shows the results of the main experiment. In this experiment, the study of the variance was conducted over the three foreground conditions (happy, neutral, sad) and the three background conditions (happy, neutral, sad). In both foreground and background emotions there was a strong significance (p ≤ 0.01), but with no interaction between background and foreground emotions (p ≤ 0.11). For foreground emotions, the results show that happy (mean = 4.12, s.d. = 0.77) was rated significantly more positive than both neutral (m = 2.95, s.d. = 0.41) and sad (m = 1.81, s.d. = 0.49). Neutral was also rated significantly more positive than sad.

36 5.2. PERCEPTUAL STUDY

Figure 5.6. Results of rating of foreground characters (the dependent variable is the rating of the emotion of the foreground characters). Participants rated foreground characters that made neutral, happy and sad behaviours while the background was varied between neutral, happy and sad.

Foreground (mean, SD) Background (mean, SD) Neutral 2.95 (0.41) 3.03 (1.09) Happy 4.12 (0.77) 3.09 (1.12) Sad 1.80 (0.49) 2.75 (1.09) Table 5.3. Collapsed conditions from the main experiment. Collapsed foreground emotions affect significantly the results.

For background emotions, the results indicate that sad (mean = 2.75, s.d. = 1.09) was rated significantly more negative than both neutral (m = 3.03, s.d. = 1.09) and happy (m = 3.09, s.d. = 1.12). Neutral was not rated differently from happy. Additionally, Table 5.3 shows the mean and the statistical deviation of the con- ditions collapsed both for foreground characters and for background characters. Collapsed background emotions keep a balance close to neutral rating in all the cases, with a slightly difference for sad. In contrast, collapsed foreground emotions affect more significantly the results in each case.

37 CHAPTER 5. EVALUATION

Eye-Tracking Data

As commented in Section 5.2.1, the eye movements of the participants were tracked in order to validate the correct performance of the tasks and the ratings. This allowed to identify potential issues that may have arose in the ratings and also to investigate the overt attention of participants during the tasks. Master heat maps were created over all participants for the different experiments (see Figure 5.7). These heat maps confirmed that the tasks were, in general, performed as expected: when asked to focus the view onto the foreground characters (main experiment, see Section 5.2.4), these characters received more overt visual attention; likewise, in tasks were the attention was asked to be focused on the overall scene (pre-study, see Section 5.2.4), the eye-tracked data was also better distributed over other scene characters. In addition to this, the eye-tracker data allowed to observe some other tendencies about behavioural visual attention, such as the propensity to avoid fixations at the periphery of view and the preference of fixation in the central character in tasks related with the foreground characters. Given the physical conditions of the experiments (distance to screen, screen size and resolution), these tendencies may simply reflect optimal strategies for observing multiple characters at the same time (i.e. fixating the view in the central character facilitate the concurrent observation of the other two characters of the foreground).

Figure 5.7. Master heat maps showing the distribution of the eye-gaze combined over all participants from the main experiment (top) and the pre-study (bottom). These maps demonstrate consistency of eye movements for each task, in the first case (top) relating solely to the foreground characters and in the second (bottom), relating to the overall crowd.

38 5.3. EVALUATION SUMMARY

5.2.6. Discussion The results of the pre-study (see Figure 5.5) shows that the annotated MoCap behaviours mapped onto the characters were perceived as expected (happy was rated as the most positive, followed by neutral, and then by sad). These results allow for validating the animations in the emotional model, so it is possible to state that the characters portray the intended emotional behaviour in each case. In relation with the results of the main experiment (see Figure 5.6), both fore- ground and background emotions affected the rating of the participants. However, as participants were focused in the three foreground characters, the emotional be- haviour of these characters seem to have an stronger effect in the final ratings, as Table 5.3 shows when conditions for foreground and background emotions are separately collapsed. Nevertheless, background emotions had also a slightly effect on participant ratings (see Figure 5.6), being more noticeable for the case of sad (which was significantly more negatively rated than both neutral an happy). This suggest the slightly influence of crowd behaviour even when the centre of attention is focused in the characters of the foreground. It is interesting to highlight here the significant influence of the virtual crowd when the prevailing emotion has a negative valence (sad), in contrast with neutral and positive cases. In Figure 5.6 it is possible to see clearly this effect: when the background emotion is sad, the ratings are significantly lower in comparison with the other cases. The collapsed background emotions in Table 5.3 remarks this effect, being the mean of sad background slightly lower than happy and neutral. This negative effect generated by the background crowd has a possible explanation related with negativity bias, which has already been observed in other studies in relation with this [21, 1, 40]. This effect suggests the predominant tendency of humans at spotting negative behaviours as part of a remaining instinct of self- protection, which explains this general negative effect generated by the crowd. This finding could be an important consideration when attempting to transfer virtual characters to context where the emotional perception could be affected, such as virtual crowds with certain emotional behaviours.

5.3. Evaluation Summary

Chapter 5 has presented the two evaluations carried out for the computational model: first, the controlled scenario simulations in which the emotional behaviour of a group of virtual agent was tested through a total of 40 simulations; and, second, the perceptual study in which the effects of emotional behaviour of virtual crowds were investigated. Next chapter concludes this report with the main conclusions of the research, in- cluding some reflections and some comments about potential future work in related areas.

39

Chapter 6

Conclusions

Chapter 6 closes this report with some final conclusions of the work done during the research and with some comments about potential applications and future work.

6.1. General Findings

This thesis focused on the development of a computational model able to gener- ate virtual crowds with emotional behaviour and emotional awareness. The work of this research was divided, mainly, in two parts: the implementation of the model (in- cluding an algorithm for simulating the emotion contagion effect in virtual crowds), and its evaluation through a controlled simulation and a perceptual study. In parallel to the two previous points, a research question was also addressed in this project:

How is the emotional behaviour of virtual crowds perceived in virtual scenes?

After carrying out the perceptual study presented in this research, and looking at the results and the general discussion of the experiments (see Section 5.2.5 and 5.2.6), it seems that particular emotional behaviours of virtual crowds are generally recognized in virtual scenes, and they also influence the final perception of viewers even when the crowds are in the background as another element of the environment. Also, this seems even more remarkable in cases when virtual crowds portray a negative behaviour, such as the case of being sad. This finding is particularly important in cases where the composition of a particular scene aims to look for the emergence of certain emotions in viewers, in which care must be taken when virtual crowds are one of the elements of that particular scene. Although the answer to this problem statement is based in the scenes rendered from the computational model implemented, it is important to bear in mind the influence of other possible factors that were not studied in this research, particularly in relation with camera movements or different points of view. These considerations are addressed in more detail in Section 6.4.

41 CHAPTER 6. CONCLUSIONS

6.2. Outcomes

As part of the results of this research, the development of the computational model and the performance of the perceptual study allow for the generation of several outcomes, which are listed in the next lines:

Computational model: A final prototype of a functional model able to generate a real-time virtual crowd of androgynous mannequins in the Unity Engine, including three different emotional behaviours (happy, neutral, sad) and an algorithm to simulate the emotion contagion effect between virtual agents. This prototype also allows for some interaction with the cameras and the emotions of the virtual characters, and also for the adjustment of several parameters of the crowd simulation, such as population number, susceptibility of contagion and initial mood, among others.

Stimuli for the experiments: A total of 42 scenes were produced with the computational model developed, being the main stimuli for the two experi- ments of the perceptual study. Each scene lasts 8 seconds and presents three foreground characters and a background crowd of 200 characters, all of them displaying several emotional behaviours.

Research papers: During the development of the project, two research papers were written with basis on the implementation of the computational model and the perceptual study carried out, namely:

• Ramos, M., Peters, C., Qureshi, A. Modelling Emotional Behaviour in Virtual Crowds through Expressive Body Movements and Emotion Con- tagion. Submitted to SIGRAD 2014. • Ramos, M., Qureshi, A., Peters, C. Evaluating the Perception of Group Emotion from Full Body Movements in the Context of Virtual Crowds. Submitted to the ACM Symposium on Applied Perception 2014.

Demos: Three demo videos were rendered during the development of the project, showing the functionality of each prototype (see Appendix A). The third demo also presents the development of the perceptual study and the main results extracted from it (this demo was presented along with the second paper).

Thesis report: The work of the research, including theory, implementation and evaluation of the computational model, was presented thoroughly in this written report.

42 6.3. REFLECTIONS

6.3. Reflections

The next lines present the own reflections of the author of this master thesis, analysing the experience gained during the process, the difficulties encountered and the comments from external reviewers.

6.3.1. Gained Experience

In general terms, the development of the master thesis contributed to an en- riching experience both personally and professionally. The dimension of the project and the necessity of designing a work-plan allowed for the learning and practice of several methodologies in relation with management, implementation and research. It was remarkable as well the knowledge acquired both in the field of computer science and in other interdisciplinary areas related with the project, specially in relation with crowd simulation, emotional behaviour and designing of perceptual studies. Also, the implementation of the computational model was a good oppor- tunity to work with technologies and software such as motion capture libraries, eye-tracking devices, 3D Studio Max and the Unity Game Engine, all of them be- ing tools which gave an invaluable knowledge in the fields of computer science and computer graphics. Finally, it is worth to mention the experience gained in technical writing both through the production of this thesis report and through the development of two technical papers (see Section 6.2) in collaboration with the supervisor and a third professor from the Department of Psychology of Edge Hill University (United King- dom).

6.3.2. Difficulties

Regarding difficulties found during the research, the main problem faced was the difficulty to follow the plan and to meet the deadlines of the prototypes defined for the project (see Appendix A). One of the reasons that delayed the research was the lack of knowledge with some of the tools used in the project (specially the Unity Game Engine and the motion capture libraries), which required additional efforts and learning in order to move forward with the project. Also, the design of the perceptual study required more time than expected, also for the novelty of these kind of experiments and for the necessity of gathering participants. All these issues delayed the work-plan a total of two months. Despite these unexpected problems, the three prototypes defined were finally implemented and the goals of the project (see Section 1.1.1) were met. For future potential projects similar to this, the problems encountered in this research will be useful for the design of more accurate plans, taking into account potential difficulties in relation with software and new methodologies. Also, the experience gained in the search of knowledge for the thesis will contribute to the gathering of relevant sources faster and more accurately.

43 CHAPTER 6. CONCLUSIONS

6.3.3. External Reviews As commented in Section 6.2, two papers were presented to two international conferences (SIGRAD and the ACM Symposium on Applied Perception, respec- tively). Each of these papers was related to one of the main parts of this master thesis, being the first one related to the implementation of the computational model (Chapter 4), and the second one related to the evaluation through the perceptual study (Chapter 5). Appendix C presents a summary of some of the most significant comments received from six international experts in the area of computer graphics and applied perception. From the commentaries of the first paper (Appendix C.1), it is worth clarifying the role of the distance between characters (point 1 and 5) in the computational model. For the implementation of this research, additional features with distance were not considered due to lack of time but, as discussed in Section 6.4, this and other features are considered for potential future work. Also, regarding local avoid- ance (point 3), this feature was not supported in the A* path-finding asset used in the Unity Engine (see Section 3.2.2) and, since the aim of this project was not focused in implementing steering behaviour, occlusion between characters was al- lowed. Nevertheless, certain features such as random variation in velocities and random initial positions in the scenario helped to mitigate this problem in the sim- ulations. In relation to the second paper (Appendix C.2), one of the main commentaries was addressed to the emotional animation and the intensity of each emotion (point 1). This was an interesting comment since the intensity of the emotions were not taken into account, and the Annotated UCL Affective Motion Capture Database (see Section 3.2.2) does not consider this in the annotations. This could be some- thing to take into account in potential future studies. Also, regarding the sound (point 2), the soundtrack used was aimed to help the viewers to get immerse in the scene, being the same for all the scenes, so side effects were not considered for this during the experiments. Finally, it is worth mentioning that certain comments in relation to the analysis of the data (point 3) helped to simplify the results in order to address more clearly the research question of the thesis.

6.4. Final Summary and Future Work

The work presented in this report is the result of a master thesis developed in the area of computer graphics. In general terms, this work can be divided in two different parts: the development of a computational model, and its evaluation through a controlled simulation and a perceptual study. The first part of this project focuses on the development of a computational model for virtual crowds with emotional behaviour and, specifically, in the simula- tion of the emotion contagion phenomenon. For the latter, the algorithm developed has its basis in a previous research work [35]. Although the model presented here has full functionality, the simulation of the emotion contagion effect can be im-

44 6.4. FINAL SUMMARY AND FUTURE WORK proved in several ways. In this model, only three emotional states were taken into account (happy, neutral, sad), but the consideration of more complex emotions (such as anger, panic or surprise, for example) could lead to a more realistic com- putational model in relation to the emotional behaviour of the characters of the crowd. In addition to this, a more complex appraisal module could be designed with non-static susceptibility values or with records of previous contagion for each agent. Also, additional efforts in the personality of each character could bring more variety to the crowd behaviour through the implementation of non-discrete values or multi-dimensional mood scales, for example. In the second part of the project, the perception of the behaviour of virtual crowds is investigated through a perceptual study, particularly the influence of background crowd behaviour in the perception of viewers. The results of this study suggest that, under certain circumstances, background virtual crowds can influence the general perception of a scene. Particularly, this effect seems to be stronger when the behaviour of the crowd has a negative connotation (such as sadness). This arises an important factor to take into account when attempting to convey certain emotions in scenarios in which virtual crowds play as an additional element of the environment. Nevertheless, this study is a first step in the research question addressed in this thesis, but further efforts are needed and additional factors may need to be taken into consideration. Particularly, in this research the virtual camera was fixed to an eye-level point of view, but different camera angles could affect the general perception of the crowd (see, for example, [15]), as well as certain camera movements for non-static scenes. Also, a computational model with more complex emotions and behaviours could bring additional effects in the perception of virtual crowd behaviour. To conclude, it is possible to define some potential future paths of investigation based on this research. In relation to the eye-tracked data captured during the experiments, its only purpose was to confirm the correct performance of the tasks made by the participants, but a more extensive analysis of these data could bring interesting findings in relation to certain characters and motions in the scenes. Also, it could be interesting to investigate gender effects in both participants and virtual characters (see, for example, [43, 28]). Additional improvements in the computational models could also lead to other studies related with perception, such as simulation of panic situations or emergency evacuations. Last, in relation with the outstanding area of virtual reality, immersing the viewers with the virtual crowd and allowing them to interact with the characters could also open an exciting area of investigation not only in the entertainment industry and academia, but also in other aspects related with virtual life and immersive multimedia.

45

Appendix A

Prototypes

Appendix A contains the description of the three prototypes outlined for the implementation of the computational model of this research.

A.1. First prototype

The first prototype presents a single character imported from 3D Studio Max in a basic virtual environment in the Unity Game Engine. Steering paths and basic animation behaviour are implemented.

Features:

Low-level detail character (stickman).

Basic animation for the stickman: idle position, standard walking cycle.

Low-level detail environment.

Environment scenario and stickman imported into Unity (models and anima- tion).

Implementation of steering paths.

A.2. Second prototype

In addition to the features of the first deliverable, the second prototype presents a virtual crowd in the Unity Game Engine and the implementation of the first ver- sion of the emotional model (steering paths and basic controls in crowd behaviour).

Features:

Low-level detail crowd (made from stickman models).

47 APPENDIX A. PROTOTYPES

Environment scenario and stickman models imported into Unity (models and animation).

Steering paths for the walking crowd

First version of the emotional model:

• Basic crowd behaviour: basic animation of expressive motion cues and basic gestures (head position, trunk movement, posture) for walkers and standing groups. • Control in crowd emotions: parameters to control the emotions of the crowd (happy, neutral, sad). These parameters will change the animation of the previous point.

A.3. Third prototype

The final prototype presents the full implementation of the emotional model in the Unity Game Engine, including interactive parameters to control the emotional behaviour of the crowd and the implementation of the emotion contagion effect.

Features:

High-level detail crowd (made from stickman models).

High-level detail environment modelled in 3D Studio Max.

Environment scenario and character models imported into Unity (models and animation).

Steering paths for the walking crowd.

Advances parameters to control the simulation.

Second version of the emotional model:

• Full crowd behaviour: full animation of expressive motion cues and ges- tures (head position, trunk movement, posture) for walkers and standing groups. • Control in crowd emotions: parameters to control the emotions of the crowd (happy, neutral, sad). These parameters will change the animation of the previous point. • Emotion contagion: full implementation of an algorithm able to simulate emotional awareness and emotion contagion at the individual agent level.

48 Appendix B

Participant Sheets

Appendix B presents the information handed in to the participants before start- ing the experiments of the perceptual study (Section5.2). This information mainly relates with the ethical clearance of the research and the instructions of the study itself.

B.1. Ethical Clearance

1. Aims and objectives of the study The aim of the study is to test some aspects about the perception of a crowded virtual scene, and how different setups affect the general impression of the viewers. This experiment will be useful for the development of a master thesis related with this aim.

2. Why have I been chosen? For the purposes of the study it is needed the recruitment of some participants with good vision conditions. They must be able to watch some videos and be able to read and write. These are the only criteria needed.

3. Do I have to take part? No, the participation is voluntary. If you change your mind about taking part in the study you can withdraw at any point during the session. If you decide to withdraw all your data will be destroyed and will not be used in the study. There are no consequences if you no longer wish to participate in the study.

4. What do I have to do? The experiment will last approximately fifteen minutes. You will complete a series of trials, consisting of watching some scenes and answering some general questions. There will be an eye-tracker to capture where you are looking at in the screen, but you will not be recorded whilst participating in the experiment.

49 APPENDIX B. PARTICIPANT SHEETS

5. What are the risks associated with this project? There are no risks associated with this experiment.

6. What are the benefits of taking part? As a student, by taking part in this study you will gain an insight into how a perceptual experiment is conducted and what it is like to be a participant in such a study. You will also gain an insight into the area of perception of virtual agents and virtual crowds.

7. Withdrawal options As mentioned before, if you change your mind about taking part in the study you can withdraw at any point during the session. If you decide to withdraw all your data will be destroyed and will not be used in the study. You can also withdraw up to four weeks after participating by contacting Miguel Ramos Carretero (e-mail: [email protected]) with your participant number.

8. Data protection and confidentiality The data will be confidential. Only the master student and the supervisor of the master thesis will have access to the raw data. All the consent forms will be stored in a separate, secure (locked) location from the raw data itself. You will only be identified by your participant code number. The raw data from the project will be retained until final data analyses are completed. They will then be destroyed. When the data has been entered into a computer file, your scores will only be associated with your code number and access to the file will be password protected.

9. What if things go wrong? Who to complain to? If anything goes wrong or you wish to complain about any aspect of the study, please contact Miguel Ramos Carretero (e-mail: [email protected]) or Dr. Christopher Peters (e-mail: [email protected]) explaining the nature of your complaint.

10. What will happen with the results of the study? The results will be used in a research project.

11. Who has reviewed this study? The study has been reviewed by Dr. Christopher Peters.

12. Further information/Key contact details Miguel Ramos Carretero (e-mail: [email protected]).

50 B.2. INSTRUCTIONS OF THE EXPERIMENT

B.2. Instructions of the Experiment

The following study is a perceptual experiment involving vision. In it, you will be asked about your own perception of several video scenes with virtual agents.

Scenario:

There is a gathering in front of the KTH, and lots of people are waiting for something to happen... today is not a conventional day, and the gathering has been convened by the KTH student association. It seems that they are going to announce something important. Max, one of the students from the KTH, is going to meet another two friends in that gathering. Like the other students, they do not know what the student association is going to announce...

Figure B.1. Contextual story of the experiment.

51 APPENDIX B. PARTICIPANT SHEETS

Process:

After playing each scene a blank screen will appear and you will be asked then to fill a little questionnaire. The experiment is divided in four blocks, and each of them asks you about your own impression of the scene. At the beginning of the experiment there will be an example (scene 0) to show you the process. You will also have the opportunity to write commentaries, suggestions or whatever comes to your mind right after the experiment. Finally, remember that if you change your mind about taking part in the study you can withdraw at any point during the session by immediately informing the experiment supervisor. If you decide to withdraw all your data will be destroyed and will not be used in the study. If something is unclear, please do not hesitate to ask any question before starting the experiment.

52 Appendix C

Critic from Reviewers

Appendix C presents some of the commentaries received for the two papers written as part of this master thesis. These commentaries were made by a total of six international experts when the papers were submitted to the international conferences SIGRAD 2014 and the ACM Symposium of Applied Perception 2014, respectively. Some of this commentaries were very useful to review and improve the work of this project.

C.1. Commentaries from SIGRAD 2014

Main commentaries for the paper Modelling Emotional Behaviour in Virtual Crowds through Expressive Body Movements and Emotion Contagion, submitted to SIGRAD 2014:

1. In relation to the perception module of the computational model, the distance of the characters could consider having an influence on the emotion contagion.

2. The emotional animation (sad, happy, neutral) could be better defined, that is, explaining the body movements of each emotional state.

3. Local avoidance for steering behaviour could have been implemented.

4. The emotion contagion model is rather basic, but it provides a good basis for further development.

5. Details about the emotional contagion algorithm to specify: longitude of the field of view, how the changes in mood are handled, convergence in mood over time.

53 APPENDIX C. CRITIC FROM REVIEWERS

C.2. Commentaries from the ACM Symposium of Applied Perception 2014

Main commentaries for the paper Evaluating the Perception of Group Emotion from Full Body Movements in the Context of Virtual Crowds, submitted to the ACM Symposium of Applied Perception 2014:

1. There is not a clear definition of the intensity of the mood in each case (happy, neutral, sad). For this, valence is only one of the dimensions of emotional experience, but intensity is another important one.

2. The soundtrack of the scenes could have introduced a confounding factor since it did not match the emotion of the crowd.

3. Certain figures included in the paper seem to be not relevant to show the main results of the study. Other figures could give more information.

4. The use of eye-tracking data to validate participant tasks is a good practice since few studies deal with this issue properly.

5. Details to clarify: the stimuli scenes and the repetitions, the proportion of standing and walking characters.

54 Appendix D

Colour Figures

Appendix D presents the figures of the report which require colour to be under- standable.

Figure 2.2. This is an example of how the A* path-finding algorithm works for virtual agents. Supposing that the red rectangle is an obstacle, if the blue agent wants to reach the yellow agent, the A* path-finding algorithm will calculate the shortest path on the graph mesh of the scenario.

Figure 4.5. On the left, top view of the KTH scenario with the navigation mesh on it. The grey zones show the areas in which the virtual characters are able to walk. On the right, snapshot of the crowd simulation.

55 APPENDIX D. COLOUR FIGURES

Figure 4.7. This is an example of how the emotional model works. In image A, a sad agent (red) and a happy agent (yellow) are walking, no-one seeing each other. Then, in image B, the sad agent spots the other when the ray intersects the capsule of influence of the happy agent. After checking for susceptibility, the sad agent is affected, so contagion takes place. Image C shows the result of strong contagion, and in this case the sad agent changes its mood to happy. Likewise, image D shows the result of contagion by steps, changing the mood in this case to neutral (orange).

Figure 5.1. Examples of conversational groups for the controlled simulation. The characters were automatically coloured according to their current emotional state: yellow (happy), orange (neutral) and red (sad).

56 Figure 5.7. Master heat maps showing the distribution of the eye-gaze combined over all participants from the main experiment (top) and the pre-study (bottom). These maps demonstrate consistency of eye movements for each task, in the first case (top) relating solely to the foreground characters and in the second (bottom), relating to the overall crowd.

57

Bibliography

[1] Roy Baumeister, Ellen Bratslavsky, Catrin Finkenauer, and Kathleen Vohs. Bad is stronger than good. Review of General Psychology, 5(4):323–370, 2001.

[2] Tibor Bosse, Mark Hoogendoorn, Michel CA Klein, Jan Treur, C Natalie van der Wal, and Arlette van Wissen. Modelling collective decision making in groups and crowds: Integrating social contagion and interacting emotions, beliefs and intentions. Autonomous Agents and Multi-Agent Systems, 27(1):52– 84, 2013.

[3] G. Castellano, M. Mancini, C. Peters, and P.W. McOwan. Expressive copying behavior for social agents: A perceptual analysis. IEEE Transactions on Sys- tems, Man, and Cybernetics - Part A: Systems and Humans, pages 776–783, 2012.

[4] G. Castellano, S.D. Villalba, and A. Camurri. Recognising human emotions from body movement and gesture dynamics. ACII, LNCS 4738, pages 71–82, 2007.

[5] Ginevra Castellano, Loic Kessous, and George Caridakis. Emotion recognition through multiple modalities: face, body gesture, speech. In Affect and emotion in human-computer interaction, pages 92–103. Springer, 2008.

[6] Rene Coenen and Joost Broekens. Modeling emotional contagion based on experimental evidence for moderating factors. Organising Committee, page 26, 2012.

[7] Douglas W Cunningham. Understanding and designing perceptual experi- ments. In Eurographics 2013-Tutorials, pages t3–undefined. The Eurographics Association, 2013.

[8] Marco De Meijer. The contribution of general features of body movement to the attribution of emotions. Journal of Nonverbal behavior, 13(4):247–268, 1989.

[9] Edsger W Dijkstra. A note on two problems in connexion with graphs. Nu- merische mathematik, 1(1):269–271, 1959.

59 BIBLIOGRAPHY

[10] Funda Durupinar, Nuria Pelechano, Jan M. Allbeck, Ugur Gudukbay, and Norman I. Badler. How the OCEAN personality model affects the perception of crowds. IEEE Computer Graphics and Applications, 31(3):22–31, 2011.

[11] Paul Ekman. An argument for basic emotions. Cognition & Emotion, 6(3- 4):169–200, 1992.

[12] Paul Ed Ekman and Richard J Davidson. The nature of emotion: Fundamental questions. Oxford University Press, 1994.

[13] Hillary Anger Elfenbein and Nalini Ambady. Universals and cultural differ- ences in recognizing emotions. Current Directions in Psychological Science, 12(5):159–164, 2003.

[14] C. Ennis, L. Hoyet, A. Egges, and R. McDonnell. Emotion capture: Emo- tionally expressive characters for games. Proceedings of the ACM SIGGRAPH Conference on Motion in Games, pages 53–60, 2013.

[15] Cathy Ennis, Christopher Peters, and Carol O’Sullivan. Perceptual effects of scene context and viewpoint for virtual pedestrian crowds. ACM Transactions on Applied Perception (TAP), 8(2):10, 2011.

[16] H. Feng, M.-J. Lesot, and M. Detyniecki. Using association rules to discover color-emotion relationships based on social tagging. KES 2010, Part I, LNAI 6276, pages 544–553, 2010.

[17] X. Gao, J.H. Xin, T. Sato, A. Hansuebsai, M. Scalzo, K. Kajiwara, S. Guan, J. Valldeperas, M.J. Lis, and M. Billger. Analysis of cross-cultural color emo- tion. Color Research and Application, pages 223–229, 2007.

[18] Stephen J. Guy, Jatin Chhugani, Changkyu Kim, Nadathur Satish, Ming C. Lin, Dinesh Manocha, and Pradeep Dubey. Clearpath: Highly parallel col- lision avoidance for multi-agent simulation. In ACM Siggraph/Eurographics Symposium on Computer Animation, pages 177–187. ACM, 2009.

[19] B. Hartmann, M. Mancini, S. Buisine, and C. Pelachaud. Design and evaluation of expressive gesture synthesis for embodied conversational agents. In Proceed- ings of the Fourth International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS ’05, pages 1095–1096, New York, NY, USA, 2005. ACM.

[20] E. Hatfield and J. T. Cacioppo. Emotional Contagion. Cambridge university press, 1994.

[21] T. Ito, J. T. Larsen, N. K. Smith, and J. T. Cacioppo. Negative information weighs more heavily on the brain: the negativity bias in evaluative categoriza- tions. Journal of Personality and Social Psychology, 75:887–900, 1998.

60 BIBLIOGRAPHY

[22] Nicholas R Jennings, Katia Sycara, and Michael Wooldridge. A roadmap of agent research and development. Autonomous agents and multi-agent systems, 1(1):7–38, 1998.

[23] Anja Johansson. Affective Decision Making in Artificial Intelligence : Making Virtual Characters With High Believability. PhD thesis, Linköping University, Media and Information Technology, 2012.

[24] G. Johansson. Visual perception of biological motion and a model for its anal- ysis. Perception and Psychophysics, pages 201–211, 1973.

[25] A. Kleinsmith, P.R. De Silva, and N. Bianchi-Berthouze. Cross-cultural differ- ences in recognizing affect from body posture. Interact. Comput., 18(6):1371– 1389, December 2006.

[26] John Lasseter. Principles of traditional animation applied to 3d computer animation. In ACM Siggraph Computer Graphics, volume 21, pages 35–44. ACM, 1987.

[27] Margaux Lhommet, Domitile Lourdeaux, and Jean-Paul Barthès. Never alone in the crowd: A microscopic crowd model based on emotional contagion. In Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology-Volume 02, pages 89–92. IEEE Computer Society, 2011.

[28] M. Mancini, A. Ermilov, G. Castellano, F. Liarokapis, Gi. Varni, and C. Peters. Effect of gender mapping on the perception of emotion from upper body move- ment in virtual characters. In Proceedings of the 16th International Conference on Human-Computer Interaction. HCII, Greece, 2014.

[29] Stacy Marsella, Jonathan Gratch, and Paolo Petta. Computational models of emotion. A Blueprint for Affective Computing-A sourcebook and manual, pages 21–46, 2010.

[30] Joanna Edel McHugh, Rachel McDonnell, Carol O’Sullivan, and Fiona N Newell. Perceiving emotion in crowds: the role of dynamic body postures on the perception of emotion in crowded scenes. Experimental brain research, 204(3):361–372, 2010.

[31] S.R. Musse and D. Thalmann. A model of human crowd behavior : Group inter- relationship and collision detection analysis. In Daniel Thalmann and Michiel Panne, editors, Computer Animation and Simulation ’97, Eurographics, pages 39–51. Springer Vienna, 1997.

[32] J. Pamerneckas, C. Ennis, and A. Egges. Perception of complex emotional body language of a virtual character with limb modifications. Proceedings of the ACM Symposium on Applied Perception, 2013.

61 BIBLIOGRAPHY

[33] YE Papelis, LJ Bair, S Manepalli, P Madhavan, R Kady, and E Weisel. Mod- eling of human behavior in crowds using a cognitive feedback approach. In Pedestrian and Evacuation Dynamics, pages 265–273. Springer, 2011.

[34] Nuria Pelechano, Jan M Allbeck, and Norman I Badler. Virtual crowds: Meth- ods, simulation, and control. Synthesis Lectures on Computer Graphics and Animation, 3(1):1–176, 2008.

[35] Gonçalo Pereira, Joana Dimas, Rui Prada, Pedro A Santos, and Ana Paiva. A generic emotional contagion computational model. In Affective Computing and Intelligent Interaction, pages 256–266. Springer, 2011.

[36] Rosalind W Picard. Affective computing. MIT press, 2000.

[37] C.W. Reynolds. Flocks, herds and schools: A distributed behavioral model. SIGGRAPH Comput. Graph., 21(4):25–34, August 1987.

[38] Julien Saunier and Hazaël Jones. Mixed agent/social dynamics for emotion computation. In Proceedings of the 2014 international conference on Au- tonomous agents and multi-agent systems, pages 645–652. International Foun- dation for Autonomous Agents and Multiagent Systems, 2014.

[39] Wei Shao and Demetri Terzopoulos. Autonomous pedestrians. Graph. Models, 69(5-6):246–274, September 2007.

[40] N. Kyle. Smith, John. T. Cacioppo, Jeff. T. Larsen, and Tanya. L. Chartrand. May I have your attention, please: Electrocortical responses to positive and negative stimuli. Neuropsychologia, 41(2):171–183, 2003.

[41] Daniel Thalmann, Helena Grillon, Jonathan Maim, and Barbara Yersin. Chal- lenges in crowd simulation. In CyberWorlds, 2009. CW’09. International Con- ference on, pages 1–12. IEEE, 2009.

[42] Daniel Thalmann and Soraia Raupp Musse. Behavioral animation of crowds. pages 111–168, 2013.

[43] Katja Zibrek, Ludovic Hoyet, Kerstin Ruhland, and Rachel McDonnell. Eval- uating the effect of emotion on gender recognition in virtual humans. In Pro- ceedings of the ACM Symposium on Applied Perception, SAP ’13, pages 45–49, New York, NY, USA, 2013. ACM.

62