Evaluation of the effect of mimicry on facial expression in Avatar-Mediated Communication Mikihiro Suda1, Mizuki Oka1 1University of Tsukuba, Tsukuba, Ibaraki 305-8577 Japan [email protected] Abstract will continue, along with the concomitant spread of avatar Downloaded from http://direct.mit.edu/isal/proceedings-pdf/isal/33/48/1929975/isal_a_00394.pdf by guest on 24 September 2021 use. Against the background of remote work and labor-saving The process of creating, exchanging, and perceiving in- being promoted globally, the use of avatars is becoming widespread in our daily lives. Concurrently, the environments formation by encoding, transmitting, and decoding mes- in which avatars are used are also diversifying, with environ- sages using a computer is called computer-mediated com- ments appearing wherein communication is possible between munication (CMC) (Jonassen and Driscoll, 2003). Avatar- humans and avatars as well as between avatars themselves. In mediated communication (AMC) is a typical CMC because this social situation, the effects of use of avatars on commu- it uses computers to display avatars and transmit various in- nication must be investigated. However, research to compare the effects of non-verbal information in avatar–avatar and formation (L. and Fox, 2018). In the context of CMC, re- human–avatar environments is inadequate. In this study, we searchers have studied the types and number of informa- created an avatar of which every facial feature can be moved tion exchanged and the attributes of the people engaging independently and then measured the effects of facial expres- in CMC. Non-verbal expressions exchanged in communi- sions on communication in avatar–avatar and human–avatar cation are known as social cues. Social cues such as gesture, environments. The results of a communication experiment based on negotiating in the context of the Prisoner’s Dilemma gaze, head movement, and breathing are fully exchanged in game showed that mimicking facial expressions resulted in face-to-face communication; however, in CMC, social cues negotiations having a more cooperative outcome. Further- are partially lost because the communication is processed by more, the results suggest that this tendency is stronger in the the computer. Based on the loss of social cues, many stud- avatar–avatar environment. ies have reported on the inferiority of CMC compared to face-to-face communication (Walther and Parks, 2002). For example, Rutter and Stephenson (1979) reported that the Introduction number of social cues that can be exchanged has a signif- Covid-19 pandemic affected human communication signif- icant effect on communication (cuelessness model). These icantly. As face-to-face communication was discouraged, studies reporting the recessive nature of CMC based on the video chatting via the Internet such as, the Zoom video loss of social cues have been summarized as the cue filtered- conferencing and Microsoft Teams applications, became in- out model, and it is now widely known that deficits in social creasingly used, and tools such as FaceRig1 and Snap Cam- cues by computer negatively affect communication. 2 era , which introduce avatars into video chatting, attracted However, as multimedia develops, some argue that the greater attention. These tools use web cameras to analyze inferiority of CMC is decreasing. With multimedia usage facial positioning and user’s facial rotation while making spreading, a concept called social presence has been attract- expressions, and then locate and introduce virtual objects ing attention. Biocca (1997) defined social presence as “the (avatars) into the video data. Among them, the use of Fac- subjective feeling of being with a real person and being able eRig is growing remarkably, with the total number of users to touch his or her thoughts and feelings”. In their review pa- 3 in 2020 being 275% larger than in 2018 . In part, this growth per on social presence, Oh et al. (2018) reported that many in the use of avatars is due to the social ubiquity, lower cost, studies suggest that multimedia enhances social presence and higher performance of personal computers (Reeves, compared to text-only communication. In other words, mul- 2012). The expected trend following the COVID-19 epi- timedia brings CMC closer to face-to-face communication. demic is that the increasingly widespread use of computers In addition to multimedia, avatar-mediated communica- 1https://facerig.com/ tion (AMC) is thought to have the same effect to eliminate 2https://snapcamera.snapchat.com/ the recessive nature of CMC (Bente et al., 2008). This 3https://steamdb.info/app/274920/ (Retrieved on 2021-01-10) is because AMC allows more social cues than a traditional medium such as text or sound as avatars can reproduce and a lack of research on the effects of avatar performance and transmit non-verbal expressions such as facial expressions accuracy on communication. and gestures. To understand the characteristics of AMC, we The most important characteristic of the avatar–avatar en- need to understand those of face-to-face communication. vironment is that the self is represented as an avatar. More- Before the CMC study, face-to-face communication was over, as the user’s face and the communicating person’s face the subject of many psychological studies and numerous re- were juxtaposed in the video chatting interfaces, the users ports have been made thereon. As mentioned above, non- tended to view their own face during the communication. verbal expressions exchanged in face-to-face communica- In this environment, it is important to investigate and under- tion are also known as social cues, and include gestures, stand the sense of ownership; the sense in which the avatar is breathing, facial expressions, head movements, etc. Pent- a part of myself, and the sense of agency; the sense in which land (2010) especially focused on non-verbal expressions the communication outcome results from my actions. In this which are exchanged unconsciously, which he called honest study, we investigate how mimicry affects communication in signals. He argues that we can accurately predict the out- both the avatar–avatar environment and human–avatar envi- come of communication by analyzing face-to-face commu- ronment, which mimics the video chatting systems. Does nication of short duration based on the four components of the mimicry effect arise in the avatar–avatar environment Downloaded from http://direct.mit.edu/isal/proceedings-pdf/isal/33/48/1929975/isal_a_00394.pdf by guest on 24 September 2021 honest signals: mimicry, activity-level, cooperation, and agi- and human–avatar environment similar to the face-to-face tation. For example, Curhan and Pentland (2007) shows that environment? In this study, we conducted an experiment of a task requiring high-level cooperation such as salary ne- two-party communication based on free-talk negotiation us- gotiations with one’s boss demands high cooperation, high ing the Prisoner’s Dilemma scenario. mimicry, and low agitation. Or, Stoltzman (2006) shows that a task requiring high-level leadership such as selling a new Avatar business plan calls for high activity-level, high cooperation, Existing Avatars and low agitation. Mimicry, which Pentland (2010) refers to as one of the Avatars on the open market have become highly diverse in honest signals, is the phenomenon of humans unconsciously recent years. In this section, we give an overview of these imitating the words and actions of others in communication. avatars in Table 1. We categorize them as static or dynamic, He argues that mimicry has the effect of increasing positive then the dynamic avatars are further classified by their real- feelings and trust. It is known that some animals includ- ism and operability. ing humans instinctively mimic others, and some researchers Static avatars are of the type for which users create an like Masumori et al. (2021) focus on the learning effect of illustration of themselves and send or receive it as an image mimicry. or a short video via chat or messaging system. Facebook The effect of a non-verbal expression like mimicry is Avatar by Facebook 4, LINE Avatar by LINE 5, Bitmoji one of the targets of avatar-mediated communication(AMC) by Snap 6 are classified as static avatars. Using these tools, studies. Especially, mimicry is one of the most focused users can create avatars by combining prepared illustrations study targets and many studies suggest that mimicry has of body parts. Of static avatars, it can be said that they are some effect on communication despite it being uncon- Emoji variants since they are already-created images. sciously communicated to participants (Hale and Hamil- Dynamic avatars are the type that follows the user’s be- ton, 2016; Bailenson and Yee, 2005). Bogdanovych et al. havior in real-time and can be processed as a streaming (2018) conducted an experiment based on the Prisoner’s video. Memoji by Apple 7, MetaHuman by Epic Games Dilemma game using an avatar and mimicry and found that 8, open-source software MakeHuman 9, Japanese platform mimicry increases the smile quotient of communication par- Live2D 10 and Reality 11 are classified as dynamic avatars. ticipants. According to such studies, we can evaluate that Dynamic avatars are further classified by their life-like ap- non-verbal expression such as the honest signals affect face- pearance. Memoji, Live2D, and Reality provide anime- to-face communication and also (perhaps partially) avatar- like avatars. Live2D, especially, is a tool