journal name manuscript No. (will be inserted by the editor) 1 A Comparison of Humans and Machine Learning 2 Classifiers Detecting Emotion from Faces of People 3 with Different Coverings 4 Harisu Abdullahi Shehu · Will N. 5 Browne · Hedwig Eisenbarth 6 7 Received: 9th August 2021 / Accepted: date 8 Abstract Partial face coverings such as sunglasses and face masks uninten- 9 tionally obscure facial expressions, causing a loss of accuracy when humans 10 and computer systems attempt to categorise emotion. Experimental Psychol- 11 ogy would benefit from understanding how computational systems differ from 12 humans when categorising emotions as automated recognition systems become 13 available. We investigated the impact of sunglasses and different face masks on 14 the ability to categorize emotional facial expressions in humans and computer 15 systems to directly compare accuracy in a naturalistic context. Computer sys- 16 tems, represented by the VGG19 deep learning algorithm, and humans assessed 17 images of people with varying emotional facial expressions and with four dif- 18 ferent types of coverings, i.e. unmasked, with a mask covering lower-face, a 19 partial mask with transparent mouth window, and with sunglasses. Further- Harisu Abdullahi Shehu (Corresponding author) School of Engineering and Computer Science, Victoria University of Wellington, 6012 Wellington E-mail: [email protected] ORCID: 0000-0002-9689-3290 Will N. Browne School of Electrical Engineering and Robotics, Queensland University of Technology, Bris- bane QLD 4000, Australia Tel: +61 45 2468 148 E-mail: [email protected] ORCID: 0000-0001-8979-2224 Hedwig Eisenbarth School of Psychology, Victoria University of Wellington, 6012 Wellington Tel: +64 4 463 9541 E-mail: [email protected] ORCID: 0000-0002-0521-2630 2 Shehu H. A. et al. 20 more, we introduced an approach to minimize misclassification costs for the 21 computer systems while maintaining a reasonable accuracy. 22 Computer systems performed significantly better than humans when no 23 covering was present (>15% difference). However, the achieved accuracy dif- 24 fered significantly depending on the type of coverings and, importantly, the 25 emotion category. It was also noted that while humans mainly classified unclear 26 expressions as neutral when the fully covering mask was used, the misclassifi- 27 cation varied in the computer systems, which would affect their performance 28 in automated recognition systems. Importantly, the variation in the misclas- 29 sification can be adjusted by variations in the balance of categories in the 30 training set. 31 Keywords COVID-19 · Emotion classification · Facial expression · Facial 32 features · Face masks 33 34 1 Introduction 35 Facial expressions convey information regarding human emotion in non-verbal 36 communication. Thus, an accurate categorization of a person's emotional facial 37 expression allows understanding of their needs, intentions, as well as poten- 38 tial actions. This is valuable insight for psychological experiments, but time 39 consuming to collect manually. Thus, computational methods are needed to 40 analyze facial expressions but how do they differ from human classifiers, espe- 41 cially in different circumstances? 42 Face coverings such as sunglasses and face masks are used to cover the face 43 for different reasons. For instance, people use sunglasses to either protect the 44 eyes from sunlight or to improve appearance, and people use face masks often 45 to prevent the spread of infectious diseases such as the coronavirus (COVID- 46 19) or for hygienic reasons in hospitals or food preparation. However, these 47 coverings not only cover parts of a face but might also affect social interaction 48 as they obscure parts of facial expressions, which might contain socially rele- 49 vant information even if the expression does not represent the emotional state 50 of a person. While sunglasses cover an estimated 10-15% of the face, approxi- 51 mately 60-65% of the face is covered with face masks { exact numbers are hard 52 to tell as this might vary across different people, depending on the size of the 53 face (Carbon, 2020). More importantly, these coverings obscure different parts 54 of the face conveying information about emotional expressions to a varying 55 extent (Roberson et al., 2012). 56 Different studies have shown that humans are far from perfect in accessing 57 the emotional states of people by just looking at their face (Picard et al., 2001). 58 Further, humans are still far from perfect even when it comes to categorizing 59 emotional labels from facial expressions (Lewinski et al., 2014). A further 60 occlusion of the face leads to an even higher decrease in the performance 61 of humans in categorizing emotion from the faces of people. For instance, it 62 has been observed that a partial occlusion of the eyes (Noyes et al., 2021) Emotion classification of covered faces 3 63 impaired the achieved performance by humans especially for categorizing sad 64 expressions (Miller et al., 2020; Wegrzyn et al., 2017; Yuki et al., 2007). On 65 the other hand, while negative emotions such as sadness and fear are the most 66 difficult to categorize by humans (Camras & Allison, 1985; Guarnera et al., 67 2018) compared to positive emotions like happiness (Calvo & Lundqvist, 2008; 68 Hugenberg, 2005; Van den Stock et al., 2007; Wacker et al., 2003), obscuring 69 the mouth leads to less accurate categorization even for happiness (Wegrzyn 70 et al., 2017). 71 More specifically, Carbon (2020) investigated the impact of face masks 72 on humans in evaluating six different categories of emotion (anger, disgust, 73 fear, happy neutral, and sad) from images of people. They found a significant 74 decrease in performance of their human participants across all expressions 75 except for fear and neutral expressions. In addition, emotional expressions such 76 as happy, sad, and anger were repeatedly confused with neutral expressions 77 whereas other emotions like disgust were confused with anger expressions. In 78 accordance with Carbon, several studies confirmed the detrimental effect of 79 face masks on emotion categorization by humans (Grundmann et al., 2021; 80 Marini et al., 2021; Noyes et al., 2021). However, none of these studies directly 81 compared the impact of face masks on humans with machine learning classifiers 82 (computer systems). 83 In contrast to humans, computer systems have done well in categorizing 84 emotion from in-distribution faces images (images from the same dataset) of 85 people, achieving an accuracy of over 95%. Nevertheless, the performance of 86 these systems is greatly affected when it comes to categorizing emotion from 87 natural imagery where the sun is free to shine and people are free to move, 88 changing dramatically the appearance of images received by these computer 89 systems (Picard et al., 2001). For instance, a change in the colour information 90 of the pixels, foreground colour of the whole or a small region of the face, which 91 are nonlinearly cofounded with emotional expressions can lead to a decrease 92 in the performance of these models by up to 25% (Shehu et al., 2020a; Shehu, 93 Siddique, et al., 2021). 94 Furthermore, face coverings such as sunglasses and face masks affect the 95 performance of computer systems by up to 74% (Shehu, Browne, et al., 2021). 96 However, it is unknown to which extent the same type of sunglasses and face 97 masks would affect the performance of humans in direct comparison to com- 98 puter systems in emotion categorization of the same naturalistic stimuli. 99 Moreover, the ecological validity of these computer systems remains ques- 100 tionable as they are mainly trained and evaluated with in-distribution data. 101 We aim to analyze the limitations of humans and computer systems in 102 categorizing emotion of covered and uncovered faces, as well as the misclassi- 103 fication cost of the computer systems: 104 Initially, we will compare the impact of sunglasses and different types of 105 face masks (i.e. fully covered and recently introduced face masks with a trans- 106 parent window (Coleman, 2020)) on humans' and computer systems' ability 107 to categorize emotional facial expressions. This direct comparison using the 108 same stimuli across different emotion categories and different naturalistic face 4 Shehu H. A. et al. 109 coverings addresses open questions about the differential relevance of facial 110 features for both types of observers. 111 With respect to the computer systems, classifier induction often assumes 112 that the best distribution of classes within the data is an equal distribution, 113 i.e. where there is no class imbalance (Grzymala-Busse, 2005; Japkowicz et 114 al., 2000). However, research has shown that this assumption is often not 115 true and might lead to a model that is very costly when it misclassifies an 116 instance (Ciraco et al., 2005; Kukar et al., 1999; Pazzani et al., 1994). This 117 might also be true for emotion classification, e.g. when the fully covering mask 118 is used to cover the face, emotion categories like happiness were classified 119 badly by the computer systems, where the majority of the happy expression 120 images were misclassified as anger expression (Shehu, Browne, et al., 2021). 121 Thus, a secondary objective is to investigate how data distribution effects 122 misclassification between humans and computer systems. This has a practical 123 importance as when positive expressions such as happiness are misclassified 124 as negative emotional expressions like anger or sadness, artificial intelligence 125 systems like robots might refrain from interacting with a person, taking actions 126 on the misbelief that they are angry or sad. Conversely, the computer systems 127 might continuously try to interact with a person if an expression such as angry 128 or sad were misclassified as happiness, which could lead to a potential threat 129 to the safety of humans interacting with these computer systems.