Running head: EMOTION RESIDUE 1 Emotion Residue: Emotion Lingers in a Face Intended to Display Neutrality Daniel N. Albohn & Reginald B. Adams, Jr. The Pennsylvania State University Author Note Daniel N. Albohn, 463 Moore building, University Park, PA 16802, E-Mail [email protected] Reginald B. Adams, Jr., 464 Moore building, University Park, PA 16802, E-Mail [email protected] Correspondence concerning this article should be addressed to Daniel Albohn, 463 Moore building, University Park, PA 16802 E-mail [email protected], or Reginald Adams, Jr., 464 Moore building, University Park, PA 16802, E-Mail [email protected]. EMOTION RESIDUE 2 Abstract Despite the prevalent use of neutral faces in expression research, the term neutral still remains ill-defined. A general assumption is that one’s overt attempt to pose a nonexpressive face results in a neutral display, one devoid of any expressive information. Ample research has demonstrated that nonexpressive faces do convey meaning through emotion-resembling appearance. Here, we examined whether actual prior expressive information lingers on a face, in the form of emotion residue, despite overt attempts to display a neutral face, and whether these subtle emotion cues influence trait impressions. Across three studies we found that explicit attempts at posing neutral displays retained emotion residue from a prior expression. This residue in turn significantly impacted impressions formed based on these otherwise “neutral” displays. We discuss implications of this work for better understanding how accurate impressions are derived from so- called “neutral” faces and underscore theoretical and methodological considerations for future research. Keywords: Neutral face, Neutral, Facial expressions, Face perception, Impression formation EMOTION RESIDUE 3 Emotion Residue: Emotion Lingers in a Face Intended to Display Neutrality It is difficult to think of a social exchange in which the face does not play an important role. With over 40 muscles that can be arranged into more than 7,000 unique configurations, it is a constantly changing canvas that portrays a wide range of social information such as peoples’ emotions, gender, age, and social interest (Hess, Adams, Grammer, & Kleck, 2009; Kalick, Zebrowitz, Langlois, & Johnson, 1998; Rule & Ambady, 2008; Zebrowitz, 2017). Despite its relatively confined space it is the principal vehicle of human expressive exchange. Even those who are face blind (i.e., prosopagnosic) are able to identify emotion cues in a face (Tranel, Damasio, & Damasio, 1988) and appear to use them to make trait impressions (Todorov & Duchaine, 2008). Likewise, neurotypical perceivers tend to use overt expression and even emotion-resembling facial features in otherwise neutral faces to form stable trait impressions of others (Knutson, 1996; Zebrowitz, 2017). Together this work points to emotion cues, even in a neutral display, as a primary source of the impressions we form of others. Expressions can convey basic social motives such as dominance and affiliation, and more complex information such as intelligence and trustworthiness (Adams, Nelson, Soto, Hess, & Kleck, 2012; Knutson, 1996). Social judgments derived from images of the same individual displaying different expressions at two different times varied widely, sometimes even more so than judgments of two different individuals (Todorov & Porter, 2014). This work highlights the primacy of expressions in impression formation even above and beyond appearance cues, and thus cautions against static photos conveying much accurate information. Yet, people purposefully pose for pictures, and in these cases emotional expression may be revealing of one’s actual character. For example, individuals tend to associate anger more so with straight EMOTION RESIDUE 4 males and conservatives, and happiness with gay males and liberals and can use these associations to identify, at above chance levels, the social group memberships others belong to (Tskhay & Rule, 2015). Further, some degree of expressivity, however subtle, is arguably an expected part of an individual’s facial baseline in typical day-to-day interactions. The lack of any expressivity can even be disturbing. For instance in the classic “still-face paradigm” babies become upset when a parent’s expressive face abruptly becomes “blank” (Tronick, Als, Adamson, Wise, & Brazelton, 1978). A lack of expression on inanimate human-like objects (e.g., mannequins, robots, dolls) is often viewed as unsettling despite approaching a life-like human status, the so-called “Uncanny Valley” effect (Mori, 1970). Taken together, it stands to reason that we do not expect the faces we encounter in our social worlds to be completely devoid of emotionality. We are so attuned to expressive cues that when forming impressions of others based on their neutral facial displays, we tend to focus on expression-resembling appearance features (e.g., large, wide eyes, high eyebrows versus small, beady eyes, low hooded eyebrows), an effect referred to as emotion overgeneralization (Zebrowitz, 2017). Thus, emotion-resembling cues can powerfully bias impressions, even when the observer knows the face they are viewing is emotionally nonexpressive. Here we go further to suggest that actual emotion information may be present in these so-called nonexpressive displays, from which we can derive accurate impressions as well. In a typical face, facial muscles are never fully relaxed, and when they are, this can disrupt face processing. Consider, for example, the characteristic drooping due to facial paralysis. Faces with paralysis preferentially bias attention to specific parts of the face and disrupts normal face processing patterns (Dey, Ishii, Byrne, Boahene, & Ishii, 2014). Thus, a EMOTION RESIDUE 5 typical face, even when posed to be emotionally nonexpressive (and even when subsequently rated by others as being “neutral”), may contain in those contracted muscles some subtle, untended emotional tone. If so, this could help explain a growing body of research suggesting that individuals can consistently and accurately derive certain personality traits and social category judgments even from “neutral” faces (see, Rule & Sutherland, 2017; Zebrowitz, 2017). Residual emotional tone in the neutral display could drive many of these judgments. Overview In the present research, we examine the influence of actual, residual emotional tone on impression formation in otherwise neutral displays. We refer to this as emotion residue, which we define as any observable transient emotional tone remaining on a face once an actor has finished making an expression and has intentionally returned to a neutral baseline. We hypothesized that residual cues remaining after an actor makes an expression and then is explicitly instructed to return to neutral would: 1) be detectable on post-expression neutral facial images, and 2) alter impressions of these faces similar to what has been previously found for overt expressive and expressive-resembling cues in faces. We expect that neutral displays made after a negative expression will be associated with more negative trait impressions, while those made after a positive expression will be evaluated more positively. Study 1: Demonstrating That Emotion Residue is Present After Making an Expression Before testing whether individuals are able to detect emotion residue, and if it influences impressions, it is important to examine what residual information, if any, remains on a post- expressive neutral face in the first place. Study 1 examines this by isolating on a pixel-by-pixel level the differences between the pre- and post-expressive images. Study 1 also tests whether EMOTION RESIDUE 6 participants are able to use this information to make meaningful trait impression judgements related to the prior expression. Method Each face used as stimuli in our studies included both a pre- and post-expressive neutral display. Because we had two images per actor, we can extract from each actors’ post-expressive neutral face what is unique to that image by subtracting from it that actors’ pre-expressive neutral face. Since each image is greyscaled, what remains after subtraction is a matrix of values between 0-1 representing the intensity of the pixel. This matrix of pixel intensities is what is unique to the post-expressive neutral, accounting for the pre-expressive neutral (Figure 1). Supplemental Materials 1 reports detailed procedures for the subtraction method, as well as observational results on areas of significant difference between the two images utilizing pixel- wise tests. Power Analysis We analyzed our data in this study using a linear mixed model. Our fixed effect of interest is the interaction between prior expression (anger vs. happy) and impression valence (positive vs. negative) on scores. Therefore, we obtained an effect size from a prior face rating study that utilized linear mixed effects modeling and had two categorical fixed effects variables (Ebner, 2010). Next, we conducted power estimates based on simulations of our model’s critical two-way interaction term by substituting in our model the observed fixed effect value for one that corresponds roughly to that observed by averaging the absolute value of several of the reported fixed effects coefficients in Ebner (2010). Based on these simulated calculations, we estimated that a minimum of 30 participants would be necessary to detect an effect size of 0.31 at 80% power. We estimated our power (based on our sample size) to be 88.90%, 95 % CIs [86.79,