Emotion Algebra Reveals That Sequences of Facial Expressions Have Meaning Beyond Their Discrete Constituents

Emotion Algebra reveals that sequences of facial expressions have meaning beyond their discrete constituents

Carmel Sofer1, Dan Vilenchik2, Ron Dotsch3, Galia Avidan1

1. Ben Gurion University of the Negev, Department of Psychology 2. Ben Gurion University of the Negev, Department of Communication Systems Engineering 3. Utrecht University, Department of Psychology

1 Abstract

Studies of emotional facial expressions reveal agreement among observes about the meaning of six to fifteen basic static expressions. Other studies focused on the temporal evolvement, within single facial expressions. Here, we argue that people infer a larger set of emotion states than previously assumed, by taking into account sequences of different facial expressions, rather than single facial expressions.

Perceivers interpreted sequences of two images, derived from eight facial expressions. We employed vector representation of emotion states, adopted from the field of Natural Language Processing and performed algebraic vector computations. We found that the interpretation participants ascribed to the sequences of facial expressions could be expressed as a weighted average of the single expressions comprising them, resulting in 32 new emotion states. Additionally, observes’ agreement about the interpretations, was expressed as a multiplication of respective expressions. Our study sheds light on the significance of facial expression sequences to perception. It offers an alternative account as to how a variety of emotion states are being perceived and a computational method to predict these states.

Keywords: word embedding, emotions, facial expressions, vector representation

2 Studies of emotional facial expressions reveal agreement among observers about the meaning of basic expressions, ranging between six to fifteen emotional states

(Ekman, 1994, 2016; Ekman & Cordaro, 2011; Keltner & Cordaro, 2015; Russell, 1994), creating a vocabulary conveyed during face to face interactions. . Other studies which used a different approach found 21 expression categories which combine muscle movements of two expressions (e.g., a single image of sadly fearful expression combined muscle movements of sad and fear expressions; Du, Tao, & Martinez, 2014) .

However, if the vocabulary of the emotion-states is indeed limited to that small number of discrete facial expressions, it would have limited people’s ability to express and perceive finer emotions and emotional states. Thus, it is conceivable that people can perceive a much wider collection of emotion states, using a combination of basic, discrete facial expressions. Such mechanisms, which expand a very small number of basic components to a much wider number of meanings, exist in other human communications modalities, such as spoken language and music. For example, in the English language – a combination of 26 letters, forms a rich language of more than 200,000 words (Merriam-

Webster, 2002) and an unlimited number of sentences; More so in western music; a combination of seven notes creates endless musical compositions.

Here, we argue that people infer a larger set of emotion states from facial expressions than previously assumed, by taking into account sequences of facial expressions, rather than single facial expressions.

Some studies already revealed that an expression presented to participants affects their interpretation of subsequent facial expressions (Russell, 1991; Russell & Fehr,

1987; Thayer, 1980). For example, Russell (1991) found that a contempt facial

3 expression, followed by a sadness expression was interpreted as disgust. However, when a disgust expression was presented first, the contempt facial expression was perceived as a sign of sadness. Other studies which focused on the temporal evolvement within single facial expressions (lasting one to two seconds), have found that early in their evolvement, facial expressions allow for crude differentiation of expression cues, while later transmissions consist of more complex cues that support finer categorization (Jack,

Garrod, & Schyns, 2014).

Critically, these previous studies were constrained by focusing on effects of single facial expressions or on the effect of specific facial expressions on the interpretation of subsequent expressions, while a significant social interaction usually includes various facial expressions. As such, they might have overlooked a more general system of rules governing the interpretation of sequences of facial expressions as a whole, whereby sequences of basic facial expressions are mapped onto specific integrated interpretations of emotion states. For example, would a person displaying a surprise, followed by a smile, followed by sadness facial expressions, be perceived as surprised? happy? sad? Or alternatively the integrated interpretation of the sequence, would be helplessness, a breakdown or maybe disappointment.

Several processes might affect one’s interpretation of a sequence of expressions.

One possibility is that during a face to face interaction, observers accumulate the emotional information conveyed by facial expressions. Emotional information is integrated (summed), taking into account the different facial expressions and their duration. Another alternative is that weights are assigned to discrete events

(“snapshots”), based on their location in the sequence, irrespective of their duration

4 (Fredrickson & Kahneman, 1993). The resulted impression would then be the weighted average of the judgments assigned to these snapshots. Studies that focused on scene perception support the latter alternative (Fredrickson & Kahneman, 1993; McDuff, El

Kaliouby, Cohn, & Picard, 2015; Noë et al., 2017; Varey & Kahneman, 1992).

The goal of this study is to model, using a data-driven approach, how the order of serially presented facial expressions, map onto a perceiver’s interpretation of the sequence. We adopted language modeling and feature learning methods from the field of Natural Language Processing (NLP), which rely on computational methodologies and machine learning techniques. There are various approaches used in this domain, for example, the word2vec family of methods (Mikolov, Chen, Corrado, & Dean, 2013)and the GloVe - (Global Vectors for Word Representation; Pennington, Socher, & Manning,

2014). Generally, the application of these methods, termed word embedding, maps words from a text corpus to numerical vectors. Each word in the corpus is represented as a vector in a multi-dimensional word space, enabling the usage of vector arithmetic. For example, one can perform a simple and useful "emotion algebra", by averaging observers’ interpretation of “happy” facial expression (see Figure 1). The expression may be perceived as a sign of happiness by one participant or satisfaction by another, hence represented by the vectors of the words happy or satisfaction respectively. However, averaging the judgment vectors of all participants, may reveal that on average, the

“happy” expression is closest to the vector of the word excitement. By using word embedding methods, here we integrated a mathematical framework with participants’ empirical data, arriving at conclusions, which would have been hard to achieve with traditional statistical approaches.

5 We examined 3 hypotheses: (i) The meaning of sequences of emotional facial expressions is manifested by the weighted average of its discrete constituents, rather than their integration (summation). (ii) The size of the extended vocabulary of emotion expression, based on sequences of emotions, is larger than the number of their discrete constituents, and smaller than the upper bound of all possible theoretical combinations of facial expressions. (iii) Frequent sequences of facial expressions that are more likely to appear in real life (Seger, 1994) elicit greater agreement among observers compared to other, less frequent sequences. We predict that this agreement, reflected in participants’ judgment variability, can be modeled by vector arithmetics.

Figure 1: A 2D (D1, D2) illustration of the averaging process of participants’ interpretation of the “happy” expression, perceived by participants as signs of happiness or satisfaction. By applying vector arithmetics, we find that, on average, participants judge the “happy” expression as a sign of excitement (dashed black line).

To examine these 3 hypotheses, participants were presented with 64 sequences of two facial expressions. They were required to describe, using a single word in free text format, the “state of mind” of the person in the picture, based on all expressions in the sequence.

6 Methods

Participants

Forty-seven participants1 fluent in English (19 female participants) aged 20 to 70

(M= 36.65) took part in an on-line study, conducted from their homes at their own pace using Amazon’s Mechanical-Turk platform. Before the study, participants were requested to repeat in writing the instructions given to them, using their own words. Their descriptions, reviewed independently by two researchers served as an indication for their language fluency and comprehension of the task. Seven participants were excluded from the study because their language was unreasonably not fluent. Participants received a payment of 1 USD, for their participation, which lasted 27 minutes on average.

Stimuli

Stimuli included eight facial expressions consisted of seven most consensual expressions: anger, contempt, disgust, happiness, fear, sadness and surprise (Ekman &

Friesen, 1971) complemented by a neutral expression. Images were created by digitally averaging, using PsychoMorph 5 software (Tiddeman, Burt, & Perrett, 2001), the shape and reflectance of facial expressions portrayed by nine Caucasion female actors in the

Radboud Faces Database (Langner et al., 2010). We used only female faces in order to limit the number of optional sequences, while still employing all eight basic expressions.

Averaging nine different faces expressing the same facial expression yields a stimulus that poses the common facial configuration of a specific expression. The process resulted in images which conveyed eight facial expressions (Figure 2a).

7 Sequences

Each trial (Figure 2b) included a fixation image presented for 2 sec., followed by either one or two images of facial expressions. Each face image was presented for 1 sec, and portrayed one of eight facial expressions followed by a blank image (2 sec).

Design and procedure

Collecting participants’ judgments

The study was composed of two parts. In Part 1, all eight single facial expressions were presented randomly. These single expressions have elicited consensual judgments in numerus previous studies (Ekman & Friesen, 1976; Keltner & Ekman, 2000; Tomkins &

McCarter, 1964), thus establishing a reference for the analysis of the two-expression judgments. At the end of each trial, participants were asked to report the state of mind of the woman, whose face was presented using a single word that “freely comes to mind”.

Free text allows the observer to have complete freedom in their interpretation of the sequence, without being constrained by the a priori hypotheses of researchers. In Part 2, each participant reviewed 28 sequences, randomly selected from the total of 56 combinations of sequences of two images consisting of the eight facial expressions of the first part. The number of sequences per participant, was limited to 28, in order to avoid exhausting the participants. Similarly to Part 1, participants were asked to describe their impressions about the “woman’s state of mind” using a single word but were asked not to repeat the words they used in Part 1, encouraging them to use finer interpretations than those used to describe the basic expressions.

Happi- Anger Contempt Disgust Fear Surprise Sadness Neutral ness

2 1 2 2 1 Presentation 1 2 Presentation duration (sec) duration (sec)

Figure 2: Facial emotional expressions, used in the study and study flow. Figure 2a (top): Composite faces presenting seven basic emotional expressions plus a neutral expression (from left to right): anger, contempt, disgust, fear, happiness, surprise, sadness, neutral. Each composite face is a result of averaging face images of the same nine actors, posing the emotional expressions. Figure 2b(bottom) –Illustration of experimental trials. Left: timeline of a single expression trial presented in Part 1; Right: timeline of two expressions trial presented in Part 2.

9 Results

Pre- processing participants’ judgments

We collected 1920 words which described the “woman’s state of mind” as perceived from the 64 image sequences, collected in Part 1 and 2 as a whole. Words were pre-processed, eliminating word duplications, fixing spelling mistakes and transferring all verb-based impressions to a noun form.

The analysis yielded 372 unique words descriptors, which represented each of the

1920 original words. Lastly, we replaced every word from the original list (of 1920) by its unique word descriptor from the list of 372 words.

Transforming participants’ responses to vectors

To assign vector representation to each of the 372 words, we used the Twitter word corpus, relying on the most up-to-date embedding approach termed GloVe -

(Global Vectors for Word Representation; Pennington et al., 2014). These authors have already transformed (using word embedded) Twitter’s word-corpus data to vectors. The corpus included two billion tweets used for the vectors training, resulting in 250,000 words. Each word was represented by 50-dimensions, thought to represent various characteristics of the word. The Twitter word corpus was chosen for the present study because its writing style is non-edited-real-life, short, concise and emotional, presumably similar in nature to participants’ free text words. For each of the 8 stimuli of single facial expressions and 56 sequences in Part 1 and 2 respectively, we averaged participants’ judgments vectors (across each of the 50 dimensions) and normalized them for similarity and for enabling dot product calculations (Levy, Goldberg, & Dagan, 2015).

10 Predicting emotion states from sequences of facial expressions by applying vector

representations.

To test our first hypothesis suggesting that sequences of emotional facial expressions have meaning beyond their discrete constituents, we conducted a multiple regression analysis in which we predicted participants’ average judgments of the sequences, using the first facial expression (FE-T1) and the second facial expression (FE-

T2) and their (scalar) interactions. FE-T1, FE-T2 and participants’ judgments were represented by 50D vectors. We added 16 dummy variables (dT1.1 – dT1.8, dT2.1- dT2.8) to account for the effect of different sequences of facial expressions. (for more details see Table 1 of Supplementary data). Each of the dT1’s and dT2’s represented a different expression out of eight facial expressions presented to participants as first or second expression of the sequence, respectively.

We structured the data matrix for the regression such that the criterion was the average judgment vector (one element per dimension) and the predictors were the vectors

FE-T1, FE-T2, and their interaction FE-T1. FE-T2, as well as the 16 dummy variables for each of the facial expressions, F (17, 3182) = 444.2, p < .001, R2 = .703. None of the effects of the dummy variables were statistically significant (smallest p >.14) and therefore they were removed from further analyses. Importantly, both facial expressions affected the meaning that the participants ascribed to the sequences of facial expressions,

β1 = 0.38, p < .001, β2 = 0.52, p <.001. The effect of the second expression was stronger compared to the first, with a significant difference between them, F (1) = 48.88, p

< .001. These findings confirmed our prediction that the overall meaning of a sequence of facial expressions is the weighted average of the discrete constituents, and that the

11 order of the expressions affects the interpretation of the scene. It also confirmed our prediction that vector representation of sequences of two facial expressions predict participants’ judgments. Moreover, the interaction between EF-T1 and EF-T2, β = -0.07, p < .001, reflected the existence of a third, semantic component, pointed by the interaction vector. This third component complemented the two expressions in creating the meaning of the sequence, which was more than the sum of the visible parts.

The size of the vocabulary of emotion states

To test our second hypothesis suggesting that the number of emotion states, derived from the sequences of facial expressions, is larger than the lower bound of the discrete seven basic emotional expressions and smaller than the upper bound of the possible 64 combinations, we computed the semantic interpretation of participants’ average judgment vectors. We used the k- nearest neighbor (k-NN) analysis algorithm(Li, Harner, & Adjeroh, 2011), a non-parametric machine learning method for classification. Unlike the more standard usage of the NN- algorithm, here we used k-NN as a mean for examining the Euclidian distance between the average vectors of participants’ judgments and the base vectors we obtained (i.e. 372 words).

For each of the 64 average vectors (i.e., participants’ judgement data), the k-NN algorithm created a list of five nearest emotion states (k=5), from the list of the 372 unique word vectors obtained as described above, (i.e., Twitter’s data; Table 1 and Table

2 of Supplementary data). At k=5, nearest neighbor analysis yielded 41 emotion states, over and above the discrete seven emotional states (excluding neutral expression). At k=3 the analysis yielded a total of 38 (31 new) emotion states- 84.8% of total emotion states at k=5. A closer look shows that the increase rate of emotion states, sharply dropped from

12 87.5% at K =2 to 26.6%, 13.2% and 11.6% at K=3,4 and 5 respectively (Figure 3; for the

list of words see Table 3 of Supplementary data). These results confirmed our prediction

that the number of emotion states exceeds the number of its discrete constituents, but

does not reach its upper theoretical bound (i.e., 64). The finding is related to the fact that

participants ascribe similar interpretation to different sequences (e.g., both surprised- sad

and disgusted-happy sequences, were interpreted as a sign of anger).

Discrete Discrete Facial Nearest Nearest Nearest Nearest Nearest Facial Expression Emotion State Emotion State #2 Emotion State #3 Emotion State #4 Emotion State #5 Expression #2 #1 #1 1 Disgusted Disgust Disgust Disbelief Sympathy Anger Embarrassment 2 Disgusted Neutral Despair Disgust Hopelessness Disappointment Nervousness 3 Disgusted Fear Scare Fear Harm Anger Sickness 4 Disgusted Surprise Disbelief Embarrassment Despair Disgust Disappointment 5 Disgusted Happiness Anger Disappointment Happiness Misery Frustration 6 Disgusted Contempt Hopelessness Apathy Despair Disbelief Disgust 7 Disgusted Anger Anger Fear Frustration Disappointment Harm 8 Disgusted Sadness Despair Disappointment Frustration Anger Annoyance

Table 1: Examples of the five nearest emotion states resulting from the nearest neighbor

algorithm. The first discrete expression is disgust and the second expression is one of the eight

facial expressions. For the full list, see Table 2 supplementary data.

An important issue is the extent to which the emotion states, selected by the k-NN

algorithm, represent participants’ interpretation of facial expressions. To test this, we

computed, for each emotion sequence, the Euclidian distance (D) between the vector of

the emotion state selected by k-NN algorithm, and the average vector of participants’

interpretations of that sequence. First, we computed, participants’ average interpretation

of each of the seven basic consensual emotion states of Part 1 (see Figure 4 for an

illustration of one emotion state).

13 Increase rate of number of emotion states 140 120 )

% 100 (

t 80 a r

e 60 s a e

r 40 c n I 20 0 1 2 3 4 5 Nearest Neighbor columns

Figure 3: Increase rate of the number of unique emotion states within the five nearest neighboring emotional words (k-NN analysis, k=5). There were 16 new unique emotion states at k= 1, compared with the seven basic expressions at k=0 (not shown) – an increase of 128%. At K=2, 14 additional new unique emotion states were added, an increase of 87.5 % relative to K=1. The increase rate of emotion states, sharply dropped from 87.5% at K =2 to 26.6%, 13.2% and 11.6% at K=3,4, 5respectively.

Since there is an agreement among observers and scholars about the meaning of these expressions (Ekman, 2016; Ekman & Friesen, 1969; Keltner & Cordaro, 2015), they can serve as a baseline for evaluating the the extent to which the rest of the 56 emotion states (Part 2), represent participants’ interpretation of these sequences.

Computing D values of the seven basic consensual emotion states, we found that the mean Euclidian distance (Dmean= 0.245, SD= 0.135) was significantly different from D =

0, t (6) = 4.471, p < 0.005. Among the seven basic emotions, the vector representations of anger, disgust, happiness and sadness, were the closest to the average vector of participants’ interpretations of these emotional expressions, Dmean= 0.139, SD= 0.047 and

14 not different from D = 0, t(3) = 5.871, p =0.1. These results confirmed that emotion states, selected by the k-NN algorithm, represent participants’ interpretation of facial expressions.

Figure 4: A 2D (D1, D2) illustration of vector representation of participants’ judgments, judging a happy facial expression. The vector of the word “happy” was the closest to the average vector of participants’ judgments of the happy facial expression (dashed blue line). Repeating the process for the 56 sequences of two expressions, we computed the average interpretation of each sequence. The mean distance between vector representations of the words which best describe the expression sequences (resulting from k- NN computation) and the average vectors of participants’ judgments was Dmean=

0.517, SD= 0.080. This distance was greater than the mean Euclidian distance of the basic seven emotion states, Dmean= 0.245, t (55) = 25.202, p < 0.001, thus indicating that the alignment between participants’ perception of facial expressions and the semantic meaning of the words describing them is greater for single basic expressions, than for sequences of two- expressions. It is possible that these differences arise as some sequences of facial expressions, that are not likely to appear in real life, elicit low agreement among observers compared to other, more frequent sequences.

15 The degree to which participants agree on a certain emotion state is an important measure of the social meaning of this state, affecting its viability as a social communication signal. Therefore, it is important to predict and model the degree of agreement. We tested whether inter-relation between vectors of the constituting components of sequences predicts agreement across participants. We used participants’ judgment variability as an operational definition of agreement, such that high variance indicated low agreement, while low variance indicated high agreement.

A multiple regression analysis was conducted, similarly to the analysis described above for predicting emotion states from sequences facial expressions, but with participants’ judgment variance vector as a criterion (one element per dimension). The criterion reflected the variability in participants’ interpretations of the various expression sequences. As expected, the interaction between vectors of two facial expression FE-T1 and FE-T2 predicted the participants’ judgment variability and agreement, F (3, 3196) =

2 556, p < .001, R = .342; β = 0.55, p < .001. There were also main effects β1 = 0.04, p

= .039 and β2 = 0.05, p <.01 with very small contributions to the overall effect, compared to the interaction effect. These results support the view that the product of vector representations of two sequential emotion states, is a good proxy of participants’ actual agreement about the judgments, indicating the likelihood of a sequence to appear in real life.

Discussion

In the present study we demonstrated that people use basic expressions as a basis for a much larger vocabulary of emotion states. Richness of emotion states, based on facial expressions, is a result of using combinations of the discrete facial expressions. Our

16 findings revealed 32 new emotion states, in addition to the seven basic expressions, used in the study (totaling in 39 emotion states).

To test our hypotheses, we developed a computational framework, utilizing language modeling and feature learning methods, used in Natural Language

Processing (NLP) and relying on computational methodologies and machine learning techniques. Using vector representation of emotion states and performing simple algebraic vector operations, we found consistently with our prediction, that participants’ interpretation of the sequences of facial expressions, represented by their respected vectors, was expressed as a weighted average of the single expressions comprising them.

This has resulted in new meanings that differed from the original meanings ascribed to each of the expressions separately. We also found an interaction between the first and the second emotion expressions. Having its own semantic meaning (i.e., the word, pointed by the interaction vector), the interaction adds to the meaning of the two expressions in creating a combined meaning to the sequence. Importantly, this meaning is more than the sum of its two visible parts.

To model judgement agreement, we conducted an additional analysis, in which the first and the second facial expressions and their interactions, predicted participants’ judgment variance - a measure of participants’ agreement about the meanings of the expression sequences; high variance indicated low agreement, while low variance indicated high agreement. As expected, the interaction between the first and the second emotion states, represented by their respected vectors, predicted participants’ judgment agreement, a possible indication of participants’ expectations to encounter certain sequences of facial expressions in daily life interactions.

17 Our findings support the notion offered in the context of experienced utility

(Fredrickson, 1991; Kahneman, 1999) that weights are assigned to discrete moments

(“snapshots”), based on their order of appearance. Our results show that the snapshot of the second facial expression is weighted more than the first one. Future studies are still warranted in order to better understand this issue. For example, testing participants’ affective responses after reviewing three or more “snapshots”, would enable us to reveal whether more snapshots increase the potential vocabulary of facial emotional meanings and how these snapshots are weighed, when forming the perceived meaning. Measuring the effect of N-expression sequences on the interpretation of person’s state of mind is not a scalable method of measurement, because the number of possibilities grows exponentially with N. Therefore, finding a computationally trackable framework to evaluate this effect is of great interest.

In sum, our results go beyond existing studies which investigated various aspects associated with emotional states such as the number of basic facial expressions (Ekman,

1994, 2016; Ekman & Cordaro, 2011; Keltner & Cordaro, 2015; Russell, 1994), the combined muscle movements of two expressions within a third expression (e.g., a single image of sadly fearful expression combined muscle movements of sad and fear expressions; Du et al., 2014) and studies which focused on the temporal evolvement within single facial expressions (Jack et al., 2014).Thus, the present study demonstrated that people use basic facial expressions as a basis for a much larger vocabulary of emotion states. These findings offer an alternative account as to how the variety of emotion states, such as excitement, awe or sorrow, are being produced and perceived.

18 Utilizing computational methods which enable vector representation of emotion states, we have shown that computed algebraic relationships of vectors of two emotional expressions, predict participants’ measured inferences of emotion states. Participants’ interpretation of the sequences of facial expressions was predicted (mostly) by the weighted average of the vectors representing the constituting expressions; Participants’ actual agreement about the interpretations was predicted by the interaction of these vectors.

Acknowledgement We acknowledge the financial support of the ABC robotics initiative at the Ben Gurion University of the Negev and The Israeli Ministry of Defense. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We thank Tal Weisman for data collection.

Notes

1. The word embedding process was based on Twitter’s word corpus, which included two billion tweets used for the vectors training, resulting in 250,000 words. This huge sample size ensures high reliability of the computed vectors, regardless of the participants’ sample size used in the present study. As the vectors were trained separately from the present study, we regard them as having constant values when performing the regression analyses.

19 References

Du, S., Tao, Y., & Martinez, A. M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1322355111 Ekman, P. (1994). Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. Ekman, P. (2016). What scientists who study emotion agree about. Perspectives on Psychological Science, 11(1), 31-34. Ekman, P., & Cordaro, D. (2011). What is meant by calling emotions basic. Emotion review, 3(4), 364-370. Ekman, P., & Friesen, W. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1(1), 49-98. Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124. Ekman, P., & Friesen, W. V. (1976). Measuring facial movement. Environmental psychology and nonverbal behavior, 1(1), 56-75. Fredrickson, B. L. (1991). Anticipated endings: An explanation for selective social interaction. ProQuest Information & Learning. Fredrickson, B. L., & Kahneman, D. (1993). Duration neglect in retrospective evaluations of affective episodes. Journal of Personality and Social Psychology, 65(1), 45. Jack, R. E., Garrod, O. G., & Schyns, P. G. (2014). Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time. Current biology, 24(2), 187-192. Kahneman, D. (1999). Experienced Utility and Objective Happiness: A Moment-Based Approach. Keltner , D., & Cordaro, D. (2015). Understanding Multimodal Emotional Expressions: Recent Advances in Basic Emotion Theory. Emotion Researcher. Keltner, D., & Ekman. (2000). Facial expression of emotion. Handbook of Emotions Lewis, M. & Haviland-Jones, J.M.,, 8, 236-249. Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D. H., Hawk, S. T., & van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8), 1377-1388. Levy, O., Goldberg, Y., & Dagan, I. (2015). Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3, 211-225. Li, S., Harner, E. J., & Adjeroh, D. A. (2011). Random KNN feature selection-a fast and stable alternative to Random Forests. BMC bioinformatics, 12(1), 450. McDuff, D., El Kaliouby, R., Cohn, J. F., & Picard, R. W. (2015). Predicting Ad Liking and Purchase Intent: Large-Scale Analysis of Facial Responses to Ads. IEEE Transactions on Affective Computing, 6(3), 223-235. Merriam-Webster, D. (2002). Merriam-Webster. On-line at http://www. mw. com/home. htm. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. CoRR abs/1301.3781. URL http://arxiv. org/abs/1301.3781. Noë, B., Turner, L. D., Linden, D. E. J., Allen, S. M., Maio, G. R., & Whitaker, R. M. (2017). Timing rather than user traits mediates mood sampling on smartphones. BMC Research Notes, 10(1), 481. doi:10.1186/s13104-017-2808-1

20 Pennington, J., Socher, R., & Manning, C. D. (2014). Glove: Global Vectors for Word Representation. Paper presented at the EMNLP. Russell, J. A. (1991). The contempt expression and the relativity thesis. Motivation and emotion, 15(2), 149-168. Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological bulletin, 115(1), 102. Russell, J. A., & Fehr, B. (1987). Relativity in the perception of emotion in facial expressions. Journal of Experimental Psychology: General, 116(3), 223. Seger, C. (1994). Implicit learning. Psychological bulletin, 115(2), 163-196. Thayer, S. (1980). The effect of facial expression sequence upon judgments of emotion. The Journal of Social Psychology. Tiddeman, B., Burt, M., & Perrett, D. (2001). PsychMorph Version 5; Prototyping and transforming facial textures for perception research. Computer Graphics and Applications, IEEE, 21(5), 42-50. Tomkins, S. S., & McCarter, R. (1964). What and where are the primary affects? Some evidence for a theory. Perceptual and motor skills, 18(1), 119-158. Varey, C., & Kahneman, D. (1992). Experiences extended across time: Evaluation of moments and episodes. Journal of Behavioral Decision Making, 5(3), 169-185.