Multi-label emotion detection in Twitter Vannesa Rinny Berhitoe July 2017 Data Science in Action - Master’s Thesis Data Science: Business and Governance Faculty of Humanities Tilburg University, Tilburg Thesis supervisor : Chris Emmery Second reader : prof. dr. E.O. Postma i Preface I would like to dedicate the thesis to my mother and father in heaven, I hope I make both of you proud. I would like to thank my dearest friends, Deve, Krista, Emily, my siblings, Ricardo and Re- becca, my family in GKIN Tilburg and in Indonesia, especially my partner Nofardo, for the end- less support, both spiritually and materially. My greatest appreciation for Chris Emmery for his undertanding, advices, support, critics and in supervising the entire process of this thesis. July 2017 (Vannesa Rinny Berhitoe) ii Summary The present study intended to investigate differences in emotional writing on social media, with a specific focus on gender differences and the effect of politics on emotion shift. To achieve this, emotion detection was performed on Twitter’s textual content (i.e. tweets) during a year starting from March 2016, with an assumption that a tweet contained more than one emotion types. To measure emotion shift in political situation, the United States presidential election of 2016 were set as the context. Two multi-label classifiers were trained for emotion detection, and two bi- nary classifiers were trained for politics/non-politics categorization. The top emotions emerged from tweets were either positive-related emotions (e.g. joy, surprise, trust) or negative-related emotions (i.e. anger, disgust, fear). The combination of both positive and negative emotions were rarely the case. Findings from gender-emotions stereotype study by Plant et al.(2000) was tested on the Twitter data and resulted in statistically significant result with a minor difference between male and female. In the political context, significant result was found in the emotion shift (i.e. positive-related and negative-related emotions) of each gender group and politics- labeled tweets negative-related emotions, between the period of two months before and after the election date. However, the effect size between groups were small (r 0.30), demonstrating Ç a trivial change. Contents Preface................................................i Summary............................................... ii 1 Introduction 2 1.1 Research questions......................................4 2 Related work 6 2.1 Sentiment analysis......................................6 2.2 Emotion detection......................................7 2.2.1 Types of emotions..................................8 2.2.2 Emotion analysis................................... 10 2.3 Multi-label classification................................... 13 2.4 Description of present study................................ 14 3 Experimental setup 16 3.1 Project Instruments...................................... 16 3.2 Data Annotation....................................... 17 3.2.1 Annotation procedure................................ 17 3.2.2 Annotation analysis................................. 18 3.3 Data preprocessing...................................... 19 3.3.1 Data cleaning..................................... 20 3.3.2 Data reduction and filtering............................. 21 3.3.3 Data labeling..................................... 22 3.4 Task A: Classification of multi-label emotions and political tweets.......... 22 iii CONTENTS 1 3.4.1 Feature extraction.................................. 23 3.4.2 Implementation: Multi-label emotions...................... 24 3.4.3 Implementation: Politics Label........................... 26 3.5 Task B: Statistical Analysis of gender and emotion labels................ 27 4 Results 28 4.1 Classifier performance.................................... 28 4.2 Dominant emotion labels.................................. 29 4.2.1 Wilcoxon rank-sum test: gender and the positive-related emotions..... 31 4.3 Emotion theory........................................ 32 4.3.1 Female-emotion stereotypes............................ 34 4.3.2 Male-emotion stereotypes.............................. 34 4.4 Emotion shift in situational context............................ 37 4.4.1 Observation of the non-politics/politics tweets................. 38 4.4.2 Observation of gender-emotion shift....................... 40 5 Discussion 42 6 Conclusion 47 Bibliography 48 Chapter 1 Introduction A person may have written their thoughts because they wanted to record an idea in a thorough and concise manner. It could also be that they had psychological problem (i.e. mental illness) or exercise in writing as a healing process. Studies in psychology (Kennedy-Moore and Watson, 2001; Di Blasio et al., 2015) have found that writing helps the brain to regulate emotions and to cope with emotional distress. It may also assist to heal trauma and severe depression. The con- tent of these writings was shown to be related to the disclosure of the person’s deepest thoughts and feelings (Krpan et al., 2013), as participants were specifically asked to explore and reflect on major traumatic events (Ironson et al., 2013). The format of these conventional writings (e.g. a diary written with pen and paper) usually involved chronologically ordered, detailed de- scription of emotional moments, written in long, nicely-worded sentences. The advancement of technology is gradually replacing traditional tools, and tweets — the messages posted on the micro-blogging social media platform Twitter1 — are a good example of how digital forms are different from conventional types of written language. Twitter provides a smaller space (in the form of a text-box) to write, physically and literally, with a limit of 140 characters. This limit leads to the increased use of slang and abbreviation, and the frequent occurrence of grammatical er- rors. Users of social media overcome the space constraint by using ill-formed words to express their thoughts. This opens up wide opportunity for researchers to conduct related studies deal- ing with this noisy user-generated text, and trying to automatically interpret the thoughts users express on these media. One of such areas is the Natural Language Processing sub-field of emo- 1www.twitter.com 2 CHAPTER 1. INTRODUCTION 3 tion detection. Emotion detection deals with inferring certain classes of emotions from written text. From everyday experience, it seems that some emotions are distinct and occur independently. The inherently contradictory emotions, such as love and hate, might need a disjoint set of classes to accommodate the unique aspects of each class. On the other hand, identical emotions com- monly fall under the same emotional valence and they are frequently co-occur in a certain situational context. Therefore, these multiple emotions may be grouped together after being processed through some empirical justification. Emotion detection, cast as a multi-label classi- fication problem, could aid in illuminating the complex nature of emotions co-occurrence, thus providing an understanding of the characteristics of each emotion. Psychology studies have revealed that emotion co-occurrence might originate from bipolar emotional valence (i.e. positive and negative); this is known as emotional ambivalence (Berrios et al., 2015; Heavey et al., 2017). Emotional ambivalence, or mixed emotional states, often occur as often as homogeneous emotions. Plant et al.(2000) studied gender stereotypes in emotional co-occurrence, and found that females were often associated with happiness, fear, love, sadness, and sympathy; whereas males were associated with anger and pride. Later, Durik et al.(2006) extended Plant et al.(2000)’s study by including four ethnic groups as their subject (African Americans, Hispanic Americans, Asian Americans, and European Americans) and found sim- ilar results. The present study will try to verify Plant’s results by applying emotion detection on tweets from males and females, respectively. Gender roles in emotional states are put into context to learn social attributes in online en- vironment. Coates (1991) and Hall (1995) have implied that social attributes are performances, since they manifest as styles of writing that can be adapted and can vary amongst their users in different situations (as cited in Bamman et al.(2014)). This has been confirmed by a study by Bailey et al.(2013) which showed that girls who posted online tended to follow the main- stream stereotype of young women in online spaces: (girls) show themselves to be attractive, have a boyfriend, and a part of the party scene. Following this concept, this study also uses a political situation as a context for deeper investigation into the differences of emotive language between gender. There are several possible motivations for specifically choosing political sit- uations: many believe that recent historical moments in the world of politics have caused a CHAPTER 1. INTRODUCTION 4 paradigm shift in the Overton window (Lehman, 2014). For instance, President Donald Trump was once regarded as an ‘abnormal’ politician, meaning that he is not the traditional figure of a president candidate (EASTMAN and GIILDER, 2017). Moreover, the United Kingdom’s vote to leave the European Union, known as Brexit, also came as a surprise. These turbulent times could trigger people to disclose not only their hopes and anxieties, but also their thoughts and opinions in various formats (e.g. via demonstrations, video campaigns, blogs, articles, and so- cial networking sites). Twitter’