Emotion Classification Using Massive Examples Extracted from the Web Ryoko TOKUHISA†‡ Kentaro INUI‡ Yuji MATSUMOTO‡ †Toyota Central R&D Labs., INC. ‡Nara Institute of Science and Technology Nagakute Aichi JAPAN Ikoma Nara JAPAN [email protected] ryoko-t,inui,matsu @is.naist.jp { } Abstract given task as precisely and efficiently as possi- ble by exchanging information required for the In this paper, we propose a data-oriented task through dialog (Allen et al., 1994, etc.). method for inferring the emotion of a More recent research (Foster, 2007; Tokuhisa and speaker conversing with a dialog system Terashima, 2006, etc.), on the other hand, has been from the semantic content of an utterance. providing evidence for the importance of the af- We first fully automatically obtain a huge fective or emotional aspect in a wider range of di- collection of emotion-provoking event in- alogic contexts, which has been largely neglected stances from the Web. With Japanese cho- in the context of task-oriented dialogs. sen as a target language, about 1.3 million A dialog system may be expected to serve, for emotion provoking event instances are ex- example, as an active listening 1 partner of an el- tracted using an emotion lexicon and lexi- derly user living alone who sometimes wishes to cal patterns. We then decompose the emo- have a chat. In such a context, the dialog system tion classification task into two sub-steps: is expected to understand the user’s emotions and sentiment polarity classification (coarse- sympathize with the user. For example, given an grained emotion classification), and emo- utterence I traveled far to get to the shop, but it tion classification (fine-grained emotion was closed from the user, if the system could infer classification). For each subtask, the the user’s emotion behind it, it would know that collection of emotion-proviking event in- it would be appropriate to say That’s too bad or stances is used as labelled examples to That’s really disappointing. It can be easily imag- train a classifier. The results of our ex- ined that such affective behaviors of a dialog sys- periments indicate that our method signif- tem would be beneficial not only for active listen- icantly outperforms the baseline method. ing but also for a wide variety of dialog purposes We also find that compared with the single- including even task-oriented dialogs. step model, which applies the emotion To be capable of generating sympathetic re- classifier directly to inputs, our two-step sponses, a dialog system needs a computational model significantly reduces sentiment po- model that can infer the user’s emotion behind larity errors, which are considered fatal er- his/her utterence. There have been a range of stud- rors in real dialog applications. ies for building a model for classifying a user’s emotions based on acoustic-prosodic features and 1 Introduction facial expressions (Pantic and Rothkrantz, 2004, etc.). Such methods are, however, severely lim- Previous research into human-computer interac- ited in that they tend to work well only when the tion has mostly focused on task-oriented dialogs, user expresses his/her emotions by “exaggerated” where the goal is considered to be to achieve a c 2008. Licensed under the Creative Commons 1Active listening is a specific communication skill, based Attribution-Noncommercial-Share Alike 3.0 Unported li- on the work of psychologist Carl Rogers, which involves giv- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). ing free and undivided attention to the speaker (Robertson, Some rights reserved. 2005). 881 Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 881–888 Manchester, August 2008 Web Text Emotion-provoking event corpus (EP corpus) Emotion-provoking event Emotion(polarity) I was disappointed because the shop was the shop was closed and I d disappointment closed and I d traveled a long way to get there Building traveled a long way to get there (negative) I was disappointed that it started raining emotion-provoking it started raining disappointment I am happy since the book store was open (negative) when I got back home event corpus (section 3.2) I am alone on Christmas loneliness (negative) the book store was open when I happiness (positive) OFFLINE got back home ONLINE Input Sentiment Emotion Output The restaurant was polarity classification <disappointment> very far but it was classification using EP Corpus closed (section 3.3) (section 3.4) Emotion classification Figure 1: Overview of our approach to emotion classification prosodic or facial expressions. Furthermore, what commonly used in emotion classification: a rule- is required in generating sympathetic responses is based approach and a statistical approach. Ma- the identification of the user’s emotion in a finer sum et al. (2007) and Chaumartin (2007) pro- grain-size. For example, in contrast to the above pose a rule-based approach to emotion classifica- example of disappointing, one may expect the re- tion. Chaumartin has developed a linguistic rule- sponse to My pet parrot died yesterday should be based system, which classifies the emotions engen- That’s really sad, wheras the response to I may dered by news headlines using the WordNet, Sen- have forgotten to lock my house should be You’re tiWordNet, and WordNet-Affect lexical resources. worried about that. The system detects the sentiment polarity for each In this paper, we address the above issue of word in a news headline based on linguistic re- emotion classification in the context of human- sources, and then attempts emotion classification computer dialog, and demonstrate that massive ex- by using rules based on its knowledge of sen- amples of emotion-provoking events can be ex- tence structures. The recall of this system is low, tracted from the Web with a reasonable accuracy however, because of the limited coverage of the and those examples can be used to build a seman- lexical resources. Regarding the statistical ap- tic content-based model for fine-grained emotion proach, Kozareva et al. (2007) apply the theory of classification. (Hatzivassiloglou and McKeown, 1997) and (Tur- ney, 2002) to emotion classification and propose 2 Related Work a method based on the co-occurrence distribution Recently, several studies have reported about di- over content words and six emotion words (e.g. alog systems that are capable of classifying emo- joy, fear). For example, birthday appears more of- tions in a human-computer dialog (Batliner et al., ten with joy, while war appears more often with 2004; Ang et al., 2002; Litman and Forbes-Riley, fear. However, the accuracy achieved by their 2004; Rotaru et al., 2005). ITSPOKE is a tutoring method is not practical in applications assumed dialog system, that can recognize the user’s emo- in this paper. As we demonstrate in Section 4, tion using acoustic-prosodic features and lexical our method significantly outperforms Kozareva’s features. However, the emotion classes are limited method. to Uncertain and Non-Uncertain because the pur- 3 Emotion Classification pose of ITSPOKE is to recognize the user’s prob- lem or discomfort in a tutoring dialog. Our goal, 3.1 The basic idea on the other hand, is to classify the user’s emotions We consider the task of emotion classification as into more fine-grained emotion classes. a classification problem where a given input sen- In a more general research context, while quite tence (a user’s utterance) is to be classified either a few studies have been presented about opinion into such 10 emotion classes as given later in Ta- mining and sentiment analysis (Liu, 2006), re- ble1oras neutral if no emotion is involved in the search into fine-grained emotion classification has input. Since it is a classification problem, the task emerged only recently. There are two approaches should be approached straightforwardly in a vari- 882 Table 1: Distribution of the emotion expressions and examples Sentiment 10 Emotion Emotion lexicon (349 Japanese emotion words) Polarity Classes Total Examples happiness 90 嬉しい (happy)，狂喜 (joyful)，喜ぶ (glad)，歓ぶ (glad) Positive pleasantness 7 楽しい (pleasant)，楽しむ (enjoy)，楽しめる (can enjoy) relief 5 安心 (relief)，ほっと (relief) fear 22 恐い (fear)，怖い (fear)，恐ろしい (frightening) sadness 21 悲しい (sad)，哀しい (sad)，悲しむ (feel sad) disappointment 15 がっかり (lose heart)，がっくり (drop one’s head) Negative unpleasantness 109 嫌 (disgust)，嫌がる (dislike)，嫌い (dislike) loneliness 15 寂しい (lonely)，淋しい (lonely)，わびしい (lonely) anxiety 17 不安 (anxiety)，心配 (anxiety)，気がかり (worry) anger 48 腹立たしい (angry)，腹立つ (get angry)，立腹 (angry) ety of machine learning-based methods if a suffi- subordinate clause emotion provoking event connective emotion lexicon cient number of labelled examples were available. Our basic idea is to learn what emotion is typically provoked in what situation, from massive exam- Figure 2: An example of a lexico-syntactic pattern ples that can be collected from the Web. The devel- opment of this approach and its subsequent imple- Table 2: Number of emotion-provoking events mentation has forced us to consider the following 10 Emotions EP event 10 Emotions EP event happiness 387,275 disappoint- 106,284 two issues. ment First, we have to consider the quantity and ac- pleasantness 209,682 unpleasantness 396,002 curacy of emotion-provoking examples to be col- relief 46,228 loneliness 26,493 fear 49,516 anxiety 45,018 lected. The process we use to collect emotion- sadness 31,369 anger 8,478 provoking examples is illustrated in the upper half of Figure 1. For example, from the sentence I was on the definition of emotion words proposed by disappointed because the shop was closed and I’d Teramura (1982). The distribution is shown in Ta- I traveled a long way to get there, pulled from the ble 1 with major examples.