Emotion Classification Using Massive Examples Extracted from the Web
Ryoko TOKUHISA†‡ Kentaro INUI‡ Yuji MATSUMOTO‡ †Toyota Central R&D Labs., INC. ‡Nara Institute of Science and Technology Nagakute Aichi JAPAN Ikoma Nara JAPAN [email protected] ryoko-t,inui,matsu @is.naist.jp { }
Abstract given task as precisely and efficiently as possi- ble by exchanging information required for the In this paper, we propose a data-oriented task through dialog (Allen et al., 1994, etc.). method for inferring the emotion of a More recent research (Foster, 2007; Tokuhisa and speaker conversing with a dialog system Terashima, 2006, etc.), on the other hand, has been from the semantic content of an utterance. providing evidence for the importance of the af- We first fully automatically obtain a huge fective or emotional aspect in a wider range of di- collection of emotion-provoking event in- alogic contexts, which has been largely neglected stances from the Web. With Japanese cho- in the context of task-oriented dialogs. sen as a target language, about 1.3 million A dialog system may be expected to serve, for emotion provoking event instances are ex- example, as an active listening 1 partner of an el- tracted using an emotion lexicon and lexi- derly user living alone who sometimes wishes to cal patterns. We then decompose the emo- have a chat. In such a context, the dialog system tion classification task into two sub-steps: is expected to understand the user’s emotions and sentiment polarity classification (coarse- sympathize with the user. For example, given an grained emotion classification), and emo- utterence I traveled far to get to the shop, but it tion classification (fine-grained emotion was closed from the user, if the system could infer classification). For each subtask, the the user’s emotion behind it, it would know that collection of emotion-proviking event in- it would be appropriate to say That’s too bad or stances is used as labelled examples to That’s really disappointing. It can be easily imag- train a classifier. The results of our ex- ined that such affective behaviors of a dialog sys- periments indicate that our method signif- tem would be beneficial not only for active listen- icantly outperforms the baseline method. ing but also for a wide variety of dialog purposes We also find that compared with the single- including even task-oriented dialogs. step model, which applies the emotion To be capable of generating sympathetic re- classifier directly to inputs, our two-step sponses, a dialog system needs a computational model significantly reduces sentiment po- model that can infer the user’s emotion behind larity errors, which are considered fatal er- his/her utterence. There have been a range of stud- rors in real dialog applications. ies for building a model for classifying a user’s emotions based on acoustic-prosodic features and 1 Introduction facial expressions (Pantic and Rothkrantz, 2004, etc.). Such methods are, however, severely lim- Previous research into human-computer interac- ited in that they tend to work well only when the tion has mostly focused on task-oriented dialogs, user expresses his/her emotions by “exaggerated” where the goal is considered to be to achieve a c 2008. Licensed under the Creative Commons 1Active listening is a specific communication skill, based Attribution-Noncommercial-Share Alike 3.0 Unported li- on the work of psychologist Carl Rogers, which involves giv- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). ing free and undivided attention to the speaker (Robertson, Some rights reserved. 2005).
881 Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 881–888 Manchester, August 2008 Web Text Emotion-provoking event corpus (EP corpus) Emotion-provoking event Emotion(polarity) I was disappointed because the shop was the shop was closed and I d disappointment closed and I d traveled a long way to get there Building traveled a long way to get there (negative) I was disappointed that it started raining emotion-provoking it started raining disappointment I am happy since the book store was open (negative) when I got back home event corpus (section 3.2) I am alone on Christmas loneliness (negative) the book store was open when I happiness (positive) OFFLINE got back home ONLINE Input Sentiment Emotion Output The restaurant was polarity classification
882 Table 1: Distribution of the emotion expressions and examples Sentiment 10 Emotion Emotion lexicon (349 Japanese emotion words) Polarity Classes Total Examples happiness 90 嬉しい (happy)，狂喜 (joyful)，喜ぶ (glad)，歓ぶ (glad) Positive pleasantness 7 楽しい (pleasant)，楽しむ (enjoy)，楽しめる (can enjoy) relief 5 安心 (relief)，ほっと (relief) fear 22 恐い (fear)，怖い (fear)，恐ろしい (frightening) sadness 21 悲しい (sad)，哀しい (sad)，悲しむ (feel sad) disappointment 15 がっかり (lose heart)，がっくり (drop one’s head) Negative unpleasantness 109 嫌 (disgust)，嫌がる (dislike)，嫌い (dislike) loneliness 15 寂しい (lonely)，淋しい (lonely)，わびしい (lonely) anxiety 17 不安 (anxiety)，心配 (anxiety)，気がかり (worry) anger 48 腹立たしい (angry)，腹立つ (get angry)，立腹 (angry) ety of machine learning-based methods if a suffi- subordinate clause emotion provoking event connective emotion lexicon cient number of labelled examples were available. Our basic idea is to learn what emotion is typically provoked in what situation, from massive exam- Figure 2: An example of a lexico-syntactic pattern ples that can be collected from the Web. The devel- opment of this approach and its subsequent imple- Table 2: Number of emotion-provoking events mentation has forced us to consider the following 10 Emotions EP event 10 Emotions EP event happiness 387,275 disappoint- 106,284 two issues. ment First, we have to consider the quantity and ac- pleasantness 209,682 unpleasantness 396,002 curacy of emotion-provoking examples to be col- relief 46,228 loneliness 26,493 fear 49,516 anxiety 45,018 lected. The process we use to collect emotion- sadness 31,369 anger 8,478 provoking examples is illustrated in the upper half of Figure 1. For example, from the sentence I was on the definition of emotion words proposed by disappointed because the shop was closed and I’d Teramura (1982). The distribution is shown in Ta- I traveled a long way to get there, pulled from the ble 1 with major examples. Web, we learn that the clause the shop was closed We then went on to find sentences in the Web and I’d traveled a long way to get there is an ex- corpus that possibly contain emotion-provoking ample of an event that provokes disappointment . events. A subordinate clause was extracted as an In this paper, we refer to such an example as an emotion-provoking event instance if (a) it was sub- emotion-provoking event and a collection of event- ordinated to a matrix clause headed by an emo- provoking events as an emotion-provoking event tion word and (b) the relation between the sub- corpus (an EP corpus). Details are described in ordinate and matrix clauses is marked by one of Section 3.2. the following eight connectives: ので, から, ため, Second, assuming that an EP corpus can be ob- て, のは, のが, ことは , ことが. An example is given tained, the next issue is how to use it for our in Figure 2. In the sentence “突然雨が降り出した emotion classification task. We propose a method のは がっかりだ (I was disappointed that it suddenly whereby an input utterance (sentence) is classi- started raining)”, the subordinate clause “突然雨 fied in two steps, sentiment polarity classification が降り出した (it suddenly started raining)” mod- followed by fine-grained emotion classification as ifies “がっかりだ (I was disappointed)” with the shown in the lower half of Figure 1. Details are connective “のは (that)”. In this case, therefore, given in Sections 3.3 and 3.4. the event mention “突然雨が降り出した (it suddenly 3.2 Building an EP corpus started raining)” is learned as an event instance We used ten emotions happiness, pleasantness, that provokes disappointment . relief, fear, sadness, disappointment, unpleasant- Applying the emotion lexicon and the lexical ness, loneliness, anxiety, anger in our emotion patterns to the Japanese Web corpus (Kawahara classification experiment. First, we built a hand- and Kurohashi, 2006), which contains 500 million crafted lexicon of emotion words classified into sentences, we were able to collect about 1.3 mil- the ten emotions. From the Japanese Evaluation lion events as causes of emotion. The distribution Expression Dictionary created by Kobayashi et is shown in Table 2. al. (2005), we identified 349 emotion words based Tables 3 and 4 show the results of our evalua-
883 Table 4: Examples from in the EP corpus EP-Corpus Result of evaluation Emotion-provoking Event Emotion word 10 Emotions (P/N) Polarity Emotion 花持ちが悪い (A flower died quickly) 残念だ (diappointed) disappointment(N) Correct Correct 敵が多い (There are a lot of enemies) 飽きる (lose interest) unpleasantness(N) Correct Context-dep. ちんげん菜が多い (There is a lot of 嬉しい (happy) happiness(P) Context-dep. Context-dep. Chinese cabbage) ジュースが飲みたい (I would like to 大変だ (terrible) unpleasantness(N) Error Error drink orange juice)
to make any sympathetic response. In such a case, Table 3: Correctness of samples from the EP cor- the user’s input should be classified as neutral . pus Polarity Emotion The classification between emotion-involved Correct 1140 (57.0%) 988 (49.4%) and neutral is not necessarily a simple problem, Context-dep. 678 (33.9%) 489 (24.5%) Error 182 (9.1%) 523 (26.2%) however, because we have not found yet any prac- tical method for collecting training examples of the tion for the resultant EP corpus. One annotator, class neutral . We cannot rely on the analogy to who was not the developer of the EP corpus, eval- the pattern-based method we have adopted to col- uated 2000 randomly chosen events. The “Polar- lect emotion-provoking events — there seems no ity” column in Table 3 shows the results of evaluat- reliable lexico-syntactic pattern for extracting neu- ing whether the sentiment polarity of each event is tral examples. Alternatively, if the majority of the correctly labelled, whereas the“Emotion” column sentences on the Web were neutral, one would sim- shows the correctness at the level of the 10 emo- ply use a set of randomly sampled sentences as la- tion classes. The correctness of each example was belled data for neutral . This strategy, however, evaluated as exemplified in Table 4. Correct indi- does not work because neutral sentences are not cates a correct example, Contex-dep. indicates a the majority in real Web texts. As an attempt, we context-dependent example, and Error is an error collected 1000 sentences randomly from the Web example. For example, in the case of There are a and investigated their distribution of sentiment po- lot of enemies in Table 4, the “Polarity” is Correct larity. The results, shown in Table 5, revealed that because it represents a negative emotion. How- the ratio of neutral events was unexpectedly low. ever, its emotion class unpleasantness is judged These results indicate the difficulty of collecting Context-dep. neutral events from Web documents. As Table 3 shows, the Sentiment Polarity is cor- Taking this problem into account, we adopt a rect in 57.0% of cases and partially correct (Cor- two-step approach, where we first classify a given rect + Context-dep.) in 90.9% of cases. On the input into three sentiment polarity classes, either other hand, the Emotion is correct in only 49.4% positive, negative or neutral, and then classify only of cases and partially correct in 73.9% of cases. those judged positive or negative into our 10 fine- These figures may not seem very impressive. As grained emotion classes. In the first step, i.e. sen- far as its impact on the emotion classification accu- timent polarity classification, we use only the pos- racy is concerned, however, the use of our EP cor- itive and negative examples stored in the EP cor- pus, which requires no supervision, makes remark- pus and assume sentence to be neutral if the out- able improvements upon Kozareva et al. (2007)’s put of the classification model is near the deci- unsupervised method as we show later. sion boundary. There are additional advantages in this approach. First, it is generally known that 3.3 Sentiment Polarity Classification performing fine-grained classification after coarse Given the large collection of emotion-labelled ex- classification often provides good results particu- amples, it may seem straightforward to develop a larly when the number of the classes is large. Sec- trainable model for emotion classification. Before ond, in the context of dialog, a misunderstanding moving on to emotion classification, however, it the user’s emotion at the sentiment polarity level should be noted that a user’s input utterance may would be a disaster. Imagine that the system says not involve any emotion. For example, if the user You must be happy when the user in fact feels sad. gives an utterance I have a lunch at the school cafe- As we show in Section 4.2, such fatal errors can be teria every day, it is not appropriate for the system reduced by taking the two-step approach.
884 Table 5: Distribution of the Sentiment polarity of Table 6: Examples of positive and negative words sentences randomly sampled from the Web P 子供 (child)，夏休み (summer vacation)， Sentiment Polarity Number Ratio 役立つ (useful)，成功する (succeed) positive 650 65.0% N 負担 (cost)，難しい (difficult)， 難しい ，失敗する negative 153 15.3% (difficult) (failure) neutral 117 11.7% Context-dep. 80 8.0% Ranking of similar events Ranking of emotion rank event emotion similarity rank emotion score 1. {event1}
Various methods have already been proposed for For the similarity measure, we use cosine similar- sentiment polarity classification, ranging from the ity between bag-of-words vectors; sim(I,EP)= use of co-occurrence with typical positive and neg- I EP I · EP for input sentence I and an emotion- ative words (Turney, 2002) to bag of words (Pang provoking| || | event EPin the EP corpus. The score et al., 2002) and dependency structure (Kudo and of each class is given by the sum of its similarity Matsumoto, 2004). Our sentiment polarity clas- scores. sification model is trained with SVMs (Vapnik, An example is presented in Figure 4. The emo- 1995), and the features are 1-gram, 2-gram, 3- { tion of the most similar event is disappointment , gram of words and the sentiment polarity of the } that of the second-most similar event is words themselves. Figure 3 illustrates how the sen- unpleasantness tied with loneliness . Af- tence “子供の教育の負担が増える (The cost of educat- ter calculating the sum for each emotion, the ing my child increases)” is encoded to a feature system outputs loneliness as the emotion for the vector. Here we assume the sentiment polarity of input I because the score for loneliness is the the “子供 (child)” and “教育 (education)” are pos- highest. itive, while the “負担 (cost)” is negative. These polarity values are represented in parallel with the 4 Experiments corresponding words, as shown in Figure 3. By 4.1 Sentiment polarity classification expanding 1-gram, 2-gram, 3-gram in this lat- { } We conducted experiments on sentiment polarity tice representation, the following list of features classification using the following two test sets: are extracted: 子供 (child), Positive, 子供 (child)- の (of), Positive-の (of), 子供 (child)-の (of)-教育 (ed- TestSet1: The first test set was a set of utterances ucation), etc.. The polarity value of each word is which 6 subject speakers produced interact- defined in our sentiment polarity dictionary, which ing with our prototype dialog system. This includes 1880 positive words and 2490 negative data include 31 positive utterances, 34 nega- words. To create this dictionary, one annotator tive utterances, and 25 neutral utterances. identified positive and negative words from the 50 TestSet2: For the second test set, we used the thousand most frequent words sampled from the 1140 samples that were judged Correct with Web. Table 6 shows some examples. respect to sentiment polarity in Table 3. 3.4 Emotion Classification 491 samples (43.1%) were positive and 649 (56.9%) were negative. We then added 501 For fine-grained emotion classification, we pro- neutral sentences newly sampled from the pose a k-nearest-neighbor approach (kNN) using Web. These samples are disjoint from the EP the EP corpus. corpus used for training classifiers. Given an input utterance, the kNN model re- trieves k-most similar labelled examples from the For each test set, we tested our sentiment EP corpus. Given the input The restaurant was polarity classifier in both the two-class (posi- very far but it was closed as Figure 1, for exam- tive/negative) setting, where only positive or neg- ple, the kNN model finds similar labelled exam- ative test samples were used, and the three-class ples, say, labelled example the shop was closed (positive/negative/neutral) setting. The perfor- { and I’d traveled far to get there in the EP corpus. mance was evaluated in F-measure. } 885 tions would fit a sentence. Some examples are Table 7: F-values of sentiment polarity classifica- shown in Table 9. The inter-annotator agree- tion (positive/negative) TestSet1 TestSet2 ment is κ=0.76 in the kappa statistic (Cohen, Pos Neg Pos Neg 1960). For sentences annotated with two dif- Word 0.839 0.853 0.794 0.842 ferent labels (i.e. in the cases where the two Word + Polarity 0.833 0.857 0.793 0.841 annotators disagreed with), both labels were Table 8: F-values of sentiment polarity classifica- considered correct in the experiments — a tion (positive/negative/neutral) model’s answer was considered correct if it TestSet1 TestSet2 was identical with either of the two labels. Pos Neg Pos Neg TestSet1 (1p, acceptable): One of the above two Word 0.743 0.758 0.610 0.742 Word + Polarity 0.758 0.769 0.610 0.742 annotators was asked to annotate each posi- tive or negative sentence in TestSet1 with all Table 7 shows the results for the two-class the emotions involved in it. The number of setting, whereas Table 8 shows the results for emotions for a positive sentence was 1.48 on the three-class. “Word” denotes the model average, and 2.47 for negative sentences. Ta- trained with only word n-gram features, whereas ble 10 lists some examples. In the experi- “Word+Polarity” denotes the model trained with ments, a model’s answer was considered cor- n-gram features extracted from a word-polarity lat- rect if it was identical with one of the labelled tice (see Figure 3). classes. The results shown in Table 7 indicate that both TestSet2: For TestSet2, we used the results of our the “Word” and “Word+Polarity” models are ca- judgments on the correctness for estimating pable of separating positive samples from negative the quality of the EP corpus described in Sec- ones at a high level of accuracy. This is an im- tion 3.2. portant finding, given the degree of the correctness In the experiments, the following two models of our EP corpus. As we have shown in Table 3, were compared: only 57% of samples in our EP corpus are “ex- actly” correct in terms of sentiment polarity. The Baseline: The baseline model simulates the figures in Table 7 indicate that context-dependent method proposed by (Kozareva et al., samples are also useful for training a classifier. Ta- 2007). Given an input sentence, their ble 7 also indicates that no significant difference is model first estimates the pointwise mu- found between the “Word” and “Word+Polarity” tual information (PMI) between each models. In fact, we also examined another model content word cwj included in the sen- tence and emotion expression e which used dependency-structure information as ∈ well; however, no significant gain was achieved. anger, disgust, fear, joy, sudness, surprise { hits(e,cw) } From these results, we speculate that, as far as the by PMI(e, cw)=loghits(e)hits(cw) , where two-class sentiment polarity problem is concerned, hits(x) is a hit count of word(s) x on a word n-gram features might be sufficient if a very Web search engine. The model then cal- large set of labelled data are available. culates the score of each emotion class Ei On the other hand, Table 8 indicates that the by summing the PMI scores between each three-class problem is much harder than the two- content word cwj in the input and emotion class problem. Specifically, positive sentences expression ei corresponding to that emotion tend to be classified as neutral. This method has class: score(Ei)= j PMI(ei,cwj). to be improved in future models. Finally, the model chooses the best scored 4.2 Emotion classification emotion class as an output. For our experi- ments, we selected the following 10 emotion For fine-grained emotion classification, we used expressions: the following three test sets: 嬉しい (happy), 楽しい (pleased), 安心 (re- TestSet1 (2p, best): Two annotators were asked lieved), 恐い (affraid), 悲しい (sad), 残念 to annotate each positive or negative sen- (disappointed), 嫌 (hate), 寂しい (lonely), tence in TestSet1 with one of the 10 emotion 不安 (anxious), 腹立たしい (angry) classess. The annotators chose only one emo- For hit counts, we used the Google search en- tion class even if they thought several emo- gine. 886 Table 9: Examples of TestSet1 (2p, best) Annotator A Annotator B クリスマスにプレゼントをもらった (I got a Christmas present) happiness happiness 友達の家に遊びに行く (I’m going to go to my friend’s house ) pleasantness pleasantness 花見に行ったら突然雨が降り出した (It rained suddenly when I went to sadness sadness see the cherry blossoms) 渋滞でほとんど動かない (My car can’t move because of the traffic jam) unpleasantness anger Table 10: Examples of TestSet1 (1p, acceptable) Annotator A クリスマスにプレゼントをもらった (I got a Christmas present) happiness 友達の家に遊びに行く (I’m going to go to my friend’s house ) pleasantness , happiness 花見に行ったら突然雨が降り出した (It rained suddenly when I went to see the anger , sad , unpleasantness , cherry blossoms) disappointment 渋滞でほとんど動かない (My car can’t move because of the traffic jam) unpleasantness, anger
㪫㪼㫊㫋㪪㪼㫋㪈㩿㪉㫇㪃㪹㪼㫊㫋㪀 㪫㪼㫊㫋㪪㪼㫋㪈㩿㪈㫇㪃㪸㪺㪺㪼㫇㫋㪸㪹㫃㪼㪀 㪫㪼㫊㫋㪪㪼㫋㪉 accuracy can be considered as a crucial advantage. 㪈㪇㪇㩼 㪐㪇㩼 Regarding the issue of single-step vs. two-step, 㪏㪇㪅㪇㩼 㪏㪇㩼 㪎㪍㪅㪐㩼 Figure 5 indicates that the two-step models tended 㪎㪇㪅㪏㩼 㪎㪉㪅㪊㩼 㪎㪇㩼 㪍㪍㪅㪉㩼 㪍㪇㪅㪇㩼 㪍㪇㪅㪇㩼 to outperform the single-step models for all the test 㫐 㪍㪇㩼 㪌㪍㪅㪐㩼 㪺
㪸 㪌㪇㪅㪏㩼 㪌㪇㪅㪏㩼 㪌㪇㪅㪊㩼 㪌㪇㪅㪈㩼 㫉 㪋㪎㪅㪈㩼 㫌 㪌㪇㩼 㪋㪍㪅㪉㩼 㪋㪌㪅㪉㩼 set. A paired t-test for TestSet2, however, did not 㪺 㪋㪈㪅㪌㩼 㪋㪇㪅㪎㩼 㪺 㪊㪍㪅㪌㩼 㪘 㪋㪇㩼 reach significance 2. So we next examined this is- 㪊㪇㩼 㪈㪍㪅㪐㩼 㪉㪇㩼 㪈㪊㪅㪏㩼 sue in further detail. 㪈㪇㩼 㪌㪅㪎㩼 As argued in Section 3.3, in the context of 㪇㩼 㪙㪸㫊㪼㫃㫀㫅㪼 㪈㪄㪥㪥 㪊㪄㪥㪥 㪈㪇㪄㪥㪥 㪈㪄㪥㪥 㪊㪄㪥㪥 㪈㪇㪄㪥㪥 human-computer dialog, a misunderstanding of 㪜㫄㫆㫋㫀㫆㫅㩷㪚㫃㪸㫊㫊㫀㪽㫀㪺㪸㫋㫀㫆㫅 㪪㪼㫅㫋㫀㫄㪼㫅㫋㩷㪧㫆㫃㪸㫉㫀㫋㫐㩷㪂㩷㪜㫄㫆㫋㫀㫆㫅 㪚㫃㪸㫊㫊㫀㪽㫀㪺㪸㫋㫀㫆㫅 the user’s emotion at the level of sentiment polar- Figure 5: Results of emotion classification ity would lead to a serious problem, which we call a fatal error. On the other hand, misclassifying a k-NN: We tested the 1-NN, 3-NN and 10-NN case of happiness as, for example, pleasantness models. In each model, we examined a may well be tolerable. Table 11 shows the ratio single-step emotion classification and two- of fatal errors for each model. For TestSet2, the step emotion classification. In the former single-step 10-NN model made fatal errors in 30% method, the kNN model retrieves k-most sim- of cases, while the two-step 10-NN model in only ilar examples from the all of the EP corpus. In 17%. This improvement is statistically significant the latter method, when the sentiment polar- (p<0.01). ity of the input utterance has obtained by the 5 Conclusion sentiment polarity classifier, the kNN model In this paper, we have addressed the issue of emo- retrieves similar examples from only the ex- tion classification assuming its potential applica- amples whose sentiment polarity are the same tions to be human-computer dialog system includ- as the input utterance in the EP corpus. ing active-listening dialog. We first automatically The results are shown in Figure 5. “Emo- collected a huge collection, as many as 1.3M, of tion Classification” denotes the single-step mod- emotion-provoking event instances from the Web. els, whereas “Sentiment Polarity + Emotion Clas- We then decomposed the emotion classification sification” denotes the two-step models. task into two sub-steps: sentiment polarity clas- An important observation from Figure 5 is that sification and emotion classification. In sentiment our models remarkably outperformed the base- polarity classification, we used the EP-corpus as line. Apparently, an important motivation behind training data. The results of the polarity classifi- Kozareva et al. (2007)’s method is that it does cation experiment showed that word n-gram fea- not require any manual supervion. However, our tures alone are more or less sufficient to classify models, which rely on emotion-provoking event positive and negative sentences when a very large instances, are also totally unsupervised — no su- amount of training data is available. In the emo- pervision is required to collect emotion-provoking tion classification experiments, on the other hand, event instances. Given this commonality between 2The data size of TestSet1 was not sufficient for statistical the two methods, the superiority of our method in significance test
887 Table 11: Fatal error rate in emotion classification experiments Baseline Emotion Classification Sentiment Polarity 1-NN 3-NN 10-NN + Emotion Classification TestSet1 49.2% 29.2% 26.2% 24.6% 15.4% TestSet2 41.5% 37.6% 32.8% 30.0% 17.0% we adopted the k-nearest-neighbor (kNN) method. International Conference on Language Resources The results of the experiments showed that our and Evaluation. method significantly outperformed the baseline Kobayashi, Nozomi, Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2005. Collecting Evaluative Expres- method. The results also showed that our two- sions for Opinion Extraction. Lecture Notes in Arti- step emotion classification was effective for fine- ficial Intelligence, 3248. grained emotion classification. Specifically, fatal Kozareva, Zornitsa, Borja Navarro, Sonia Vazquez, and errors were significantly reduced with sentiment Andres Nibtoyo. 2007. UA-ZBSA: A Headline Emotion Classification through Web Information. In polarity classification before fine-grained emotion Proceedings of the 4th International Workshop on classification. Semantic Evaluations. For future work, we first need to examine other Kudo, Taku and Yuji Matsumoto. 2004. A Boost- machine learning methods to see their advantages ing Algorithm for Classification of Semi-Structured Text. In Proceedings of the EMNLP. and disadvantages in our task. We also need an Litman, Diane J. and Kate Forbes-Riley. 2004. Predict- extensive improvement in identifying neutral sen- ing Student Emotions in Computer-Human Tutoring tences. Finally, we are planning to apply our model Dialogues. Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. to the active-listening dialog system that our group Liu, Bing. 2006. Web Data Mining. Springer, pages has been developing and investigate its effects on 411Ð440. the user’s behavior. Masum, Shaikh Mostafa AI, Helmut Prendinger, and Mitsuru Ishizuka. 2007. Emotion Sensitive News References Agent: An Approach Towards User Centric Emo- tion Sensing from the News. In Proceedings of the Allen, James F., Lenhart K. Schubert, George Fergu- IEEE/WIC/ACM International Conference on Web son, Peter Heeman, Chung Hee Hwang, Tsuneaki Intelligence. Kato, Marc Light, Nathaniel G. Martin, Bradford W. Pang, Bo, Lillian Lee, and Shivakumar Vaithyanathan. Miller, Massimo Poesio, and David R. Traum. 1994. 2002. Thumbs up? sentiment classification using The TRAINS Project: A case study in building a machine learning techniques. Proceedings of the conversational planning agent. Journal of Experi- Conference on Empirical Methods in Natural Lan- mental and Theoretical AI (JETAI). guage Processing, pages 76Ð86. Ang, Jeremy, Rajdip Dhillon, Ashley Krupski, Eliza- Pantic, Maja and Leon J. M. Rothkrantz. 2004. Fa- beth Shriberg, and Andreas Stolcke. 2002. Prosody- cial Action Recognition for Facial Expression Anal- Based Automatic Detection Of Annoyance And ysis From Static Face Images. IEEE Transactions on Frustration In Human-Computer Dialog. Spoken SMC-B, 34(3):1449Ð1461. Language Processing, pages 2037Ð2040. Robertson, Kathryn. 2005. Active listening: more Batliner, A., K. Fischer, R. Huber, J. Spilker, and than just paying attention. Aust Fam Physician, E. Noth. 2004. How to find trouble in communi- 34(12):1053Ð1055. cation. Speech Communication, 40(1-2):117Ð143. Rotaru, Mihai, Diane J. Litman, and Katherine Forbes- Chaumartin, Francois-Regis. 2007. A knowledge- Riley. 2005. Interactions between Speech Recog- based system for headline sentiment tagging. In Pro- nition Problems and User Emotions. Proceedings ceedings of the 4th International Workshop on Se- 9th European Conference on Speech Communication mantic Evaluations. and Technology. Cohen, Jacob. 1960. A coefficient of agreement Teramura, Hideo. 1982. Japanese Syntax and Mean- for nominal scales. Educational and Psychological ing. Kurosio Publishers (in Japanese). Measurement 20, pages 37Ð46. Tokuhisa, Ryoko and Ryuta Terashima. 2006. Re- Foster, Mary Ellen. 2007. Enhancing Human- lationship between Utterance and ”Enthusiasm” in Computer Interaction with Embodied Conversa- Non-Task-Orieinted Conversational Dialogue. In tional Agents. Lecture Notes in Computer Science, Proceedings of the 7th SIGdial Workshop on Dis- 4555:828Ð837. course and Dialogue. Turney, P.D. 2002. Thumbs up? thumbs down? seman- Hatzivassiloglou, Vasileios and Kathleen R. McKeown. tic orientation applied to unsupervised classification 1997. Predicting the semantic orientation of adjec- of reviews. Proceedings of the 40th Annual Meet- Proceedings of the 35th Annual Meeting of the tives. ing of the Association for Computational Linguistics, Association for Computational Linguistics. pages 417Ð424. Kawahara, Daisuke and Sadao Kurohashi. 2006. Vapnik, Vladimir N. 1995. The Nature of Statistical Case Frame Compilation from the Web using High- Learning Theory. Springer. Performance Computing. In Proceedings of the 5th
888