Sentence-level Emotion Classification with Label and Context Dependence
Shoushan Li†‡, Lei Huang†, Rong Wang†, Guodong Zhou†*
†Natural Language Processing Lab, Soochow University, China ‡ Collaborative Innovation Center of Novel Software Technology and Industrialization {shoushan.li, lei.huang2013, wangrong2022}@gmail.com, [email protected]
…… Abstract
1 * Corresponding author
1045 Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1045–1053, Beijing, China, July 26-31, 2015. c 2015 Association for Computational Linguistics sometimes much more difficult than classifying 2 Related Work instances with a single label. On the other hand, unlike document-level emo- Over the last decade, there has been an explosion tion classification, sentence-level emotion classi- of work exploring various aspects of emotion fication is prone to the data sparseness problem analysis, such as emotion resource creation because a sentence normally contains much less (Wiebe et al., 2005; Quan and Ren, 2009; Xu et content. Given the short text of a sentence, it is al., 2010), writer’s emotion vs. reader’s emotion often difficult to predict its emotion due to the analysis (Lin et al., 2008; Liu et al., 2013), emo- limited information therein. For example, in S2, tion cause event analysis (Chen et al., 2010), doc- only one phrase “如愿以偿(that is all I want)” ex- ument-level emotion classification (Alm et al., presses the joy emotion. Once this phrase fails to 2005; Li et al., 2014) and sentence-level or short appear in the training data, it will be hard for the text-level emotion classification (Tokushisa et al., classifier to give a correct prediction according to 2008; Bhowmick et al., 2009; Xu et al., 2012). the limited content in this sentence. This work focuses on sentence-level emotion clas- In this paper, we address above two challenges sification. in sentence-level emotion classification by mod- Among the studies on sentence-level emotion eling both the label and context dependence. Here, classification, Tokushisa et al. (2008) propose a the label dependence indicates that multiple emo- data-oriented method for inferring the emotion of tion labels of an instance are highly correlated to an utterance sentence in a dialog system. They each other. For instance, the two positive emo- leverage a huge collection of emotion-provoking tions, joy and love, are more likely to appear at the event instances from the Web to deal with the data same time than the two counterpart emotions, joy sparseness problem in sentence-level emotion and hate. The context dependence indicates that classification. Bhowmick et al. (2009) and two neighboring sentences or two sentences in the Bhowmick et al. (2010) apply KNN-based classi- same paragraph (or document) might share the fication algorithms to classify news sentences into same emotion categories. For instance, in Figure multiple reader emotion categories. Although the 1, S1, S2, and S3, from the same paragraph, all multi-label classification difficulty has been no- share the emotion category joy. ticed in their study, the label dependence is not Specifically, we propose a factor graph, namely exploited. More recently, Xu et al. (2012) pro- Dependence Factor Graph (DFG), to model the la- poses a coarse-to-fine strategy for sentence-level bel and context dependence in sentence-level emotion classification. They deal with the data emotion classification. In our DFG approach, both sparseness problem by incorporating the transfer the label and context dependence are modeled as probabilities from the neighboring sentences to various factor functions and the learning task aims refine the emotion categories. To some extent, this to maximize the joint probability of all these fac- can be seen a specific kind of context information. tor functions. Empirical evaluation demonstrates However, they ignore the label dependence by di- the effectiveness of our DFG approach to captur- rectly applying Binary Relevance to overcome the ing the inherent label and context dependence. To multi-label classification difficulty. the best of our knowledge, this work is the first Unlike all above studies, this paper emphasizes attempt to incorporate both the label and context the importance of the label dependence and ex- dependence of sentence-level emotion classifica- ploits it in sentence-level emotion classification tion into a unified framework. via a factor graph model. Moreover, besides the The remainder of this paper is organized as fol- label dependence, our factor graph-based ap- lows. Section 2 overviews related work on emo- proach incorporates the context dependence in a tion analysis. Section 3 presents our observations unified framework to further improve the perfor- on label and context dependence in the corpus. mance of sentence-level emotion classification. Section 4 proposes our DFG approach to sen- tence-level emotion classification. Section 5 eval- 3 Observations uates the proposed approach. Finally, Section 6 To better illustrate our motivation of modeling the gives the conclusion and future work. label and context dependence, we systematically investigate both dependence phenomena in our evaluation corpus.
1046 0.2 0.183 0.18 0.16 0.14 0.12 0.094 0.1 0.078 0.077 0.08 0.06 0.04 0.02 0.005 0.003 0.002 0.0003 0
Figure 2: Probability distribution of most and least frequently-occurred pairs of emotion categories, with left four most frequently-occurred and right four least frequently-occurred, among all 28 pairs
The corpus contains 100 documents, randomly ~25%. Table 2 shows the numbers of the sen- selected from Quan and Ren (2009). There are to- tences grouped by the emotion labels they contain. tally 2751 sentences and each of them is manually From this table, we can see that more than half annotated with one or more emotion labels. sentences have two or more emotion labels. This indicates the popularity of the multi-label issue in Table 1: The numbers of the sentences in each sentence-level emotion classification. emotion category To investigate the phenomenon of label de-
pendence, we first assume that XR d denotes Emotion #Sentence Emotion #Sentence an input domain of instances and Ylll{,,...,} joy 691 anxiety 567 12 m hate 532 surprise 180 be a finite domain of possible emotion labels. love 1025 anger 287 Each instance is associated with a subset of Y and sorrow 611 expect 603 this subset is described as an m-dimensional vec- 12 m i tor yyyy{,,...,} where y =1 only if in- i Table 2: The numbers of the sentences stance x has label li . and y =0 otherwise. Then, grouped by the emotion labels they contain we can calculate the probability that an instance
#Sentence takes both emotion labels li and l j , denoted as No Label 180 p l(l ,ij ) . Figure 2 shows the probability distribu- One Label 1096 tion of most and least frequently-occurred pairs of Two Labels 1081 emotion categories, with left four most fre- Three Labels 346 quently-occurred and right four least frequently- Four or more labels 48 occurred, among all 28 pairs. From this figure, we ALL 2751 can see that some pairs, e.g., joy and love, are much more likely to be taken by one sentence than Table 1 shows the sentence distribution of the some other pairs, e.g. joy and anger. eight emotion categories. Obviously, the distribu- Finally, we investigate the phenomenon of the tion is a bit imbalanced. While about to one quar- context dependence by calculating the probabili- ter of sentences express the emotion category love, ties that two instances x and x have at least one only ~6% and ~10% express surprise and anger k l （ respectively, with the remaining 5 emotion cate- identical emotion label, i.e., p ykl y ) in gories distributed rather evenly from ~20% to different settings.
1047 Sentence-1 DFG model 22 2 2 h( yy, ) y y 12 2 1 3 l 1 y2 l 2 3 y1 y 13 1 1 g( yy, ) 22 l 3 1 13 y2 g( yy, ) 11 11 h( yy, ) 12
11 f ( X11, y )
Sentence-2 11 f ( X , y ) 22 l l 2 1
2 Xl, 1 l 12 3 Xl 21, 1 Xl, 3 11 3 2 Xl, 23 Xl 13, Xl22,
Figure 4: An example of DFG when two instances are involved: sentence-1 with the label vector [1, 0, 1] and sentence-2 with the label vector [1, 1, 0] k Note: each multi-label instance is transformed into three pseudo samples, represented as Xki ( 1 ,2 ,3 ) . f () represents a factor function for modeling textual features. g() represents a factor function for mod- eling the label dependence between two pseudo samples. h() represents a factor function for modeling the context dependence between two instances in the same context.
0.8 0.68 0.69 hate and angry than some other pair of emo- tion labels, e.g., hate and happy. 0.6 0.5 2) Context dependency: Two instances from the 0.4 same context are more likely to share the same 0.22 emotion label than those from a random selec- 0.2 tion. 0 4 Dependence Factor Graph Model Neighbor Paragraph Document Random In this section, we propose a dependence factor Figure 3: Probabilities that two instances have graph (DFG) model for learning emotion labels of an identical emotion label in different settings sentences with both label and context dependence.
4.1 Preliminary Figure 3 shows the probabilities that two in- stances have at least one identical emotion label in Factor Graph different settings, where neighbor, paragraph, A factor graph consists of two layers of nodes, i.e., document and random mean two neighboring in- variable nodes and factor nodes, with links be- stances, two instances from the same paragraph, tween them. The joint distribution over the whole two instances from the same document, and two set of variables can be factorized as a product of instances from a random selection, respectively. all factors. Figure 4 gives an example of our de- From this figure, we can see that two instances pendence factor graph (DFG) when two instances, from the same context are much more likely to i.e., sentence-1 and sentence-2 are involved. take an identical emotion label than two random instances. Binary Relevance From above statistics, we come to two basic ob- A popular solution to multi-label classification is servations: called binary relevance which constructs a binary 1) Label dependency: One sentence is more classifier for each label, resulting a set of inde- likely to take some pair of emotion labels, e.g.,
1048 pendent binary classification problems (Tsouma- kkkl 1 2 kas and Katakis, 2007; Tsoumakas et al., 2009). gyGyyy iiiklii,()exp Z lk In our approach, binary relevance is utilized as a 2 yGii y() preliminary step so that each original instance is (3) transformed into K pseudo samples, where K is Where ik l is the weight of the function, rep- the number of categories. For example, in Figure resenting the influence degree of the two in- 4, X 1 , X 2 , and X 3 represent the three pseudo k l 1 1 1 stances yi and yi . samples, generated from the same original in- 3) Context dependence factor function: stance sentence-1. kk h yii, H y denotes the additional context 4.2 Model Definition dependence relationship among the instances, k Formally, let G V E X ,, represent an instance where Hy i is the set of the instances con- network, where V denotes a set of sentence in- k k nected to y . and y are the labels of stances. E V V is a set of relationships be- i i tween sentences. Two kinds of relationship exist the pseudo instances from the same context but in our instance network: One represents the label generated from different original instances. dependence between each two pseudo instances The context dependence factor function is in- generated from the same original instance, while stantiated as follows: the other represents the context dependence when 1 2 hyHyyykkkk,()exp the two instances are from the same context, e.g., iiijkij Z kk 3 yHyji () the same paragraph. X is the textual feature vec- tor associated with a sentence. (4) We model the above network with a factor Where ijk is the weight of the function, repre- graph and our objective is to infer the emotion cat- senting the influence degree of the two in- egories of instances by learning the following stances and yk . joint distribution: j 4.3 Model Learning PYG Learning the DFG model is to estimate the best (1) kkkkkk parameter configuration ({},{},{}) to fXyg iiiiii,,, yG yh yHy ki maximize the log-likelihood objective function where three kinds of factor functions are used. LPYG log , i.e., 1) Textual feature factor function: fXy kk, ii denotes the traditional textual feature factor *argmax L (5) functions associated with each text X k . The i In this study, we employ the gradient decent textual feature factor function is instantiated as method to optimize the objective function. For ex- follows: ample, we can write the gradient of each kj with 1 kkkk (2) regard to the objective function: fXyxy iikjiji,exp, Z1 j Where xykk, is a feature function and xk L ij i ij kk E xij,, y i EPYG| x ij y i (6) kj represents a textual feature, i.e., a word feature kj in this study. k Where E xij, y i is the expectation of feature 2) Label dependence factor function: k kk function xyiji, given the data distribution. gyGy ii, denotes the additional label de- k EPYG| xij, y i is the expectation of feature pendence relationship among the pseudo in- kj k stances, where Gy i is the label set of the function under the distribution instances connected to yk . and are PYG given by the estimated model. Figure 5 i kj labels of the pseudo instances generated from illustrates the detailed algorithm for learning the the same original instance. The label depend- parameter . Note that LBP denotes the Loopy ence factor function is instantiated as follows:
1049 Belief Propagation (LBP) algorithm which is ap- Ren, 2009). In our experiments, we use 80 docu- plied to approximately infer the marginal distribu- ments as the training data and the remaining 20 tion in a factor graph (Frey and MacKay, 1998). documents as the test data. A similar gradient can be derived for the other pa- Features rameters. Each instance is treated as a bag-of-words and Input: Learning rate transformed into a binary vector encoding the Output: Estimated parameters presence or absence of word unigrams. Initialize 0 Evaluation Metrics Repeat In our study, we employ three evaluation metrics 1) Calculate E x y , k using LBP iji to measure the performances of different ap- proaches to sentence-level emotion classification. k 2) Calculate ExyPYG | iji, using LBP kj These metrics have been popularly used in some multi-label classification problems (Godbole and 3) Calculate the gradient of according to Sarawagi, 2004; Schapire and Singer, 2000). Eq. (6) 1) Hamming loss: It evaluates how many times 4) Update parameter with the learning an instance-label pair is misclassified consid- rate ering the predicted set of labels and the L ground truth set of labels, i.e., newold 1 q m hloss 11jj' (8) Until Convergence yyii mq ij11 Figure 5: The learning algorithm for DGP model where q is the number of all test instances and j' m is the number of all emotion labels. yi is 4.4 Model Prediction j the estimated label while yi is the true label. With the learned parameter configuration , the 2) Accuracy: It gives an average degree of the prediction task is to find a Y U* which optimizes similarity between the predicted and the the objective function, i.e., ground truth label sets of all test examples, i.e., YPUUL* YYG argmax,, (7) 1 q yy ' Where Y U* are the labels of the instances in the Accuracy ii (9) ' testing data. qyyi1 ii Again, we utilize LBP to calculate the marginal kL 3) F1-measure: It is the harmonic mean between probability of each instance PyYG i ,, and precision and recall. It can be calculated from predict the label with the largest marginal proba- true positives, true negatives, false positive bility. As all instances in the test data are con- and false negatives based on the predictions cerned, above prediction is performed in an itera- and the corresponding actual values, i.e., tion process until the results converge. q ' 1 yyii F1 (10) 5 Experimentation ' q i1 yyii We have systematically evaluated our DFG ap- Note that smaller Hamming loss corresponds to proach to sentence-level emotion classification. better classification quality, while larger accuracy and F-measure corresponds to better classifica- 5.1 Experimental Setting tion quality. Corpus The corpus contains 100 documents (2751 sen- tences) from the Ren-CECps corpus (Quan and
1050 0.7 0.634 0.6 0.477 0.5 0.461 0.378 0.391 0.379 0.4 0.3 0.242 0.261 0.269 0.2 0.1 Hloss Accuracy F1
Baseline LebelD(Wang et al.,2014) DFG-label(Our approach)
Figure 6: Performance comparison of different approaches to sentence-level emotion classification with the label dependence only
0.6 0.569
0.4770.472 0.5 0.45 0.443 0.416 0.407 0.4 0.3780.382 0.295 0.292 0.3 0.2610.264 0.275 0.215 0.2 Hloss Accuracy F1 Baseline Tansfer(Xu et al.,2012) DFG-context(Neighbor) DFG-context(Paragraph) DFG-context(Document)
Figure 7: Performance comparison of different approaches to sentence-level emotion classification with the context dependence only
baseline approach with an impressive improve- 5.2 Experimental Results with Label De- ment in all three kinds of evaluation metrics, i.e., pendence 23.5% reduction in Hloss, 25.6% increase in Ac- In this section, we compare following approaches curacy, and 11.8% increase in F1. This result ver- which only consider the label dependence among ifies the effectiveness of incorporating the label pseudo instances: dependence in sentence-level emotion classifica- Baseline: As a baseline, this approach applies tion. Compared to the state-of-the-art LabelD ap- a maximum entropy (ME) classifier with only proach, our DFG approach is much superior. Sig- textual features, ignoring both the label and nificant test show that our DFG approach signifi- context dependence. cantly outperforms both the baseline approach and LabelD: As the state-of-the-art approach to LabelD (p-value<0.01). One reason that LabelD handling multi-label classification, this ap- performs worse than our approach is possibly due proach incorporates label dependence, as de- to their separating learning on textual features and scribed in (Wang et al., 2014). Specifically, label relationships. Also, different from ours, their this approach first utilizes a Bayesian network approach could not capture the information be- to infer the relationship among the labels and tween two conflict emotion labels, such as “happy” then employ them in the classifier. and “sad” (they are not possibly appearing to- DFG-label: Our DFG approach with the label gether). dependence. 5.3 Experimental Results with Context De- Figure 6 compares the performance of different pendence approaches to sentence-level emotion classifica- tion with the label dependence. From this figure, In this section, we compare following approaches we can see that our DFG approach improves the which only consider the context dependence among pseudo instances:
1051 Baseline: same as the one in Section 5.2, performance with a large margin, irrespective of which applies a maximum entropy (ME) clas- the amount of training data available. sifier with only textual features, ignoring both the label and context dependence. Table 3: Performance of our DFG approach Transfer: As the state-of-the-art approach to with both label and context dependence incorporating contextual information in sen- tence-level emotion classification (Xu et al., Hloss Accuracy F1 2012), this approach utilizes the label transfor- Baseline 0.447 0.378 0.261 mation probability to refine the classification DFG-label 0.254 0.621 0.372 results. DFG-context 0.416 0.443 0.292 DFG-label (Neighbor): Our DFG approach with the context dependence only. Specifically, DFG-both 0.242 0.634 0.379 the neighboring instances are considered as context. Hloss DFG-label (Paragraph): Our DFG approach 0.6 with the context dependence only. Specifically, 0.5 the instances in the same paragraph are consid- 0.4 ered as context. 0.3 DFG-label (Document): Our DFG approach with the context dependence only. Specifically, 0.2 the instances in the same document are consid- 20% 40% 60% 80%
ered as context. Accuracy Figure 7 compares the performance of different 0.7 approaches to sentence-level emotion classifica- 0.6 tion with the context dependence only. From this 0.5 figure, we can see that our DFG approach consist- 0.4 ently improves the state-of-the-art in all three 0.3 kinds of evaluation metrics, i.e., 6.1% reduction in 0.2 Hloss, 6.5% increase in Accuracy, and 3.1% in- crease in F1 when the neighboring instances are 20% 40% 60% 80% considered as context. Among the three kinds of F1 context, the neighboring setting performs best. 0.4 We also find that using the whole document as the 0.35 context is not helpful and it performs even worse than the baseline approach. Compared to the state- 0.3 of-the-art Transfer approach, our DFG approach 0.25 with the neighboring context dependence is much 0.2 superior. Significant test show that our DFG ap- 20% 40% 60% 80% proach with the neighboring context dependence significantly outperforms the baseline approach Baseline DFG-both and the state-of-the-art LabelD approach (p- value<0.01). Figure 8: Performance of our DGF-both ap- 5.4 Experimental Results with Both Label proach when different sizes of training data are and Context Dependence used Table 3 shows the performance of our DFG ap- 6 Conclusion proach with both label and context dependence, denoted as DGF-both. From this table, we can see In this paper, we propose a novel approach to sen- that using both label and context dependence fur- tence-level emotion classification by incorporat- ther improves the performance. ing both the label dependence among the emotion Figure 8 shows the performance of our DGF- labels and the context dependence among the con- both approach when different sizes of training textual instances into a factor graph, where the la- data are used to train the model. From this figure, bel and context dependence is modeled as various we can see that incorporating both the label and factor functions. Empirical evaluation shows that context dependence consistently improves the
1052 our DFG approach performs significantly better eVector Feature Representations. In Proceedings of than the state-of-the-art. NLP&CC-14, pp.217-228. In the future work, we would like to explore bet- Lin K., C. Yang, and H. Chen. 2008. Emotion Classi- ter ways of modeling the label and context de- fication of Online News Articles from the Reader’s pendence and apply our DFG approach in more Perspective. In Proceedings of the International applications, e.g. micro-blogging emotion classi- Conference on Web Intelligence and Intelligent fication. Agent Technology-08, pp.220-226. Liu H., S. Li, G. Zhou, C. Huang, and P. Li. 2013. Joint Acknowledgments Modeling of News Reader’s and Comment Writer’s Emotions. In Proceedings of ACL-13, short paper, This research work has been partially supported pp.511-515. by three NSFC grants, No.61273320, No.61375073, No.61331011, and Collaborative Quan C. and F. Ren. 2009. Construction of a Blog Innovation Center of Novel Software Technology Emotion Corpus for Chinese Emotional Expression and Industrialization. Analysis. In Proceedings of EMNLP-09, pp.1446- 1454. References Schapire R. E and Y. Singer. 2000. A Boosting-based System for Text Categorization. Machine learning, Alm C., D. Roth and R. Sproat. 2005. Emotions from pp. 135-168 Text: Machine Learning for Text-based Emotion Prediction. In Proceedings of EMNLP-05, pp.579- TOKUHISA R., K. Inui, and Y. Matsumoto. 2008. 586. Emotion Classification Using Massive Examples Extracted from the Web. In Proceedings of COL- Bhowmick P., A. Basu, P. Mitra, and A. Prasad. 2009. ING-2008, pp.881-888. Multi-label Text Classification Approach for Sen- tence Level News Emotion Analysis. Pattern Tsoumakas G. and I. Katakis. 2007. Multi-label Clas- Recognition and Machine Intelligence. Lecture sification: An Overview. In Proceedings of Interna- Notes in Computer Science, Volume 5909, pp 261- tional Journal of Data Warehousing and Mining, 266. 3(3), pp.1-13. Bhowmick P., A. Basu, P. Mitra, and A. Prasad. 2010. Tsoumakas G., I. Katakis, and I. Vlahavas. 2009. Min- Sentence Level News Emotion Analysis in Fuzzy ing Multi-label Data. Data Mining and Knowledge Multi-label Classification Framework. Research in Discovery Handbook, pages 1–19. Computing Science. Special Issue: Natural Lan- Wen S. and X. Wan. 2014. Emotion Classification in guage Processing and its Applications, pp.143-154. Microblog Texts Using Class Sequential Rules. In Bollen J., H. Mao, and X.-J. Zeng. 2011. Twitter Mood Proceedings of AAAI-14, 187-193. Predicts the Stock Market. Journal of Computa- Wang S., J. Wang, Z. Wang, and Q. Ji. 2014. Enhanc- tional Science, 2(1):1–8, 2011. ing Multi-label Classification by Modeling Depend- Chen Y., S. Lee, S. Li and C. Huang. 2010. Emotion encies among Labels. Pattern Recognition. Vol. 47. Cause Detection with Linguistic Constructions. In Issue 10: 3405-3413, 2014. Proceedings of COLING-10, pp.179-187. Wiebe J., T. Wilson, and C. Cardie. 2005. Annotating Frey B. and D. MacKay. 1998. A Revolution: Belief Expressions of Opinions and Emotions in Language. Propagation in Graphs with Cycles. In Proceedings Language Resources and Evaluation, 39, 65-210. of NIPS-98, pp.479–485. Xu G., X. Meng and H. Wang. 2010. Build Chinese Galik M. and S. Rank. 2012. Modelling Emotional Tra- Emotion Lexicons Using A Graph-based Algorithm jectories of Individuals in an Online Chat. In Pro- and Multiple Resources. In Proceedings of COL- ceedings of Springer-Verlag Berlin Heidelberg-12, ING-10, pp.1209-1217. pp.96-105. Xu J., R. Xu, Q. Lu, and X. Wang. 2012. Coarse-to- Godbole S. and S. Sarawagi. 2004. Discriminative Methods for Multi-labeled Classification. In Ad- fine Sentence-level Emotion Classification based on vances in knowledge discovery and data mining. pp. the Intra-sentence Features and Sentential Context. 22-30. In Proceedings of CIKM-12, poster, pp.2455-2458. Kiritchenko S., X. Zhu, and S. Mohammad. 2014. Sen- timent Analysis of Short Informal Texts. Journal of Artificial Intelligence Research, 50(2014), pp.723- 762. Li C., H. Wu, and Q. Jin. 2014. Emotion Classification of Chinese Miroblog Text via Fusion of BoW and
1053