Leveraging Cognitive Features for Sentiment Analysis Abhijit Mishray, Diptesh Kanojiay,|, Seema Nagar ?, Kuntal Dey?, Pushpak Bhattacharyyay yIndian Institute of Technology Bombay, India |IITB-Monash Research Academy, India ?IBM Research, India yfabhijitmishra, diptesh, [email protected] ?fsenagar3, [email protected] Abstract e.g., finding appropriate senses of a word given the context (e.g., His face fell when he was dropped Sentiments expressed in user-generated from the team vs The boy fell from the bicycle, short text and sentences are nuanced by where the verb “fell” has to be disambiguated) (3) subtleties at lexical, syntactic, semantic Domain Dependency, tackling words that change and pragmatic levels. To address this, polarity across domains. (e.g., the word unpre- we propose to augment traditional features dictable being positive in case of unpredictable used for sentiment analysis and sarcasm movie in movie domain and negative in case of un- detection, with cognitive features derived predictable steering in car domain). Several meth- from the eye-movement patterns of read- ods have been proposed to address the different ers. Statistical classification using our en- lexical level difficulties by - (a) using WordNet hanced feature set improves the perfor- synsets and word cluster information to tackle lex- mance (F-score) of polarity detection by ical ambiguity and data sparsity (Akkaya et al., a maximum of 3:7% and 9:3% on two 2009; Balamurali et al., 2011; Go et al., 2009; datasets, over the systems that use only Maas et al., 2011; Popat et al., 2013; Saif et al., traditional features. We perform feature 2012) and (b) mining domain dependent words significance analysis, and experiment on (Sharma and Bhattacharyya, 2013; Wiebe and Mi- a held-out dataset, showing that cognitive halcea, 2006). features indeed empower sentiment ana- lyzers to handle complex constructs. 1.2 Syntactic Challenges Difficulty at the syntax level arises when the 1 Introduction given text follows a complex phrasal structure and, This paper addresses the task of Sentiment Anal- phrase attachments are expected to be resolved be- ysis (SA) - automatic detection of the sentiment fore performing SA. For instance, the sentence A polarity as positive versus negative - of user- somewhat crudely constructed but gripping, quest- generated short texts and sentences. Several sen- ing look at a person so racked with self-loathing, timent analyzers exist in literature today (Liu and he becomes an enemy to his own race. requires Zhang, 2012). Recent works, such as Kouloumpis processing at the syntactic level, before analyzing arXiv:1701.05581v1 [cs.CL] 19 Jan 2017 et al. (2011), Agarwal et al. (2011) and Barbosa the sentiment. Approaches leveraging syntactic and Feng (2010), attempt to conduct such analy- properties of text include generating dependency ses on user-generated content. Sentiment analysis based rules for SA (Poria et al., 2014) and lever- remains a hard problem, due to the challenges it aging local dependency (Li et al., 2010). poses at the various levels, as summarized below. 1.3 Semantic and Pragmatic Challenges 1.1 Lexical Challenges This corresponds to the difficulties arising in the Sentiment analyzers face the following three chal- higher layers of NLP, i.e., semantic and prag- lenges at the lexical level: (1) Data Sparsity, i.e., matic layers. Challenges in these layers in- handling the presence of unseen words/phrases. clude handling: (a) Sentiment expressed implic- (e.g., The movie is messy, uncouth, incomprehen- itly (e.g., Guy gets girl, guy loses girl, audience sible, vicious and absurd) (2) Lexical Ambiguity, falls asleep.) (b) Presence of sarcasm and other forms of irony (e.g., This is the kind of movie you The rest of the paper is organized as follows. go because the theater has air-conditioning.) and Section 2 presents a summary of past work done (c) Thwarted expectations (e.g., The acting is fine. in traditional SA and SA from a psycholinguis- Action sequences are top-notch. Still, I consider tic point of view. Section 3 describes the avail- it as a below average movie due to its poor story- able datasets we have taken for our analysis. Sec- line.). tion 4 presents our features that comprise both tra- Such challenges are extremely hard to tackle ditional textual features, used for sentiment anal- with traditional NLP tools, as these need both ysis and cognitive features derived from annota- linguistic and pragmatic knowledge. Most at- tors’ eye-movement patterns. In section 5, we dis- tempts towards handling thwarting (Ramteke et cuss the results for various sentiment classification al., 2013) and sarcasm and irony (Carvalho et techniques under different combinations of textual al., 2009; Riloff et al., 2013; Liebrecht et al., and cognitive features, showing the effectiveness 2013; Maynard and Greenwood, 2014; Barbieri et of cognitive features. In section 6, we discuss on al., 2014; Joshi et al., 2015), rely on distant su- the feasibility of our approach before concluding pervision based techniques (e.g., leveraging hash- the paper in section 7. tags) and/or stylistic/pragmatic features (emoti- cons, laughter expressions such as “lol” etc). Ad- 2 Related Work dressing difficulties for linguistically well-formed Sentiment classification has been a long standing texts, in absence of explicit cues (like emoticons), NLP problem with both supervised (Pang et al., proves to be difficult using textual/stylistic fea- 2002; Benamara et al., 2007; Martineau and Finin, tures alone. 2009) and unsupervised (Mei et al., 2007; Lin and He, 2009) machine learning based approaches ex- 1.4 Introducing Cognitive Features isting for the task. We empower our systems by augmenting cogni- Supervised approaches are popular because of tive features along with traditional linguistic fea- their superior classification accuracy (Mullen and tures used for general sentiment analysis, thwart- Collier, 2004; Pang and Lee, 2008) and in such ing and sarcasm detection. Cognitive features approaches, feature engineering plays an impor- are derived from the eye-movement patterns of tant role. Apart from the commonly used bag- human annotators recorded while they annotate of-words features based on unigrams, bigrams etc. short-text with sentiment labels. Our hypothe- (Dave et al., 2003; Ng et al., 2006), syntactic prop- sis is that cognitive processes in the brain are erties (Martineau and Finin, 2009; Nakagawa et related to eye-movement activities (Parasuraman al., 2010), semantic properties (Balamurali et al., and Rizzo, 2006). Hence, considering readers’ 2011) and effect of negators. Ikeda et al. (2008) eye-movement patterns while they read sentiment are also used as features for the task of sentiment bearing texts may help tackle linguistic nuances classification. The fact that sentiment expression better. We perform statistical classification using may be complex to be handled by traditional fea- various classifiers and different feature combina- tures is evident from a study of comparative sen- tions. With our augmented feature-set, we observe tences by Ganapathibhotla and Liu (2008). This, a significant improvement of accuracy across all however has not been addressed by feature based classifiers for two different datasets. Experiments approaches. on a carefully curated held-out dataset indicate a Eye-tracking technology has been used recently significant improvement in sentiment polarity de- for sentiment analysis and annotation related re- tection over the state of the art, specifically text search (apart from the huge amount of work in with complex constructs like irony and sarcasm. psycholinguistics that we find hard to enlist here Through feature significance analysis, we show due to space limitations). Joshi et al. (2014) de- that cognitive features indeed empower sentiment velop a method to measure the sentiment anno- analyzers to handle complex constructs like irony tation complexity using cognitive evidence from and sarcasm. Our approach is the first of its kind eye-tracking. Mishra et al. (2014) study sentiment to the best of our knowledge. We share various detection, and subjectivity extraction through an- resources and data related to this work at http: ticipation and homing, with the use of eye track- //www.cfilt.iitb.ac.in/cognitive-nlp ing. Regarding other NLP tasks, Joshi et al. NB SVM RB P R F P R F P R F D1 66:15 66 66.15 64:5 65:3 64.9 56:8 60:9 53.5 D2 74:5 74:2 74.3 77:1 76:5 76.8 75:9 53:9 63.02 Table 1: Classification results for different SA systems for dataset 1 (D1) and dataset 2 (D2). P! Precision, R! Recall, F! F˙score (2013) propose a studied the cognitive aspects if quotes. Each snippet is annotated by seven par- Word Sense Disambiguation (WSD) through eye- ticipants with binary positive/negative polarity la- tracking. Earlier, Mishra et al. (2013) measure bels. Their eye-movement patterns are recorded translation annotation difficulty of a given sen- with a high quality SR-Research Eyelink-1000 eye- tence based on gaze input of translators used to tracker (sampling rate 500Hz). The annotation ac- label training data. Klerke et al. (2016) present curacy varies from 70%−90% with a Fleiss kappa a novel multi-task learning approach for sentence inter-rater agreement of 0:62. compression using labelled data, while, Barrett and Søgaard (2015) discriminate between gram- 3.2 Dataset 2 matical functions using gaze features. The recent This dataset consists of 1059 snippets comprising advancements in the literature discussed above, movie reviews and normalized tweets. Each snip- motivate us to explore gaze-based cognition for pet is annotated by five participants with positive, sentiment analysis. negative and objective labels. Eye-tracking is done We acknowledge that some of the well perform- using a low quality Tobii T120 eye-tracker (sam- ing sentiment analyzers use Deep Learning tech- pling rate 120Hz). The annotation accuracy varies niques (like Convolutional Neural Network based from 75% − 85% with a Fleiss kappa inter-rater approach by Maas et al.