Application of Common Sense Computing for the Development of a Novel Knowledge-Based Opinion Mining Engine A thesis submitted in accordance with the requirements of the University of Stirling for the degree of Doctor of Philosophy by Erik Cambria Principal Supervisor: Amir Hussain (University of Stirling, UK) Additional Supervisor: Catherine Havasi (MIT Media Laboratory, USA) Industrial Supervisor: Chris Eckl (Sitekit Solutions Ltd, UK) Department of Computing Science & Mathematics University of Stirling, Scotland, UK December 2011 I Declaration I, Erik Cambria, hereby declare that this work has not been submitted for any other degree at this University or any other institution and that, except where reference is made to the work of other authors, the material presented is original. Erik Cambria II Abstract The ways people express their opinions and sentiments have radically changed in the past few years thanks to the advent of social networks, web communities, blogs, wikis and other online collaborative media. The distillation of knowledge from this huge amount of unstructured information can be a key factor for marketers who want to create an image or identity in the minds of their customers for their product, brand, or organisation. These online social data, however, remain hardly accessible to computers, as they are specifically meant for human consumption. The automatic analysis of online opinions, in fact, involves a deep understanding of natural language text by machines, from which we are still very far. Hitherto, online information retrieval has been mainly based on algorithms relying on the textual representation of web-pages. Such algorithms are very good at retrieving texts, splitting them into parts, checking the spelling and counting their words. But when it comes to interpreting sentences and extracting meaningful information, their capabilities are known to be very limited. Existing approaches to opinion mining and sentiment analysis, in particular, can be grouped into three main categories: keyword spotting, in which text is classified into categories based on the presence of fairly unambiguous affect words; lexical affinity, which assigns arbitrary words a probabilistic affinity for a particular emotion; statistical methods, which calculate the valence of affective keywords and word co-occurrence frequencies on the base of a large training corpus. Early works aimed to classify entire documents as containing overall positive or negative polarity, or rating scores of reviews. III Such systems were mainly based on supervised approaches relying on manually la- belled samples, such as movie or product reviews where the opinionist’s overall positive or negative attitude was explicitly indicated. However, opinions and sentiments do not occur only at document level, nor they are limited to a single valence or target. Contrary or complementary attitudes toward the same topic or multiple topics can be present across the span of a document. In more recent works, text analysis granu- larity has been taken down to segment and sentence level, e.g., by using presence of opinion-bearing lexical items (single words or n-grams) to detect subjective sentences, or by exploiting association rule mining for a feature-based analysis of product reviews. These approaches, however, are still far from being able to infer the cognitive and af- fective information associated with natural language as they mainly rely on knowledge bases that are still too limited to efficiently process text at sentence level. In this thesis, common sense computing techniques are further developed and ap- plied to bridge the semantic gap between word-level natural language data and the concept-level opinions conveyed by these. In particular, the ensemble application of graph mining and multi-dimensionality reduction techniques on two common sense knowledge bases was exploited to develop a novel intelligent engine for open-domain opinion mining and sentiment analysis. The proposed approach, termed sentic com- puting, performs a clause-level semantic analysis of text, which allows the inference of both the conceptual and emotional information associated with natural language opinions and, hence, a more efficient passage from (unstructured) textual information to (structured) machine-processable data. The engine was tested on three different resources, namely a Twitter hashtag repos- itory, a LiveJournal database and a PatientOpinion dataset, and its performance com- pared both with results obtained using standard sentiment analysis techniques and using different state-of-the-art knowledge bases such as Princeton’s WordNet, MIT’s ConceptNet and Microsoft’s Probase. Differently from most currently available opin- ion mining services, the developed engine does not base its analysis on a limited set of IV affect words and their co-occurrence frequencies, but rather on common sense concepts and the cognitive and affective valence conveyed by these. This allows the engine to be domain-independent and, hence, to be embedded in any opinion mining system for the development of intelligent applications in multiple fields such as Social Web, HCI and e-health. Looking ahead, the combined novel use of different knowledge bases and of common sense reasoning techniques for opinion mining proposed in this work, will, eventually, pave the way for development of more bio-inspired approaches to the design of natural language processing systems capable of handling knowledge, retrieving it when necessary, making analogies and learning from experience. V Dedication In memory of John McCarthy (September 4th, 1927 - October 24th, 2011), who helped design the foundation of today’s Internet-based computing and coined the term for a frontier of research he helped pioneer, AI. VI Acknowledgements The humble accomplishment of this thesis would not have been possible without the contribution of many individuals, to whom I express my appreciation and gratitude. Firstly, I am deeply indebted to my research supervisors, specifically: Amir Hussain, my principal supervisor and founding Head of COSIPRA Lab at Stirling, who gave me the opportunity to work on this exciting project, guided me every step of the way, funded my research and training visits to the project’s prestigious partner institutions, and was an immense source of inspiration throughout; Catherine Havasi, whose guidance, encouragement and support in the past three years has been simply invaluable; and Chris Eckl, who helped me to look at my research from many different points of view. I am also grateful to all the other mentors and colleagues who supported me during my short term scientific missions and internships, in particular Robert Speer, Kenneth Arnold, Dustin Smith, Jason Alonso and Henry Lieberman, who assisted me in get- ting started, using and further developing common sense computing tools (both during my research visits at MIT Media Lab and remotely), Campbell Grant, for his critical opinions and sound advice about the commercial aspects of my research work, Anna Esposito, for her unwavering support within the COST 2102 program, Thomas Maz- zocco and Marco Grassi, for their invaluable research contributions, Joseph Lyons, for his help in the design and refinement of the Hourglass model, and James Munro, for the support and the data provided for the development of patient centred applications. Special thanks also go to Praphul Chandra and Sudhir Dixit, who helped me expand the horizons of my research during my internship at Hewlett-Packard Labs India (within VII the Innovations for the Next Billion Customers Initiative) in Bangalore, Tariq Durrani, Tieniu Tan, Cheng-Lin Liu and Chengqing Zong, for their guidance during my research visit at the National Laboratory of Pattern Recognition (NLPR) in the Institute of Automation of the Chinese Academy of Sciences (within the China-Scotland SIPRA programme) in Beijing, and Haixun Wang and Yangqiu Song, for helping me improve my skills and expertise in the field of knowledge-based systems during my internship at Microsoft Research Asia (within the Probase Project) in Beijing. A last, but not least, acknowledgement goes to my family and all the old and new friends who, in the past three years, have cheered me up in difficult moments, celebrated with me for my achievements and never blamed me for being often far away from them. This PhD was the best experience of my life as it gave me the possibility to: work shoulder to shoulder with scientists from top research institutes, get in touch with both far West and far East cultures, meet special people who will always be part of my life, and even exchange ideas with beautiful minds such as Tim Berners-Lee, the inventor of the Web, Bebo White, Web pioneer in the United States, and Marvin Minsky, one of the fathers of AI. VIII IX Contents 1 Introduction 1 1.1 TheThesis .................................. 2 1.1.1 Motivations . 3 1.1.2 Aims.................................. 7 1.1.3 Original Contributions . 10 1.2 PublicationsArising ............................. 12 1.2.1 Accepted for Publication . 13 1.2.2 UnderReview............................. 15 1.2.3 In Preparation . 16 2 Background 17 2.1 Opinion Mining and Sentiment Analysis . 18 2.1.1 TheBuzzMechanism ........................ 18 2.1.2 Origins and Peculiarities . 19 2.1.3 Sub-Tasks............................... 21 2.2 Main Approaches to Opinion Mining . 23 2.2.1 From Heuristics to Discourse Structure . 24 2.2.2 From Coarse to Fine Grained . 25 2.2.3 From Keywords to Concepts . 26 2.3 Towards Machines with Common Sense . 28 2.3.1 The Importance of Common Sense . 29 X 2.3.2 Knowledge Representation . 31 2.3.3 History . 37 2.3.4 The Open Mind Common Sense Project . 41 2.3.5 SenticComputing .......................... 45 2.4 Conclusions .................................. 47 3 Sentic Knowledge Base Design 49 3.1 AffectNet: An Affective Common Sense Knowledge Base . 51 3.1.1 ConceptNet . 51 3.1.2 AffectiveBlending .......................... 54 3.2 The Hourglass of Emotions: A Novel Emotion Categorisation Model . 57 3.2.1 Categorical Versus Dimensional Approaches . 59 3.2.2 A New Cognitive Model for Representing Human Emotions .