Learning Negation Scope Semantics with Structure Nicholas McKenna Master of Science Artificial Intelligence School of Informatics University of Edinburgh 2019 Abstract In natural language, negation is a semantic operation that inverts meaning and is trig- gered by a word or affix. The process of determining which words, and thus which parts of meaning, are affected by the trigger is called Negation Scope Detection (NSD). This project begins with a critical review of methods in NSD and presents a new ap- proach to the task based on this research. It is hypothesized that negation scope may be determined purely through syntactic reasoning, and the methods developed test this theory. A Tree Recursive Neural Network, which uses only syntactic reasoning and does not identify words in a sentence, is shown to successfully detect negation scope with performance very close to state-of-the-art. This model confirms the project hy- pothesis and is further analyzed to better understand how it processes information. i Acknowledgements Thank you to my supervisor, Professor Mark Steedman, for all your support, construc- tive criticism, seemingly unending knowledge, and most of all for believing in me. Thank you to my family for supporting me all the way here, and for cheering me on every day. My accomplishments mean so much more when I share them with you. Thank you to my friends, both those in the UK and those many time zones away. You enlighten me, delight me, and make me appreciate everything about the journey. Finally, thank you to the city of Edinburgh which changes me every time I come back. You are a special place to me. ii Table of Contents 1 Introduction 1 1.1 Introduction to Negation and Scope Detection . .1 1.2 Research Hypothesis and Objectives . .2 1.3 Key Conclusions of this Report . .2 1.4 Contribution of this Project . .2 2 Background 3 2.1 Understanding Negation . .4 2.1.1 Negation by Example . .4 2.1.2 Combinatory Categorial Grammar . .7 2.2 Previous Work on Negation Scope Detection . .8 2.2.1 Rule-Based Models . .9 2.2.2 Classical Machine Learning Models . .9 2.2.3 Neural Network Models . 10 2.2.4 Semantic Models . 12 2.3 Other Related Work . 13 3 Description of Work 15 3.1 Reframing the Problem . 15 3.2 Data Preprocessing . 15 3.3 The Graph Convolutional Neural Network . 16 3.3.1 Motivation . 16 3.3.2 Experiments and Iteration . 17 3.4 The Tree Recursive Neural Network . 18 3.4.1 Motivation . 18 3.4.2 TRNN Learned Parameters . 18 3.4.3 The Upward Pass . 19 iii 3.4.4 The Downward Pass . 20 3.4.5 Classification . 21 3.5 TRNN Model Variations . 21 3.6 Optimization . 22 3.6.1 Optimization and Objective Function . 22 3.6.2 Regularization . 23 4 Evaluation 24 4.1 Performance of the Negation Scope Detector Learned with Structure . 24 4.1.1 TRNN Self-Comparison with Ablation Studies . 26 4.1.2 Comparison of the TRNN to Other Models . 27 4.1.3 A Note on TRNN Performance Testing . 28 4.2 Analysis of Model Predictions . 28 4.2.1 Tree Reporting Notation . 28 4.2.2 Overall Performance . 29 4.2.3 Error Analysis . 32 4.3 Qualitative Inferences about Scope Colorings . 34 4.3.1 Is Scope Processed in Distinguishable Grammatical Units? . 35 4.3.2 When is Scope Propagation Blocked? . 35 5 Conclusion 36 5.1 Conclusions from this Work . 36 5.2 Directions for Future Work . 37 Bibliography 38 A Supplementary Parse Tree Colorings 43 iv Chapter 1 Introduction 1.1 Introduction to Negation and Scope Detection In linguistics and formal logic, negation is an important semantic operator which in- verts the meaning of an expression [16]. Properly understanding the impact of negation cues in language is thus vital for understanding an utterance, both for humans and al- gorithms. For example, imagine a software agent which makes rapid stock trading decisions by reading online news articles. It might make drastically different decisions if it encounters the sentence “British imports will be severely impacted” vs. “British imports will not be severely impacted.” The simple word “not” critically shifts the meaning of what is expressed and has downstream consequences for decision making. Understanding negation is broadly important for many automated tasks beyond this, like web search, conversing with personal assistants, machine translation, and others. Negation Scope Detection is typically a two-stage process for identifying which words of a sentence are affected by a negation cue. In the first stage, a negation cue is detected in the text, such as the words “not” and “without,” or affixes like “un-”. In the second stage, spans of text forming the negation scope are identified and judged “in scope” of the cue. This is most easily done by producing individual binary label judgments for each word of the sentence. This second stage is the focus of this project, as it is the most interesting part. The rules for determining scope can differ depending on parsing goals, but one useful method seeks to identify affected semantic events and related arguments. Such a goal formulation requires an automated system to learn the semantics of how these elements should be selected by leveraging other sources of information. Several approaches are explored in the Background chapter and an argument is made for relying on syntax to identify scope. 1 Chapter 1. Introduction 2 1.2 Research Hypothesis and Objectives The Background chapter describes a history of approaching this task as a sequence tagging problem, whereby input sentences are considered as a linear sequence of words and scope judgments are made in this context. These techniques occasionally make use of syntactic features in the decision process, but infrequently make use of full and explicit syntactic structure despite theoretical evidence of syntactic ties [36, 39]. This project poses a research question to investigate this disparity. Can negation scope semantics be learned from explicit sentence structure, in terms of larger phrase constituencies? Further, negation scopes share common traits in their structural com- position. Can scope be identified solely by inference from sentence structure? The objective of this project is to reframe NSD from a sequence tagging problem to a tree tagging problem, leveraging explicit syntactic parse trees to make scope deci- sions about constituents at the surface text and also larger constituent units in the tree. A successful machine learner developed for this task should thus learn scope semantics for every syntactic composition and produce accurate predictions in the surface text. 1.3 Key Conclusions of this Report Evaluating the experiments carried out in this report shows that explicit syntactic pro- cessing does provide an excellent means for detecting scope. The model developed here achieves near-state of the art performance on the *SEM2012 dataset, and does so without the use of a vocabulary or any other word features as do most successful pub- lished methods. This method relies solely on syntactic parse trees and a Tree Recursive Neural Network to process them compositionally, demonstrating that explicit syntactic processing alone can effectively model negation scope determination in a sentence. 1.4 Contribution of this Project The major products of this project are a new understanding of negation scope detection using explicit syntactic processing, and a model demonstrating this. This result devi- ates from common methodologies on the task which frequently undervalue the con- tribution of syntax and combine many different features to compute scope judgments. This method uses only syntax and performs at nearly the state of the art. Chapter 2 Background Negation is a tool we use in language without even thinking about it. It appears to be superficially straightforward to understand how negation operators affect the mean- ing of our utterances. However, exploring deeper reveals that it can quickly become complicated. This chapter has two purposes: to describe some of the linguistic and technical phenomena that relate to negation scope, and to analyze the history of com- putational methods developed to solve the task of Negation Scope Detection (NSD). This background serves to motivate both the project hypothesis and methods. It is important to first introduce the major NSD resources and to formally define negation scope, which will help frame the rest of the background review. NSD re- search became popularized with the releases of the Genia and BioScope medical cor- pora [19, 41]. These provided resources for many developments but are centered on the medical domain (and writing style), and until the *SEM2012 Shared Task compe- tition in NSD there were no general-domain resources. Additionally, *SEM updated the guidelines for determining negation scope from the mostly syntactic annotation guidelines of BioScope [22] to a “semantic” definition of the task [1, 2]. *SEM de- fines NSD: “the negation cue will be such that it allows to determine which events are negated in the sentence,” with scope that also encompasses semantic arguments and complements of those events. They provide a detailed handbook of these rules [35]. This project approaches NSD as defined by *SEM with analysis based on this goal. 3 Chapter 2. Background 4 2.1 Understanding Negation 2.1.1 Negation by Example Negation scope is best understood by example. This section walks through several sample sentences from the *SEM dataset of Conan Doyle writing [1] to introduce concepts and nuances in negation, and set the context for review of existing methods in NSD. In these sentences, negation triggers are written in boldface and starred* while their resulting scopes are underlinedunderlined.underlined As per *SEM convention, a trigger is never contained in its own scope.