TALN-UPF: Taxonomy Learning Exploiting CRF-Based Hypernym Extraction on Encyclopedic Definitions Luis Espinosa-Anke and Horacio Saggion and Francesco Ronzano Tractament Automatic` del Llenguatge Natural (TALN) Department of Information and Communication Technologies Universitat Pompeu Fabra Carrer Tanger,` 122-140 08018 Barcelona, Spain luis.espinosa, horacio.saggion, francesco.ronzano @upf.edu { } Abstract Given this rationale, Task 17 (Bordea et al., 2015) in the SEMEVAL 2015 set of shared tasks focuses This paper describes the system submitted on Taxonomy Extraction Evaluation, i.e. the con- by the TALN-UPF team to SEMEVAL Task struction of a taxonomy out of a flat set of terms be- 17 (Taxonomy Extraction Evaluation). We longing to one of the four domains of choice (food, present a method for automatically learning chemical, equipment and science). These terms have a taxonomy from a flat terminology, which benefits from a definition corpus obtained to be hierarchically organized, and new terms are al- by querying the BabelNet semantic network. lowed to be included in the taxonomy. As for eval- Then, we combine a machine-learning al- uation, for each domain, two taxonomies were used gorithm for term-hypernym extraction with as gold standard: One created by domain experts; linguistically-motivated heuristics for hyper- and one derived from the WordNet taxonomy rooted nym decomposition. Our approach performs at the domain node, e.g. food1. Finally, evaluation well in terms of vertex coverage and newly is carried out from two standpoints: (1) The taxon- added vertices, while it shows room for im- provement in terms of graph topology, edge omy topology and the rate of replicated nodes and coverage and precision of novel edges. edges are taken into account when compared to a gold standard taxonomy; and (2) Human experts val- idated as correct or incorrect a subset of the newly 1 Introduction added edges. Learning semantic relations out of flat terminologies In this paper we describe our contribution to is an appealing task due to its potential application this shared task. Our approach relies on a set of in tasks like Question Answering (Cui et al., 2005; definitional sentences for each term, from which term hypernym relations are extracted using a Boella et al., 2014), automatic glossary construction → (Muresan and Klavans, 2002), Ontology Learning machine-learning classifier. In a second step, (Navigli et al., 2011) or Textual Entailment (Roller linguistically-motivated rules are applied in order to et al., 2014). Today, in the context of massive web- (1) extract a hypernym candidate when the confi- enabled data, hypernym (is-a) relations are the focus dence of the classifier was below a threshold, and of much research, as they constitute the backbone of (2) decompose multiword hypernyms in more gen- eral concepts (e.g. from coca-cola carbonated soft ontologies (Navigli et al., 2011). However, one chal- → drink to carbonated soft drink soft drink and soft lenge remains open in the automatic construction of → drink drink). knowledge bases that exploit this type of relation. It → is unfeasible to have up-to-date semantic resources 1 for each domain, as they are limited in scope and For our domain notation we simply use the name of the domain for manually constructed taxonomies (e.g. “food”), domain, and their manual construction is knowledge and add the prefix wn for the WordNet taxonomies (e.g. intensive and time consuming (Fu et al., 2014). “wn food”). 949 Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 949–954, Denver, Colorado, June 4-5, 2015. c 2015 Association for Computational Linguistics The remainder of the paper is structured as fol- pus. For example, given the term botifarra (a Cata- lows: Section 3 describes the modules of our ap- lan type of sausage), we add two definitions to our proach, Section 4 presents and discusses the evalua- corpus: tion procedure as well as results, and finally Section 5 analyzes the performance of our system as well as Relevant: Botifarra is a type of sausage and the difficulties encountered, and suggests potential one of the most important dishes of the Catalan avenues for future work. cuisine. Noisy: Botifarra is a point trick-taking card 2 Background game for four players in fixed partnerships Generally, taxonomy learning from text has been played in Catalonia. carried out either following rule-based or distribu- 3.2 Hypernym Extraction tional approaches. In terms of rule-based meth- Given a set of definitional text fragments where the ods reported in the literature, (Hearst, 1992) in- 2 troduced lexico-syntactic patterns, which were ex- definiendum term is known, i.e. can be extracted ploited in subsequent work (Berland and Charniak, from the url of the Wikipedia page, our goal is to tag 1999; Kozareva et al., 2008; Widdows and Dorow, the tokens of the definition that correspond to one or more hypernyms. To this end, we train a Conditional 2002; Girju et al., 2003). Distributional approaches, 3 on the other hand, have become increasingly pop- Random Fields (Lafferty et al., 2001) classifier with ular due to the availability of large corpora. Sys- the WCL Dataset (Navigli and Velardi, 2010). We tems aimed at extracting hypernym relations from argue that CRFs are a valid approach for sequential text have exploited hybrid patterns as word-class lat- classification, and particularly for this task, due to tices (Navigli and Velardi, 2010), syntactic relations their potential to capture prior and posterior token as features for an SVM classifier (Boella et al., 2014) features on the current iteration. The WCL dataset or word-embedding-based semantic projections (Fu includes near 2000 definitional sentences with terms et al., 2014; Roller et al., 2014). Inspired by the re- and hypernyms manually annotated. We preprocess ported success in the latter methods, we opted for and parse the WCL dataset with a dependency parser combining syntactic patterns with machine learning (Bohnet, 2010), and then train our classifier with the to extract hypernyms from domain sentences. following set of features. surface: A word’s surface form. 3 Method lemma: The lemma of the word. This section describes the main modules that consti- tute our taxonomy learning system. pos: The word’s part-of-speech. 3.1 Definition corpus compilation head id: The id of the word to which the cur- rent token depends in a dependency syntactic We benefit from BabelNet, a very large multilin- tree. gual semantic network that combines, among other resources, Wikipedia and WordNet (Navigli and deprel: Syntactic function of the current word Ponzetto, 2010). We get a set of BabelNet synsets in relation to its head. associated to each term and for each synset, we ex- tract its definition. In this step we assume that a def—nodef: Whether the current token ap- term’s definition appears in the first sentence of its pears before or after the first verb of the sen- Wikipedia article, which is a regular practice in the tence. literature (see (Navigli and Velardi, 2010) or (Boella 2The classic components lexicographic genus-et-differentia et al., 2014)). This step allowed us to compile a definition are (1) Definiendum (concept being defined); (2) genus (hypernym or immediate superordinate that describes the domain corpus of definitional knowledge, and thus definiendum); and (3) definiens or cluster of words that differ- maximizing the number of relevant terms defini- entiate a definiendum from others of its kind. tions. However, noise is also introduced in our cor- 3https://code.google.com/p/crfpp/ 950 term—noterm: Whether the token is part of Nominal (NMOD). If, however, such node is a stop- the definiendum term or not. hypernym, we go down the syntactic tree one level and look for a direct Preposition node with syntactic Our CRF classifier learns the above word-level function NMOD. Then, we extract this preposition’s features in a word window of [ 2, 2]. The predic- − adjective and noun children if they have the syntac- tion the classifier must learn follows the classic BIO tic function Modifier of Prepositional (PMOD). format, i.e. whether a word is at the beginning of For example, consider the following sentence: a hypernym phrase, inside or outside. We evalu- “Whisky or whiskey is a type of distilled alcoholic ate this hypernym extraction module on the WCL beverage made from fermented grain mash”. Here, dataset (Navigli and Velardi, 2010) performing 10- type is the Predicative Complement node but it is an fold cross-validation. It achieves and F-score of uninformative word for describing the term whisky. 79.86, outperforming existing state-of-the-art sys- Therefore, our algorithm goes one level down the tems described in the literature (Navigli and Velardi, syntactic tree and identifies the token beverage as 2010; Boella et al., 2014). the direct child of the preposition and therefore ex- Despite the good performance of this module, tracts this token as hypernym. we observe two potential drawbacks in terms of its fitness for the taxonomy learning task. Firstly, 3.3 Hypernym Decomposition we aim at recovering hypernym candidates even in This step is aimed at generating deeper paths from cases in which they are predicted with low confi- a term and its hypernym by recursively decompos- dence at the classification step. We build on the as- ing a candidate hypernym. For example, consider sumption that all encyclopedic definitions are very the previous example’s term hypernym relation if → likely to include a hypernym, and hypothesize that the hypernym’s modifiers are taken into account: it will help increasing recall while keeping preci- whisky distilled alcoholic beverage. Our objective → sion at a reasonable rate. Secondly, when a mul- is to generate the following set of relations: distilled tiword hypernym is retrieved by our module, it alcoholic beverage alcoholic beverage and alco- → might not match exactly a term from the seed ter- holic beverage beverage.