Learning to Define Relations Between Entities
Total Page：16
File Type：pdf, Size：1020Kb
Open Relation Modeling: Learning to Define Relations between Entities Jie Huang Kevin Chen-Chuan Chang University of Illinois at Urbana-Champaign University of Illinois at Urbana-Champaign USA USA [email protected] [email protected] Jinjun Xiong Wen-mei Hwu University at Buffalo University of Illinois at Urbana-Champaign USA USA [email protected] [email protected] ABSTRACT while deep learning is used heavily recently for natural language Relations between entities can be represented by different instances, processing. e.g., a sentence containing both entities or a fact in a Knowledge Besides, even relations are represented, they may not be inter- Graph (KG). However, these instances may not well capture the pretable to humans. There are different ways to represent relations general relations between entities, may be difficult to understand between entities. For example, if two entities co-occur in a sentence, by humans, even may not be found due to the incompleteness of they are possibly related and the relation can be implied by the sen- the knowledge source. tence. From a structured perspective, a relation can be represented In this paper, we introduce the Open Relation Modeling task– as a fact or a multi-hop reasoning path between two entities in a given two entities, generate a coherent sentence describing the Knowledge Graph (KG). However, for humans without too much relation between them. To solve this task, we propose to teach ma- prior knowledge about the entities, it is still difficult to understand chines to generate definition-like relation descriptions by letting the relations by reading them. For example, from sentence “we study them learn from definitions of entities. Specifically, we fine-tune data mining and database” or fact “(data mining, facet of, database)”, Pre-trained Language Models (PLMs) to produce definitions con- humans can guess data mining and database are related fields, but ditioned on extracted entity pairs. To help PLMs reason between they cannot know exactly how they are related. Besides, due to entities and provide additional relational knowledge to PLMs for the limited size of the corpus or the incompleteness of the KG, for open relation modeling, we incorporate reasoning paths in KGs many related entities, we are not able to extract a sentence or a fact and include a reasoning path selection mechanism. We show that containing both entities. PLMs can select interpretable and informative reasoning paths by Based on the above observation, a system for exploring rela- confidence estimation, and the selected path can guide PLMs to tions between entities needs to meet the following requirements: generate better relation descriptions. Experimental results show 1) interpretability: the system can provide interpretable relation that our model can generate concise but informative relation de- descriptions, with which humans can easily understand relations be- scriptions that capture the representative characteristics of entities tween entities; 2) openness: the system can deal with a wide range and relations.1 of related entities, including those neither co-occur in a corpus nor be connected in a knowledge graph, where types of relations are KEYWORDS not required to be explicitly pre-specified. To achieve a system meeting with the above requirements, we open relation modeling, relation description, text generation introduce a novel task– Open Relation Modeling, i.e., generating coherent sentences describing general relations between entities, 1 INTRODUCTION where types of relations do not need to be pre-specified. Different arXiv:2108.09241v1 [cs.CL] 20 Aug 2021 People are always interested in relations between entities. To learn from open relation extraction, which aims to extract relational facts about a new concept, people want to know how this concept relates between entities from an open-domain corpus [4], open relation to the ones they are familiar with; when getting two related entities modeling aims to generate a concise but informative sentence, cap- of interest, people ask how exactly they are related. turing the representative characteristics of the given entities and However, although existing systems identify related entities, their relation. From the perspective of interpretability, compared they do not provide features for exploring relations between en- to open relation extraction whose outputs are phrases with low tities. For instance, in Figure 1, the top is the ScienceDirect Topics interpretability, e.g., (data mining methods, to be integrate within, feature of Elsevier, which lists several related terms without any the framework of traditional database systems) by Ollie [28], open annotation; the bottom is the “see also” feature of Wikipedia, where relation modeling improves the interpretability of entity relations. the annotation of deep learning is not specific to the context of nat- For example, for data mining and database, we want to generate ural language processing. Users cannot get how deep learning and a relation description like “data mining is a process of extracting natural language processing are related by reading the annotation, and discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems”. 1Code and data are available at https://github.com/jeffhj/open-relation-modeling. Figure 1: Examples of two current services: Elsevier’s ScienceDirect Topics (top) and Wikipedia’s “see also” (bottom), both of which lack open relation modeling. Such a relation description is informative and easy to understand We conduct both quantitative and qualitative experiments to since it contains important and precise information about entities validate the effectiveness of the proposed methods. Experimental and their relation. results show that, after learning from definitions of entities, PLMs To solve the task, we propose to teach machines to learn from have a great ability to describe relations between entities concisely definitions of entities. Definitions of entities are highly summarized and informatively. By incorporating reasoning paths and including sentences that capture the most representative characteristics of the reasoning path selection mechanism, machines can often gener- entities, where the general relations between the defined entity ate relation descriptions well capturing relations between entities, and other entities in the definitions are well captured. Therefore, where only minor errors that do not affect the understanding of we suggest to find the general relation between two entities by relations are included. We also show that PLMs can select inter- defining one entity in terms of the other entity. To achieve this, pretable and informative reasoning paths by confidence estimation, we first collect definitions of entities and extract entity pairs from which can in turn guide and help PLMs to generate more reasonable the definitions. Then we teach machines to generate definition-like and precise relation descriptions with the desired characteristics. relation descriptions by training a language generation model to Additionally, we conduct error analysis for the proposed methods produce definitions of entities conditioned on extracted entity pairs. and suggest several directions for future work. To generate informative relation descriptions, machines need The main contributions of our work are summarized as follows: knowledge about entities and relations. Therefore, we apply Pre- • We introduce and study the novel open relation modeling task– trained Language Models (PLMs) [6, 16, 24, 25], which have re- generating coherent sentences describing general relations be- cently been shown to contain rich relational knowledge of entities tween entities. [19, 23, 27, 35]. PLMs are usually first pre-trained on large un- • To solve this task, we propose a novel idea of teaching machines labeled corpora via self-supervision with auxiliary tasks such as to generate definition-like relation descriptions by letting them masked language modeling and next-sentence prediction [8]. They learn from producing definitions conditioned on extracted entity are typically deep neural networks with multiple (typically 12 or 24) pairs. Transformer layers [31]. After pre-training, rich world knowledge • We apply PLMs and design reasoning path-enriched PLMs for is stored in the huge number of parameters of PLMs, e.g., knowing open relation modeling. Dante was born in Florence [23]. During fine-tuning, PLMs can be • We show that PLMs can select interpretable and informative rea- adapted to specific tasks and incorporate task-specific knowledge soning paths by confidence estimation, and the selected reasoning from the training data. Thus, PLMs have a great ability to model path can guide PLMs to generate better relation descriptions. relations between entities and have the potentials to describe rela- • We release a large benchmark dataset and conduct extensive tions informatively. experiments for the proposed open relation modeling task. To utilize knowledge to describe relations between entities, ma- chines also need to reason between entities. However, recent studies have shown that, although PLMs contain rich world knowledge, 2 OPEN RELATION MODELING their reasoning ability is weak [10, 26, 38]. To alleviate this issue, we incorporate reasoning paths in KGs, which can help PLMs do 2.1 Problem Statement multi-hop reasoning and provide additional relational knowledge to The problem of Open Relation Modeling