Enhancing Neural Data-To-Text Generation Models with External Background Knowledge Shuang Chen1,∗ Jinpeng Wang2, Xiaocheng Feng1, Feng Jiang1;3, Bing Qin1, Chin-Yew Lin2 1 Harbin Institute of Technology, Harbin, China 2 Microsoft Research Asia 3 Peng Cheng Laboratory [email protected], fjinpwa, [email protected], fxcfeng, [email protected], [email protected] Abstract Infobox Description Recent neural models for data-to-text genera- Nacer Hammami (born December 28, 1980) is an Algerian football player who is tion rely on massive parallel pairs of data and currently playing for MC El Eulma in the text to learn the writing knowledge. They of- Algerian Ligue Professionnelle 1. ten assume that writing knowledge can be ac- quired from the training data alone. However, Entity Linking when people are writing, they not only rely Subject Relation Object Guelma country Algeria on the data but also consider related knowl- (Q609871) (P17) (Q262) edge. In this paper, we enhance neural data-to- defender instance of association football position text models with external knowledge in a sim- (Q336286) (P31) (Q4611891) …… … ple but effective way to improve the fidelity MC El Eulma league Algerian Ligue Professionnelle 1 of generated text. Besides relying on parallel (Q2742749) (P118) (Q647746) data and text as in previous work, our model External Background Knowledge from Wikidata attends to relevant external knowledge, en- Figure 1: An example of generating description from coded as a temporary memory, and combines a Wikipedia infobox. External background knowledge this knowledge with the context representa- expanded from the infobox is helpful for generation. tion of data before generating words. This al- lows the model to infer relevant facts which are not explicitly stated in the data table from an external knowledge source. Experimen- determines how to generate the text based on se- tal results on twenty-one Wikipedia infobox- lected contents. Traditionally, these two sub- to-text datasets show our model, KBAtt, con- problems have been tackled separately. In recent sistently improves a state-of-the-art model on years, neural generation models, especially the most of the datasets. In addition, to quantify encoder-decoder model, solve these two subprob- when and why external knowledge is effective, lems jointly and have achieved remarkable suc- we design a metric, KBGain, which shows a cesses in several benchmarks (Mei et al., 2016; strong correlation with the observed perfor- Lebret et al., 2016; Wiseman et al., 2017; Dusekˇ mance boost. This result demonstrates the rel- evance of external knowledge and sparseness et al., 2018; Nie et al., 2018). of original data are the main factors affecting Such end-to-end data-to-text models rely on system performance. massive parallel pairs of data and text to learn the 1 Introduction writing knowledge. They often assume that all writing knowledge can be learned from the train- Automatic text generation from structured data ing data. However, when people are writing, they (data-to-text) is a classic task in natural language will not only rely on the data contents themselves generation which aims to automatically gener- but also consider related knowledge, which is ne- ate fluent, truthful and informative texts based glected by previous methods. For example, as on structured data (Kukich, 1983; Holmes-Higgin, shown in Fig.1, an infobox about a person called 1994; Reiter and Dale, 1997). Data-to-text is of- Nacer Hammami is paired with its corresponding ten formulated into two subproblems: content se- biography description from the Wikipedia. How- lection which decides what contents should be in- ever, the information in the infobox is not enough cluded in the text and surface realization which to cover all the facts mentioned in the descrip- ∗Contribution during internship at Microsoft Research. tion. To generate this description from the in- 3022 Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3022–3032, Hong Kong, China, November 3–7, 2019. c 2019 Association for Computational Linguistics fobox, we need to expand information based on performance. external background knowledge from its related The contributions of our work can be summa- entities. For example, in the description: 1) “is rized as follows: an Algerian football player” indicates the nation- ality of Nacer Hammami which is not explicitly • We demonstrate that external knowledge base stated in the infobox. However, the place of birth, could be used to enhance the performance of Guelma, of Nacer Hammami is given, therefore neural data-to-text models. the nationality can be inferred from the knowledge • We propose a simple yet effective model, that Guelma is a place in Algeria. 2) “playing for KBAtt, to integrate external knowledge using MC El Eulma in the Algerian Ligue Profession- a dual-attention mechanism. nelle 1” describes the fact that MC El Eulma is a club in the Algerian Ligue Professionnelle 1 which • We design a metric, KBGain, to quantify is also not explicitly stated in the infobox which when and why external knowledge is effec- can be expanded from external knowledge base. tive. One may argue that neural models can • We contribute twenty infobox-to-text learn such knowledge when enough parallel co- datasets from a variety of domains. occurrence pairs such as (Guelma, Algerian) and (MC El Eulma, Algerian Ligue Professionnelle 1) are available. However even in such case, neu- 2 The Proposed Model ral models still tend to make mistakes for sparse Our model takes a data table D (e.g., a Wikipedia co-occurrence pairs as we will show in the experi- infobox) and a relevant external knowledge base ments section. (KB) containing a set of facts F as input and gen- In this paper we enhance neural-network- erates a natural language text y = y1; :::; yT con- based data-to-text generation models with external sisting of T words. To augment the infobox with knowledge in a simple but effective way. Besides external knowledge, we preserve the Wikipedia in- learning the association between data and text ternal hyperlink information in the field values of from parallel data-text pairs as in previous work, infobox, and track these hyperlinks to get their our model attends to relevant external knowledge, corresponding entities from Wikidata2 where we encoded as a temporary memory, and combines retrieve only one-hop facts. The backbone of our this knowledge with the context representation of model is an attention based sequence-to-sequence data before generating words. Specifically, both model (Bahdanau et al., 2014) equipped with copy infobox and background knowledge facts are en- mechanism (See et al., 2017). As shown in Fig.2, coded and a dual-attention mechanism is proposed the model consists of four main components: a to guide the decoder to generate text. table encoder, a KB encoder, the dual attention To verify the effectiveness of our proposed mechanism and a decoder. We describe each com- model, Knowledge Base enhanced Attention- ponent in the following sections. based sequence-to-sequence network (KBAtt), we conduct experiments on multiple Wikipedia 2.1 Table Encoder infobox-to-text datasets including WikiBio (Le- In Fig.2, the input data table D consists of sev- 1 bret et al., 2016) and 20 new datasets . Our ex- eral field name and field value pairs. We follow periment results show that KBAtt consistently im- (Sha et al., 2017; Liu et al., 2017) to tokenize the proves a state-of-the-art neural data-to-text model field values and transform the input table into a to achieve higher performances on most of the N flattened sequence f(ni; vi)gi=1, where each ele- datasets. To quantify when and why external ment is a token vi from a field value paired with knowledge is effective, we design a metric which its corresponding field name ni. To encode the shows a strong correlation with the observed per- flattened table, we map each (ni; vi) to vector n v n v formance boost. This result demonstrates the rel- xi = [e i ; e i ], where e i and e i are trainable evance of external knowledge and sparseness of 2We adopt Wikidata (dumps version: 20150831) as the original data are the main factors affecting system external knowledge base. Although we extend the infobox with external knowledge by using the Wikipedia hyperlink, 1Available at https://github.com/hitercs/ we can also apply entity linking to link input data to the WikiInfo2Text knowledge base in practice. 3023 Field Name Field Value Table Encoder Decoder ࢚࢝ Field Name ሺ࢔࢏ሻ Content ሺ࢜࢏ሻ Full name Nacer Hammami Full name Nacer ࢎ૚ softmax Place of birth Guelma Full name Hammami Playing position Defender Flatten Table ࢎ૛ ࢚ࢇ࢈࢒ࢋ …… GRU Table Attention ࢉ࢚ MLP Current team MC El Eulma Current team Eulma ࢎࡺ Entity Linking ࢊ࢚ Field Name ሺ࢔࢐ሻ Subject ሺ࢙࢐ሻ Relation ሺ࢘࢐ሻ Object ሺ࢕࢐ሻ KB Encoder Guelma country Algeria Place of birth ࢎ GRU (Q609871) (P17) (Q262) ૚ …… … … ࢎ KB Attention ࢑࢈ ࢐ ࢉ࢚ ࢉ࢚࢞ MC El Eulma league Algerian Ligue Professionnelle 1 ࢉ࢚ି૚ ࢊ࢚ି૚ ࢚࢝ି૚ Current team (Q2742749) (P118) (Q647746) ࢎࡺ Figure 2: A diagram of the knowledge base enhanced neural data-to-text generation model. First, we transform the table into a flattened sequence, extract entities mentioned in the field value of the infobox and link them to Wikidata where we can retrieve relevant facts. Then, the table contents and external knowledge base facts are carefully encoded. Finally, a single layer GRU decoder with a dual attention mechanism decides which part of information should be used for generation. word embeddings of ni and vi, and [:; :] is the con- to generate words conditioned on both table infor- catenation of vectors. Then each xi is encoded mation and background knowledge fact informa- into a hidden vector hi using a bi-directional GRU tion.