The WebNLG Challenge: Generating Text from DBPedia Data

Emilie Colin1 Claire Gardent1 Yassine M’rabet2 Shashi Narayan3 Laura Perez-Beltrachini1 1 CNRS/LORIA and Universite´ de Lorraine, Nancy, France {emilie.colin,claire.gardent,laura.perez}@loria.fr 2 National Library of Medicine, Bethesda, USA yassine.m’[email protected] 3 School of Informatics, University of Edinburgh, UK [email protected]

1 Introduction ties with typed literal values.1 There are several motivations for generating text With the emergence of the linked data initiative and from DBPedia. the rapid development of RDF (Resource Descrip- First, the RDF language in which DBPedia is en- tion Format) datasets, several approaches have re- coded is widely used within the Linked Data frame- cently been proposed for generating text from RDF work. Many large scale datasets are encoded in this data (Sun and Mellish, 2006; Duma and Klein, 2013; language (e.g., MusicBrainz2, FOAF3, LinkedGeo- Bontcheva and Wilks, 2004; Cimiano et al., 2013; Data4) and official institutions5 increasingly publish Lebret et al., 2016). To support the evaluation and their data in this format. Being able to generate good comparison of such systems, we propose a shared quality text from RDF data would permit e.g., mak- task on generating text from DBPedia data. The ing this data more accessible to lay users, enriching training data will consist of Data/Text pairs where existing text with information drawn from knowl- the data is a set of triples extracted from DBPe- edge bases such as DBPedia or describing, compar- dia and the text is a verbalisation of these triples. ing and relating entities present in these knowledge In essence, the task consists in mapping data to bases. text. Specific subtasks include sentence segmenta- Second, RDF data, and in particular, DBPedia, tion (how to chunk the input data into sentences), provide a framework that is both limited and arbi- lexicalisation (of the DBPedia properties), aggrega- trarily extensible from a linguistic point of view. In tion (how to avoid repetitions) and surface realisa- the simplest case, the goal would be to verbalise tion (how to build a syntactically correct and natural a single triple. In that case, the task mainly con- sounding text). sists in finding an appropriate “lexicalisation” for the property. The complexity of the generation task 2 Context and Motivation can be closely monitored however by increasing the number of input triples, using input with different DBPedia is a multilingual knowledge base that was shapes6, working with different semantic domains built from various kinds of structured information and/or enriching the RDF graphs with additional contained in Wikipedia (Mendes et al., 2012). This data is stored as RDF triples of the form (SUBJECT, 1http://wiki.dbpedia.org/ PROPERTY, OBJECT) where the subject is a URI (Uni- dbpedia-dataset-version-2015-10 form Resource Identifier), the property is a binary 2https://musicbrainz.org/ relation and the object is either a URI or a literal 3http://www.foaf-project.org/ 4 value such as a string, a date or a number. The En- http://linkedgeodata.org/ 5 http://museum-api.pbworks.com glish version of the DBpedia knowledge base cur- See for exam- ples. rently encompasses 6.2M entities, 739 classes, 1,099 6DBPedia data forms a graph. Different graph shapes induce properties with reference values and 1,596 proper- different verbalisation structures. (e.g., discourse) information. We plan to produce propose but our task differs from them in various a dataset which varies along at least some of these ways. dimensions so as to provide a benchmark for gener- KBGen generation challenge. The recent KBGen ation that will tests system on input of various com- (Banik et al., 2013) task focused on sentence genera- plexity. tion from Knowledge Bases (KB). In particular, the Third, there has been much work recently on ap- task was organised around the AURA (Gunning et plying deep learning (in particular, sequence to se- al., 2010) KB on the biological domain which mod- quence) models to generation. The training data els n-ary relations. The input data selection process used by these approaches however often have lim- targets the extraction of KB fragments which could ited variability. For instance, (Wen et al., 2015)’s be verbalised as a single sentence. The content se- data is restricted to restaurant descriptions and (Le- lection approach was semi-automatic, starting with bret et al., 2016)’s to WikiData frames. Typically the the manual selection of a set of KB fragments. Then, number of attributes (property) considered by these using patterns derived from those fragments, a new approaches is very low (between 15 and 40) and set of candidate KB fragments was generated which the text to be produced have a stereotyped structure was finally manually revised. The verbalisation of (restaurant description, biographic abstracts). By the sentence sized KB fragments was generated by providing a more varied dataset, the WebNLG data- human subjects. text corpus will permit investigating how such deep Although our task also concerns text generation learning models perform on more varied and more from KBs the definition of the task is different. Our linguistically complex data. proposal aims at the generation of text beyond sen- tences and thus involves an additional subtask that 3 Task Description is sentence segmentation. The tasks also differ on In essence, the task consists in mapping data to the KBs used, we propose using DBPedia which fa- text. Specific subtasks include sentence segmenta- cilitates changing the domain by focusing on dif- tion (how to chunk the input data into sentences), ferent categories. Moreover, the set of relations on lexicalisation (of the DBPedia properties), aggrega- both KBs pose different challenges for generation, tion (how to avoid repetitions) and surface realisa- while the AURA KB contains n-ary relations DBPe- tion (how to build a syntactically correct and natu- dia contains relations names challenging for the lex- ral sounding text). The following example illustrates icalisation subtask. A last difference with our task is this. the content selection method. Our method is com- pletely automatic and thus permits the inexpensive (1) a. Data: (JOHN E BLAHA BIRTHDATE 1942 08 26) generation of a large benchmark. Moreover, it can (JOHN E BLAHA BIRTHPLACE SAN ANTONIO) (JOHN E BLAHA OCCUPATION FIGHTER PILOT) be used to select content ranging from a single triple b. Text: John E Blaha, born in San Antonio on 1942-08- to several triples and with different shapes. 26, worked as a fighter pilot The Surface Realisation Shared Task (SR’11). The Given the input shown in (1a), generating (1b) in- major goal of the SR’11 task (Belz et al., 2011) volves lexicalising the OCCUPATION property as the was to provide a common ground for the compari- phrase worked as, using PP coordination (born in San son of surface realisers on the task of regenerating Antonio on 1942-08-26) to avoid repeating the word born sentences in a treebank. Two different tracks are (aggregation) and verbalising the 3 triples by a sin- considered with different input representations. The gle complex sentence including an apposition, a PP ’shallow’ input provides a dependency tree of the coordination and a transitive verb construction (sen- sentence to be generated and the ’deep’ input pro- tence segmentation and surface realisation). vides a graph representation where syntactic depen- Relation to Previous Shared Tasks Other NLG dencies have been replaced by semantic roles and shared task evaluation challenges have been organ- some function words have been removed. ised in the past. These have focused on different The focus of the SR’11 task was on the linguis- generation subtasks overlapping with the task we tic realisation subtask and the broad coverage of lin- guistic phenomena. The task we propose here starts ILP program is used to extract from DBPedia, sub- from non-linguistic KB data and puts forward other trees that maximise bigram probability. In effect, NLG subtasks. the extracted DBPedia trees are coherent entity de- scriptions in that the property bigram they contain Generating Referring Expressions (GRE). The GRE often cooccur together in the DBPedia graphs as- shared tasks pioneered the proposed NLG chal- sociated with entities of a given DBPedia category. lenges. The first shared task has only focused on The method can be parameterised to produce con- the selection of distinguishing attributes (Belz and tent units for different DBPedia categories, differ- Gatt, 2007) while subsequent tasks have considered ent DBPedia entities and various numbers of DBPe- the referring expression realisation subtask propos- dia triples. It is fully automatic and permit produc- ing a complete referring expression generation task ing DBPedia graphs that are both coherent, diverse (Gatt et al., 2008; Gatt et al., 2009). This tasks and that bear on different domains (e.g., Astronauts, aimed at the unique identification of the referent and Universities, Musical work). brevity of the referring expression. Slightly differ- Text To associate the DBPedia trees extracted in ent, the GREC challenges (Belz et al., 2008; Belz et the first phase with text, we will combine automatic al., 2009; Belz et al., 2010) propose the generation techniques with crowdsourcing in two ways. of referring expressions in a discourse context. The First, we will lexicalise DBPedia properties by GREC tasks use a corpus created from Wikipedia using the lexicalisations contained in the Lemon abstracts on geographic entities and people and with English Lexicon for DBPedia7(Walter et al., 2013; two referring expression annotation schemes, refer- Walter et al., 2014a; Walter et al., 2014b) and ence type and word strings. Rather than generating by manually filtering the lexicalisations produced from data input these tasks consist in labelling un- by the lexicalisation method described in (Perez- derspecified referring expressions in a given text. Beltrachini and Gardent, 2016) and by the rela- Our task concerns the generation of entity de- tion extraction and clustering method described in scriptions and requires the production of referring (c.f. (Nakashole et al., 2012))8. We will then ask expressions, specially in the cases where multiple crowdsourcers to verbalise sets of DBPEdia triples sentences will be generated. However, it does not in which properties have already been lexicalised foresee the selection of additional content (e.g. at- (e.g., CREW1UP will be lexicalised as commander of ). tributes). In contrast, our proposal targets all gener- Second, we will exploit the data-to-text alignment ation subtasks involved in content realisation. method presented in (Mrabet et al., 2016) to semi- 4 Data automatically align Wikipedia text with sets of DB- Pedia triples. The method consists in (i) automati- As illustrated in Example 1 above, the training cor- cally annotating phrases with DBPedia entities, (ii) pus consists of (D,T ) pairs such that D is a set of associating sentences with DBPedia triples relating DBPedia triples and T is an English text (possibly entities annotating these sentences and (iii) using consisting of a single sentence). This corpus will crowdsourcing to align sentences with triples. In the be constructed in two steps by first, extracting from third step, annotators are asked to “align” triples and DBPedia content units that are both coherent and sentences that is, to remove from the sentence all diverse and second, associating these content units material that is irrelevant to express the associated with English text verbalising their content. triples and vice versa, to remove any triples that is not expressed by the sentence. Data To extract content units from DBPedia, we will use the content selection procedure sketched in Statistics, Schedule and Funding The WebNLG (Mohammed et al., 2016). This procedure consists shared task will be funded by the WebNLG ANR of two steps. First, bigram models of DBPedia prop- 7http://lemon-model.net/lexica/dbpedia_ erties specific to a given DBPedia category (e.g., As- en/ tronaut) are learned from the DBPedia graphs as- 8https://d5gate.ag5.mpi-sb.mpg.de/ sociated with entities of that category. Second, an pattyweb/ Project9. We aim to produce a data-text corpus of Anja Belz, Michael White, Dominic Espinosa, Eric Kow, medium size (between 10K and 50K data-text pairs) Deirdre Hogan, and Amanda Stent. 2011. The first bearing on at least 5 different domains and consist- surface realisation shared task: Overview and evalu- ing of input data containing between 2 and 5 RDF ation results. In Proceedings of the 13th European Workshop on Natural Language Generation, ENLG triples. Ideally, training data will be made available ’11, pages 217–226, Stroudsburg, PA, USA. Associ- early in 2017 and testing will be carried out in early ation for Computational Linguistics. summer (May-June 2017). Kalina Bontcheva and Yorick Wilks. 2004. Automatic report generation from ontologies: the miakt approach. 5 Evaluation In International Conference on Application of Natu- ral Language to Information Systems, pages 324–335. Evaluation of the generated texts will be done both Springer. with automatic evaluation metrics (BLEU, TER Philipp Cimiano, Janna Luker,¨ David Nagel, and or/and METEOR) and using human judgements ob- Christina Unger. 2013. Exploiting ontology lexica tained through crowdsourcing. The human evalu- for generating natural language texts from rdf data. In ation will seek to assess such criteria as fluency, Proceedings of the 14th European Workshop on Natu- grammaticality and appropriateness (does the text ral Language Generation, pages 10–19. correctly verbalise the input data?). Daniel Duma and Ewan Klein. 2013. Generating natural language from linked data: Unsupervised template ex- Acknowledgments traction. Association for Computational Linguistics, Potsdam, Germany, pages 83–94. We thank the French National Research Agency for Albert Gatt, Anja Belz, and Eric Kow. 2008. The tuna funding the research presented in this paper in the challenge 2008: Overview and evaluation results. In 10 Proceedings of the Fifth International Natural Lan- context of the WebNLG project . guage Generation Conference, pages 198–206. Asso- ciation for Computational Linguistics. Albert Gatt, Anja Belz, and Eric Kow. 2009. The tuna- References reg challenge 2009: Overview and evaluation results. Eva Banik, Claire Gardent, and Eric Kow. 2013. The In Proceedings of the 12th European Workshop on kbgen challenge. In the 14th European Workshop on Natural Language Generation, pages 174–182. Asso- Natural Language Generation (ENLG), pages 94–97. ciation for Computational Linguistics. Anja Belz and Albert Gatt. 2007. The attribute selection David Gunning, Vinay K. Chaudhri, Peter Clark, Ken for gre challenge: Overview and evaluation results. Barker, Shaw-Yi Chaw, Mark Greaves, Benjamin Proceedings of UCNLG+ MT: Language Generation Grosof, Alice Leung, David McDonald, Sunil Mishra, and Machine Translation, pages 75–83. John Pacheco, Bruce Porter, Aaron Spaulding, Dan Anja Belz, Eric Kow, Jette Viethen, and Albert Gatt. Tecuci, and Jing Tien. 2010. Project Halo Update – AI Magazine 2008. The grec challenge: Overview and evaluation Progress toward digital aristotle. , Fall. results. Remi´ Lebret, David Grangier, and Michael Auli. 2016. Generating text from structured data with application Anja Belz, Eric Kow, and Jette Viethen. 2009. The grec to the biography domain. CoRR, abs/1603.07771. named entity generation challenge 2009: overview and Pablo N Mendes, Max Jakob, and Christian Bizer. 2012. evaluation results. In Proceedings of the 2009 Work- Dbpedia: A multilingual cross-domain knowledge shop on Language Generation and Summarisation, base. In LREC, pages 1813–1817. Citeseer. pages 88–98. Association for Computational Linguis- Rania Mohammed, Laura Perez-Beltrachini, and Claire tics. Gardent. 2016. Category-driven content selection. In Anja Belz, Eric Kow, Jette Viethen, and Albert Gatt. Proceedings of the nintth International Natural Lan- 2010. Generating referring expressions in context: guage Generation Conference, INLG 2016. The grec task evaluation challenges. In Empirical Yassine Mrabet, Pavlos Vougiouklis, Halil Kilicoglu, methods in natural language generation, pages 294– Claire Gardent, Dina DemnerFushman, Jonathon 327. Springer. Hare, and Elena Simperl. 2016. Aligning texts and 9http://talc1.loria.fr/webnlg/stories/ knowledge bases with semantic sentence simplifica- about.html tion. In Proceedings of the 2nd International Work- 10http://talc1.loria.fr/webnlg/stories/ shop on Natural Language Generation and the Seman- about.html tic Web. Ndapandula Nakashole, Gerhard Weikum, and Fabian Suchanek. 2012. Discovering and exploring relations on the web. Proceedings of the VLDB Endowment, 5(12):1982–1985. Laura Perez-Beltrachini and Claire Gardent. 2016. Learning embeddings to lexicalise rdf properties. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics. Xiantang Sun and Chris Mellish. 2006. Domain inde- pendent sentence generation from rdf representations for the semantic web. In Combined Workshop on Language-Enabled Educational Technology and De- velopment and Evaluation of Robust Spoken Dialogue Systems, European Conference on AI, Riva del Garda, Italy. Sebastian Walter, Christina Unger, and Philipp Cimiano. 2013. A corpus-based approach for the induction of ontology lexica. In Natural Language Processing and Information Systems, pages 102–113. Springer. Sebastian Walter, Christina Unger, and Philipp Cimiano. 2014a. Atolla framework for the automatic induction of ontology lexica. Data & Knowledge Engineering, 94:148–162. Sebastian Walter, Christina Unger, and Philipp Cimiano. 2014b. M-atoll: a framework for the lexicalization of ontologies in multiple languages. In The Semantic Web–ISWC 2014, pages 472–486. Springer. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksiˇ c,´ Pei-Hao Su, David Vandyke, and Steve Young. 2015. Seman- tically conditioned lstm-based natural language gener- ation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721, Lisbon, Por- tugal, September. Association for Computational Lin- guistics.