Collective Content Selection for Concept-To-Text Generation Regina Barzilay Mirella Lapata Computer Science and Artificial Intelligence Laboratory School of Informatics Massachusetts Institute of Technology University of Edinburgh [email protected] [email protected] Abstract game summary. The content selection component aims to identify this subset.1 A content selection component deter- In existing generation systems the content se- mines which information should be con- lection component is manually crafted. Specify- veyed in the output of a natural language ing content selection rules is, however, notoriously generation system. We present an effi- difficult, prohibitively so in large domains. It in- cient method for automatically learning volves the analysis of a large number of texts from a content selection rules from a corpus and domain-relevant corpus, familiarity with the associ- its related database. Our modeling frame- ated database, and consultation with domain experts. work treats content selection as a col- Moreover, the task must be repeated for each domain lective classification problem, thus allow- anew. ing us to capture contextual dependen- This paper proposes a data-driven method for cies between input items. Experiments learning the content-selection component for a in a sports domain demonstrate that this concept-to-text generation system. We assume that approach achieves a substantial improve- the learning algorithm is provided with a parallel ment over context-agnostic methods. corpus of documents and a corresponding database, in which database entries that should appear in doc- uments are marked. 1 Introduction One possible approach is to formulate content se- Content selection is a fundamental task in concept- lection as a standard binary classification task: pre- to-text generation (Reiter and Dale, 2000). A practi- dict whether an item is to be included on the basis cal generation system typically operates over a large of its attributes alone. In fact, this method is com- database with multiple entries that could potentially monly used for content selection in text summariza- be included in a text. A content selection compo- tion (e.g., Kupiec et al., 1995). However, by treating nent determines what subset of this information to each instance in isolation, we cannot guarantee that include in the generated document. the selected database entries are related in a mean- For example, consider the task of automatically ingful way, which is essential for the generation of a generating game summaries, given a database con- coherent text. taining statistics on Americal football. Table 1 Rather than selecting each item separately, we shows an excerpt from such a database, and its cor- propose a method for collective content selection, responding game summary written by a journalist. where all candidates are considered simultaneously A single football game is typically documented in for selection. Collective selection thereby allows hundreds of database entries — all actions, player us to explicitly optimize coherence in the generated positions, and scores are recorded, along with a wide 1The organization of the selected information and its sur- range of comparative and aggregate statistics. Only face realization is typically handled by other components of the a small fraction of this information is featured in a generation system, which are outside the scope of this paper. 331 Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 331–338, Vancouver, October 2005. c 2005 Association for Computational Linguistics Suggs rushed for 82 yards and scored a Passing touchdown in the fourth quarter, leading PLAYER CP/AT YDS AVG TD INT the Browns to a 17-13 win over the Wash- Brunell 17/38 192 6.0 0 0 ington Redskins on Sunday. Jeff Garcia Garcia 14/21 195 9.3 1 0 went 14-of-21 for 195 yards and a TD for ... ... ... ... ... ... the Browns, who didn’t secure the win until Coles fumbled with 2:08 left. The Redskins Rushing (1-3) can pin their third straight loss on go- PLAYER REC YDS AVG LG TD ing just 1-for-11 on third downs, mental mis- Suggs 22 82 3.7 25 1 takes and a costly fumble by Clinton Por- ... ... ... ... ... ... tis. Brunell finished 17-of-38 for 192 yards, but was unable to get into any rhythm because Fumbles Cleveland’s defense shut down Portis. The PLAYER FUM LOST REC YDS Browns faked a field goal, but holder Der- Coles 1 1 0 0 rick Frost was stopped short of a first down. Portis 1 1 0 0 Brunell then completed a 13-yard pass to Davis 0 0 1 0 Coles, who fumbled as he was being taken Little 0 0 1 0 down and Browns safety Earl Little recov- ... ... ... ... ... ered. Table 1: Sample target game description and example of database entries; boldface indicates correspon- dences between the text and the database (CP/AT: completed out of attempted, YDS: yards, AVG: average, TD: touchdown, INT: interception, REC: received, LG: longest gain, FUM: fumble). text: semantically related entries are often selected F-score, when compared to a standard classification together. In essence, the algorithm seeks a subset approach, thus demonstrating the benefits of collec- of candidates that is consistent with the individual tive content selection on this complex domain. Fur- preferences of each candidate, and at the same time thermore, our results empirically confirm the contri- maximally satisfies contextual constraints. A graph- bution of discourse constraints for content selection. based formulation of this optimization problem al- In the following section, we provide an overview lows us to find an exact, globally optimal solution, of existing work on content selection. Then, we de- using a min-cut algorithm. fine the learning task and introduce our approach for Collective content selection is particularly ben- collective content selection. Next, we present our eficial to generation systems that operate over re- experimental framework and data. We conclude the lational databases. Rich structural information paper by presenting and discussing our results. available in a database can be readily utilized to determine semantic relatedness between different 2 Related Work database entries. For instance, we can easily find The generation literature provides multiple exam- all actions (e.g., touchdowns and fumbles) associ- ples of content selection components developed for ated with a specific player in a game, which could be various domains (Kukich, 1983; McKeown, 1985; relevant for generating a summary centered around Sripada et al., 2001; Reiter and Dale, 2000). A com- an individual. We show how to utilize database re- mon theme across different approaches is the em- lations for discovering meaningful contextual links phasis on coherence: related information is selected between database entries. “to produce a text that hangs together” (McKeown, We evaluate our collective content selection 1985). Similarly, our method is also guided by co- model in a sports domain. The proposed content herence constraints. In our case these constraints are selection component operates over a large database derived automatically, while in symbolic generation containing descriptive statistics about American systems coherence is enforced by analyzing a large football games. Our model yields a 10% increase in number of texts from a domain-relevant corpus and 332 careful hand-crafting of content selection rules. The goal of the content selection component is Duboue and McKeown (2003) were the first to to select entries from a database, i.e., to determine propose a method for learning content selection whether their label values are 0 or 1. Under this for- rules automatically, thus going beyond mere corpus mulation, content selection is restricted to informa- analysis. They treat content selection as a classifi- tion available in the database; there is no attempt to cation task. Given a collection of texts associated induce new facts through inference. with a domain-specific database, their model learns In the next section, we describe our learning whether a database entry should be selected for pre- framework, and explain how it is applied to the con- sentation or not. Their modeling approach uses an tent selection task. expressive feature space while considering database entries in isolation. 3.1 The Collective Classification Approach Similarly to Duboue and McKeown (2003), we Generation of a coherent text crucially depends on view content selection as a classification task and our ability to select entities that are related in a learn selection rules from a database and its corre- meaningful way (McKeown, 1985). A content se- sponding corpus. In contrast to them, we consider lection component that considers every entity in iso- all database entries simultaneously, seeking a glob- lation does not have any means to enforce this im- ally optimal selection. Thus, we avoid the need for portant discourse constraint. We therefore formulate extensive feature engineering by incorporating dis- content selection as a collective classification task, course constraints into the learning framework. In where all entities that belong to the same database addition, we assess whether data-driven methods for (i.e., the same football game) are considered simul- content selection scale up to large databases with taneously. This framework thus enables us to en- thousands of interrelated entries, by evaluating our force contextual constraints by selecting related en- model in a sports domain. Previous work (Duboue tities. and McKeown, 2003) has tackled the content selec- When considered in isolation, some database en- tion problem for biographical summaries, a simpler tries are more likely to be selected than others. In domain with fewer entities and interactions among the American football domain, for example, entries them. of type Rushing are often extracted if they yield a touchdown.3 Other Rushing entries (e.g., which do 3 The Task not deliver scoring points) are typically omitted. In general, the attributes of an entry can provide use- We assume that the content selection component ful cues for predicting whether it should be selected.