Learning to Select, Track, and Generate for Data-to-Text Hayate Iso† Yui Uehara♠ Tatsuya Ishigaki♠♦ Hiroshi Noji♠ Eiji Aramaki†♠ Ichiro Kobayashi♠❤ Yusuke Miyao♠♣ Naoaki Okazaki♠♦ Hiroya Takamura♠♦ †NAIST ♠Artificial Intelligence Research Center, AIST ♦TITech ❤Ochanomizu Univ. ♣UTokyo

TEAM H/V WIN LOSS PTS REB AST FG PCT FG3 PCT ... The defeated the , 105-104, at on Wednesday. The Summary KNICKS H 16 19 104 46 26 45 46 ... Knicks (16-19) checked in to Wednesday’s contest looking BUCKS V 18 16 105 42 20 47 32 ... to snap a five-game losing streak and heading into the fourth quarter, they looked like they were well on their way to that goal. ... Antetokounmpo led the Bucks with 27 points, 13 ✓ Tracking entity states while summarizing table information PLAYER H/V PTS REB AST BLK STL MIN CITY ... rebounds, four assists, a and three blocks, his second consecutive double-double. actually checked H 0 37 NEW YORK ... 30 11 7 2 in as the second-leading scorer and did so in his customary H 15 3 4 0133NEW YORK ... improves adequacy while preserving fluency RG benchCS role, posting 18 points,CO along with nine boards, four COURTNEY LEE H 11 2 3 1138NEW YORK ... Method assists, three steals and a . JabariBLEU Parker contributed GIANNIS ANTETOKOUNMPO V 27 13 4 3 139MILWAUKEE ... # P% P%15 R%points, four F1%rebounds,DLD%three assists and a steal. Malcolm ✓ Incorporating the writer information is also helpful for GREG MONROE V 18 9 4 1 3 31 MILWAUKEE ... Brogdon went for 12 points, eight assists and six rebounds. JABARI PARKER V 15 4 3 0137MILWAUKEE ... Mirza Teletovic was productive in a reserve role as well, MALCOLM BROGDONPuduppullyV 12 et6 al. (820190038M) 33.06 83.17ILWAUKEE33.06... 43.59 37.60 16.97 13.96 selecting appropriate data and writing better summary generating 13 points and a . ... Courtney Lee MIRZA TELETOVIC V 13 100021MILWAUKEE ... + w in stage 1 28.43 84.75 45.00checked 49.73 in with 47.2511 points,22.16three assists,18.18two rebounds, a JOHN HENSON V220 0 014MILWAUKEE ... steal and a block. ... The Bucks and Knicks face off once ...... + w in stage 2 35.06 80.51 31.10again 45.28 in the 36.87 second game16.38 of the home-and-home17.81 series, with + w in stage 1 & 2 28.00 82.27 44.37the 48.71 meeting 46.44 taking place Friday night in Milwaukee. (a) Box score: Top contingency table shows of wins and 22.41 18.90 losses and summary of each game. Bottom table shows statistics Data-to-Text Generation PROPOSED(a) Box-score 39.05 94.38 35.77(b) 52.05 NBA basketball 42.40(b) game 19.38Gold summary:summary16.15 Each summary of each player such as points scored (PLAYER’s P TS), and total rebounds (PLAYER+ w’s R EB). 30.25 92.00 50.75consists 59.03 of game 54.58 victory25.75 or defeat20.84 of the game and highlights of valuable players. Task: Given a set of records, the system should produce a fluent Table 4: Effects of writer information. w indicates that WRITER embeddings are used. Numbers in bold are the Table 1: Example of input and output data: task defines box score (1a) used for input and summary document of largest among the variants ofGenerated each method. summary and an adequate summary. game (1b) used as output. Extracted entities are shown in bold face. Extracted values are shown in green.

The Milwaukee Bucks defeated the New York Knicks, 105-104, at Problem: t The Milwaukee199 Bucks 200defeated the New 201 York Knicks 202, 105- 203Madison 204 Square Garden 205 on Saturday. 206 The Bucks 207(18-16 208) checked in 209 104, at Madison Square Garden on Wednesday evening. The Y Jabari Parker contributed 15 pointsto Saturday’s , contest four with a well, rebounds outscoring ,the Knicks three(16-19) by assists ‣ Template system could generate the adequate text but not fluent t Bucks (18-16) have been one of the hottest teams in the league, a margin of 39-19 in the first quarter. However, New York by just a having won five of their last six games, and they have now won Z 1101001001025-foot lead at the end of the first quarter, the Bucks were able to pull t six of their last eight games. The Knicks (16-19) have now ‣ JABARI JABARI JABARI away, as they outscoredJABARI the Knicks by a 59-46 marginJABARI into the second. NN based system could generate the fluent text but not adequate Et won six of their last six games, as they continue- to battle for the -- -- - PARKER PARKER PARKER 45 points in theP thirdARKER quarter to seal the win for NewP YorkARKER with the eighth and final playoff spot in the Eastern Conference. Giannis At FIRST NAME LAST NAME -PLAYER PTS --Prest of the startersLAYER to sealREB the win.--P The Knicks wereLAYER led by AGiannisST - Antetokounmpo led the way for Milwaukee, as he tallied 27 , who tallied a game-high points, to go along with Nt ---0--1--1-Antetokounmpo 27 Adequacy Fluency points, 13 rebounds, four assists, three blocked shots and one 13 rebounds, four assists, three blocks and a steal. The game was steal, in 39 minutes . Jabari Parker added 15 points, four re- a crucial night for the Bucks’ starting five, as the duo was the most Table 2:bounds, Runningthree assists, exampleone steal of and ourone model’sblock, and 6 generation-of-8 from process.effective shooters, At every as they time posted step Milwaukeet, model to go onpredicts a pair of eachlow random Template ✔ × long range. John Henson added two points, two rebounds, one low-wise (Carmelo Anthony) and Malcolm Brogdon. Anthony added variable., Modelthree steals firstly and one determinesblock. John whetherHenson was to the refer only to data11 rebounds, recordsseven (Ztassists=1 and) ortwo notsteals (Z tot his=0 team-high). If random scoring total. variable other player to score in double digits for the Knicks, with 15 Jabari Parker was right behind him with 15 points, four rebounds, Zt =1, model selects entity Et, its attribute At and binary variables Nt if needed. For example, at t = 202, model NN × ✔ points, four assists, three rebounds and one steal, in 33 min- three assists and a block. Greg Monroe was next with a bench-leading predictsutes. random The Bucks variable were led byZDerrick202 =1 Roseand, who then tallied selects15 the18 entitypoints,J alongABARI with ninePARKERrebounds, andfour assists its attribute and three steals.PLAYER PTS. points, assists, rebounds and steal in minutes. four three one 1533 Brogdon posted 12 points, eight assists, six rebounds and a steal. Ours ✔ ✔ GivenWilly these Hernangomez values, modelstarted outputsin place of tokenPorzingis andfrom finished selectedDerrick data Rose record.and Courtney Lee were next with a pair of 11 / 11 with points, assists, rebounds and steal in { } 15 four three one 33 -point efforts. Rose also supplied four assists and three rebounds, while minutes. started in place of Jose Calderon Willy Hernangomez Lee complemented his scoring with three assists, a rebound and a steal. ( knee ) and responded with rebound and block. The ENT ‣ For document-level generation, the system has an influence of the our training, development,one test datasetone are respec- Johnthat Henson haveand beenMirza Teletovic referredwere to. nexth with a pairis also of two used/ two to up- Knicks were led by their starting backcourt of Carmelo An- { } -point efforts. LM also registered points, and he added a re- tivelythony 2,714,and 534,Carmelo and Anthony 500., but On combined average, for just each13 points sum- date hTeletovic, meaning that13 the referred data records writer’s bias. on 5-of-16 shooting. The Bucks next head to Philadelphia to bound and an assist. Jason Terry supplied eight points, three rebounds mary hastake on 384 the Sixers tokens on Friday and night, 644 while data the Knicks records. remain Each andaffect a pair of steals.the text The Cavs generation. remain in last place in the Eastern home to face the on Wednesday. Conference’s Atlantic Division. They now head home to face the match has only one summary in our dataset, as TorontoOur Raptors model on Saturday decides night. whether to refer to x, which far as we checked. We also collected the writer data record r x to be mentioned, and how to ex- Entity Tracking 2 of each document.(a)(a) PuduppullyPuduppully Our dataset et al. contains ((2019)2019) 32 differ- press a number.(b)(b) The Our Our selected model data record is used to ENT entTable writers.✔ Fluent 5: Example Thetext most summaries prolific generated writer in with ourPuduppully dataset✔ Fluent etupdate al. ( text2019h)’s model. Formally, (left) and we our use model the four (right). variables: Names Idea: Equipping an NN-based generation system with the wrotein ×bold 607Many face documents. incorrectare salient realizations There entities. (orange-colored areBlue also numbers writers) whoare correct✔ Correct relations realization derived & appropriate from input content data records selection but are not wroteobserved× lessMany in than improper reference ten content documents. summary. selectionsOrange On (blue-colored average, numbers each) are× incorrectDifficult1. Zt: to binaryrelations. capture variable complexGreen relation thatnumbers determines (underlinedare correct) whether relations the entity-tracking module enables capturing of the saliency and writermentioned wrote in 117 reference documents. summary. We call our new model refers to input x at time step t (Zt =1). the coherence while preserving the fluency. dataset ROTOWIRE-MODIFIED.2 2. Et: At each time step t, this variable indi- Experimentalcates theresults salient entity (e.g., HAWKS,LEBRON Method: Inspired by EntityNLM [1], our model dynamically porating writer information w. As discussed in generated highest quality summaries that scored 4Section Saliency-Aware4.5, w is supposed Text Generation to affect both content 20.84JAMES points). of BLEU score. updates each entity state and the tracking module. Baselineplanning and: surface realization. Our experimen- 3. At: At each time step t, this variable indicates At the core of our model is a neural language the salient attribute to be mentioned (e.g., PTS). When to ‣talTemplate: result is consistent game results, with the 6 prominent discussion. players, next game information LM N A E refer to the data model‣ Wiseman with a et memory al. [2]: Encoder-Decoder state h to generate w/ Attention a 4.Acknowledgments t: If attribute t of the salient entity t is summary‣ y1:T =(y1,...,yT ) given a set of data a numeric attribute, this variable determines if Choose whether to refer to the data Select the salient entity: 7Puduppully Conclusion et al. [3]: Two stage model records x. Our model has another memory state Wea valuewould in like the to data thank records the anonymous should be reviewers output in ENT1.Predicting the sequence of data records LM J. Parker h , which is used to remember the data records Arabic numerals (e.g., 50) or in English words MLP sigmoid In2. thisGenerating research, a wesummary proposed conditioned a new data-to-text on the predictedfor their sequence helpful suggestions. This paper is based Ent LM BiLinear C. Anthony (e.g., five). model2 that produces a summary text while track- on results obtained from a project commissioned D. Rose For information about the dataset, please follow thising the link: salienthttps://github.com/aistairc/ information that imitates a human- byTo the keep New track Energy of and the Industrial salient entity, Technology our model De- … Evaluationrotowire-modifiedwriting process. Metrics As a result,: our model outper- predictsvelopment these Organization random variables (NEDO), at JSTeach PRESTO time step Ent J. Parker ‣formedRG: the the ratio existing of the models correct in all relations evaluation out mea- of all the(Grant extracted Number relations. JPMJPR1655), and AIST-Tokyo If ≠ ‣sures.CS: the We F1 also score explored of the therelations effects extracted of incorpo- from Techthe generated Real World summary Big-Data against Computation those Open from In- Update the entity tracker: ratingthe reference writer information summary to data-to-text models. novation Laboratory (RWBC-OIL). Else ‣WithCO: writer the normalized information, Damerau-Levenshtein our model successfully Distance (DLD) between the sequences of Else Ent relations extracted from the generated and reference summary. J. Parker ‣ BLEU

Select the attribute given the entity J. Parker Compose ht’ PTS LM ht’ LM MLP tanh BiLinear REB Ent Ent AST …

Else When not to refer to the data Update the entity tracker and overwrite the selected entity states: Predict the next word, yt: The defeated the , TEAM H/V WIN LOSS PTS REB AST FG PCT FG3 PCT ... Else MilwaukeeEnt Bucks New York Knicks 105-104, at Madison Square Garden on Wednesday. The ht’ Writer Bias Evaluation: KNICKS H 16 19 104 46 26 45 46 ... Knicks (16REB-19) of checked J. Parker in to Wednesday’s contest looking BUCKS V 18 16 105Linear42 20softmax 47 32 ... to snap a five-game losing streak and heading into the fourth ‣ For Puduppully et al., concatenate the writer embedding w for decoder’s input. quarter, they looked like theyOverwrite were well on their way to that goal.J.... ParkerAntetokounmpo led the Bucks with 27 points, 13 ‣ For our model, concatenate w for hidden state of LM and Tracking module states for PLAYER H/V PTS REB AST BLK STL MIN CITY ... rebounds, four assists, a steal and three blocks, his second consecutive double-double. Greg Monroe actually checked composing ht’. CARMELO ANTHONY H 0 37 NEW YORK ... 30 11 7 2 in as the second-leading scorerIf selected and did so data in his record customary DERRICK ROSE H 15 3 4 0133NEW YORK ... bench role, posting 18 points,is along number with nine boards, four COURTNEY LEE H 11 2 3 1138NEW YORK ... assists, three steals and a block. Jabari Parker contributed GIANNIS ANTETOKOUNMPO V 27 13 4 3 139MILWAUKEE ... 15 points, four rebounds, three assists and a steal. Malcolm GREG MONROE V 18 9 4 1 3 31 MILWAUKEE ... ChooseBrogdon howwent forto12 representpoints, eight assists the anddatasix rebounds. JABARI PARKER V 15 4 3 0137MILWAUKEE ... Mirza Teletovic was productive in a reserve role as well, MALCOLM BROGDON V 12 6 8 0038MILWAUKEE ...record (e.g. 4 or four) Update the language model states: generating 13 points and a rebound. ... Courtney Lee MIRZA TELETOVIC V 13 100021MILWAUKEE ... checked in with 11 points, three assists, two rebounds, a JOHN HENSON LM V220 0 014MILWAUKEE ... LM steal and a block. ... The Bucks and Knicks face off once ...... MLP Sigmoid again inEnt the second game of the home-and-home series, with yt ht’ the meeting taking place Friday night in Milwaukee. (a) Box score: Top contingency table shows number of wins and losses and summary of each game. Bottom table shows statistics (b) NBA basketball game summary: Each summary Annotation:of each player such asTo points train scored the (PLAYER model’s P TS), and in total a fully supervised way, rebounds (PLAYER’s R EB). consists of game victory or defeat of the game and we obtain the following annotations highlightsusing of the valuable information players. Table 1: Example of input and output data: task defines box score (1a) used for input and summary document of extractiongame (1b) used system as output. Extracted developed entities are shown by in Wisemanbold face. Extracted et valuesal. [2]. are shown in green.

t 199 200 201 202 203 204 205 206 207 208 209 References Yt Jabari Parker contributed 15 points , four rebounds , three assists

Zt 11010010010 1. Ji et al. - Dynamic Entity Representations in Neural Language Models, In EMNLP 2017 JABARI JABARI JABARI JABARI JABARI Et ------PARKER PARKER PARKER PARKER PARKER 2. Wiseman et al. - Challenges in Data-to-Document Generation, In EMNLP 2017 At FIRST NAME LAST NAME -PLAYER PTS --PLAYER REB --PLAYER AST - 3. Puduppully et al. - Data-to-Text Generation with Content Selection and Planning, In AAAI 2019 Nt ---0--1--1-

Table 2: Running example of our model’s generation process. At every time step t, model predicts each random variable. Model firstly determines whether to refer to data records (Zt =1) or not (Zt =0). If random variable Zt =1, model selects entity Et, its attribute At and binary variables Nt if needed. For example, at t = 202, model predicts random variable Z202 =1and then selects the entity JABARI PARKER and its attribute PLAYER PTS. Given these values, model outputs token 15 from selected data record.

our training, development, test dataset are respec- that have been referred to. hENT is also used to up- tively 2,714, 534, and 500. On average, each sum- date hLM, meaning that the referred data records mary has 384 tokens and 644 data records. Each affect the text generation. match has only one summary in our dataset, as Our model decides whether to refer to x, which far as we checked. We also collected the writer data record r x to be mentioned, and how to ex- 2 of each document. Our dataset contains 32 differ- press a number. The selected data record is used to ent writers. The most prolific writer in our dataset update hENT. Formally, we use the four variables: wrote 607 documents. There are also writers who wrote less than ten documents. On average, each 1. Zt: binary variable that determines whether the writer wrote 117 documents. We call our new model refers to input x at time step t (Zt =1). dataset ROTOWIRE-MODIFIED.2 2. Et: At each time step t, this variable indi- cates the salient entity (e.g., HAWKS,LEBRON 4 Saliency-Aware Text Generation JAMES). 3. At: At each time step t, this variable indicates At the core of our model is a neural language the salient attribute to be mentioned (e.g., PTS). LM model with a memory state h to generate a 4. Nt: If attribute At of the salient entity Et is summary y1:T =(y1,...,yT ) given a set of data a numeric attribute, this variable determines if records x. Our model has another memory state a value in the data records should be output in hENT, which is used to remember the data records Arabic numerals (e.g., 50) or in English words (e.g., five). 2For information about the dataset, please follow this link: https://github.com/aistairc/ To keep track of the salient entity, our model rotowire-modified predicts these random variables at each time step