![Arxiv:1906.01081V1 [Cs.CL] 3 Jun 2019 (Gardent Et Al., 2017), and E2E-NLG (Dušek Et Al., 1 Introduction 2019))](http://data.docslib.org/img/8f462d3f8f50ef1908c4ebb031c8c772-1.webp)
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Bhuwan Dhingra†∗ Manaal Faruqui‡ Ankur Parikh‡ Ming-Wei Chang‡ Dipanjan Das‡ William W. Cohen†‡ † Carnegie Mellon University ‡ Google Research [email protected] {mfaruqui,aparikh,mingweichang,dipanjand,wcohen}@google.com
Abstract repeatedly anytime small modifications are made to a system. Hence, automatic metrics that com- Automatically constructed datasets for gener- pare the generated text to one or more reference ating text from semi-structured data (tables), texts are routinely used to compare models (Ban- such as WikiBio (Lebret et al., 2016), of- ten contain reference texts that diverge from galore et al., 2000). For table-to-text generation, the information in the corresponding semi- automatic evaluation has largely relied on BLEU structured data. We show that metrics which (Papineni et al., 2002) and ROUGE (Lin, 2004). rely solely on the reference texts, such as The underlying assumption behind these metrics BLEU and ROUGE, show poor correlation is that the reference text is gold-standard, i.e., it with human judgments when those references is the ideal target text that a system should gener- diverge. We propose a new metric, PAR- ate. In practice, however, when datasets are col- ENT, which aligns n-grams from the refer- ence and generated texts to the semi-structured lected automatically and heuristically, the refer- data before computing their precision and re- ence texts are often not ideal. Figure1 shows an call. Through a large scale human evalua- example from the WikiBio dataset (Lebret et al., tion study of table-to-text models for WikiBio, 2016). Here the reference contains extra informa- we show that PARENT correlates with human tion which no system can be expected to produce judgments better than existing text generation given only the associated table. We call such ref- metrics. We also adapt and evaluate the infor- erence texts divergent from the table. mation extraction based evaluation proposed in Wiseman et al.(2017), and show that PAR- We show that existing automatic metrics, in- ENT has comparable correlation to it, while cluding BLEU, correlate poorly with human judg- being easier to use. We show that PARENT ments when the evaluation sets contain divergent is also applicable when the reference texts are references (§5.4). For many table-to-text genera- elicited from humans using the data from the tion tasks, the tables themselves are in a pseudo- 1 WebNLG challenge. natural language format (e.g., WikiBio, WebNLG arXiv:1906.01081v1 [cs.CL] 3 Jun 2019 (Gardent et al., 2017), and E2E-NLG (Dušek et al., 1 Introduction 2019)). In such cases we propose to compare the The task of generating natural language descrip- generated text to the underlying table as well to tions of structured data (such as tables) (Kukich, improve evaluation. We develop a new metric, 1983; McKeown, 1985; Reiter and Dale, 1997) PARENT (Precision And Recall of Entailed N- has seen a growth in interest with the rise of se- grams from the Table) (§3). When computing pre- quence to sequence models that provide an easy cision, PARENT effectively uses a union of the way of encoding tables and generating text from reference and the table, to reward correct informa- them (Lebret et al., 2016; Wiseman et al., 2017; tion missing from the reference. When computing Novikova et al., 2017b; Gardent et al., 2017). recall, it uses an intersection of the reference and For text generation tasks, the only gold standard the table, to ignore extra incorrect information in metric is to show the output to humans for judg- the reference. The union and intersection are com- ing its quality, but this is too expensive to apply puted with the help of an entailment model to de- cide if a text n-gram is entailed by the table.2 We ∗Work done during an internship at Google. 1Code and Data: http://www.cs.cmu.edu/ ~bdhingra/pages/parent.html 2Here “entailed” means can be reasonably inferred from Figure 1: A table from the WikiBio dataset (right), its reference description and three hypothetical generated texts with scores assigned to them by automatic evaluation metrics. Text which cannot be inferred from the table is in red, and text which can be inferred but isn’t present in the reference is in green. PARENT is our proposed metric. show that this method is more effective than us- and Lapata, 2005; Liang et al., 2009). Tables can K ing the table as an additional reference. Our main be expressed as set of records T = rk k=1, contributions are: where each record is a tuple (entity, attribute,{ } value). When all the records are about the same We conduct a large-scale human evaluation of • entity, we can truncate the records to (attribute, the outputs from 16 table-to-text models on value) pairs. For example, for the table in Figure1, 1100 examples from the WikiBio dataset, many the records are {(Birth Name, Michael Dahlquist), of which have divergent references (§5.2). (Born, December 22 1965), ...}. The task is to gen- We propose a new metric, PARENT (§3), and erate a text G which summarizes the records in a • show that it improves correlation with human fluent and grammatical manner.3 For training and judgments over existing metrics, both when evaluation we further assume that we have a ref- comparing similar systems (such as different erence description R available for each table. We i i i N hyperparameters of a neural network) and when let M = (T ,R ,G ) i=1 denote an evaluation comparing vastly different systems (such as setD of tables,{ references} and texts generated from i i template-based and neural models). a model M, and Rn, Gn denote the collection of i i We also develop information extraction based n-grams of order n in R and G , respectively. We • use # i (g) to denote the count of n-gram g in metrics, inspired from Wiseman et al.(2017), Rn i R , and # i i (g) to denote the minimum of its by training a model to extract tables from the n Gn,Rn i i reference texts (§4). We find that these metrics counts in Rn and Gn. Our goal is to assign a score have comparable correlation to PARENT, with to the model, which correlates highly with human the latter being easier to use out of the box. judgments of the quality of that model. We analyze the sensitivity of the metrics to di- • Divergent References. In this paper we are in- vergence by collecting labels for which refer- terested in the case where reference texts diverge ences contain only information also present in from the tables. In Figure1, the reference, though the tables. We show that PARENT maintains technically correct and fluent, mentions informa- high correlation as the number of such exam- tion which cannot be gleaned from the associ- ples is varied. (§5.5). ated table. It also fails to mention useful infor- We also demonstrate the applicability of PAR- mation which a generation system might correctly • ENT on the data released as part of the include (e.g. candidate 3 in the figure). We call WebNLG challenge (Gardent et al., 2017), such references divergent from the associated ta- where the references are elicited from humans, ble. This phenomenon is quite common – in Wik- and hence are of high quality (§5.4). iBio we found that 62% of the references men- tion extra information (§5.5). Divergence is com- 2 Table-to-Text Generation mon in human-curated translation datasets as well We briefly review the task of generating natu- (Carpuat et al., 2017; Vyas et al., 2018). ral language descriptions of semi-structured data, How does divergence affect automatic evalua- which we refer to as tables henceforth (Barzilay 3In some cases the system is expected to summarize all the corresponding table. In practice, we use simple lexical the records (e.g. WebNLG); in others the system is expected entailment models to determine this. to only summarize the salient records (e.g. WikiBio). tion? As a motivating example, consider the three tailed by each lexical item v in the table: candidate generations shown in Figure1. Clearly, i candidate 1 is the worst since it “hallucinates” P r(gj T ) = max P r(gj v). (1) ⇐ v∈T¯i ⇐ false information, and candidate 3 is the best since it is correct and mentions more information than P r(gj v) is estimated using co-occurrence candidate 2. However, BLEU and ROUGE, which counts⇐ from a training set of table-reference only compare the candidates to the reference, pe- pairs. Then the overall probability of the n- nalize candidate 3 for both excluding the divergent gram being entailed is taken as the geometric information in the reference (in red) and includ- 1/n Qn i 5 4 average w(g) = P r(gj T ) . ing correct information from the table (in green). j=1 ⇐ PARENT, which compares to both the table and reference, correctly ranks the three candidates. We note that these models are not sensitive to paraphrases between the table and text. For tasks 3 PARENT where this is important, embedding-based similar- ities may be used, but those are beyond the scope i i i PARENT evaluates each instance (T ,R ,G ) of this paper. Next we discuss how to compute the separately, by computing the precision and recall precision and recall of the generation. of Gi against both T i and Ri. Entailed Precision. When computing precision, Entailment Probability. The table is in a semi- we want to check what fraction of the n-grams in structured form, and hence not directly compa- i Gn are correct. We consider an n-gram g to be rable to the unstructured generated or reference i 6 correct either if it occurs in the reference Rn , or texts. To bridge this gap, we introduce the notion if it has a high probability of being entailed by the of entailment probability, which we define as the i table (i.e. w(g) is high). Let P r(g Rn) = probability that the presence of an n-gram g in a # i i (g) ∈ Gn,Rn denote the probability that an n-gram # i (g) text is “correct” given the associated table. We de- Gn i i i note this probability as w(g) = P r(g T ). Es- in Gn also appears in Rn. Then, the entailed pre- ⇐ n timating this probability is in itself a challenging cision Ep for n-grams of order n is given by: language understanding task, since the informa- n tion in the table may be expressed in varied forms Ep = in text. Here, we describe two simple models of P i i i P r(g R ) + P r(g / R )w(g) # i (g) g∈Gn n n Gn lexical entailment, inspired by work on the Recog- ∈ P ∈ , i #Gi (g) nizing Textual Entailment Challenge (Dagan et al., g∈Gn n P i # i (g)w(g) + # i i (g)[1 w(g)] 2006). We found these simple models to be effec- g∈Gn Gn Gn,Rn = P − . tive; while more sophisticated models may be used i # i (g) g∈Gn Gn if there are complex inferences between the table (2) and text, they are beyond the scope of this paper. In words, an n-gram receives a reward of 1 if it 1. Word Overlap Model: Let T¯i denote all the appears in the reference, with probability P r(g i i ∈ lexical items present in the table T , including Rn), and otherwise it receives a reward of w(g). both attribute names and their values. Then, Both numerator and denominator are weighted by Pn 1 ¯i i i w(g) = j=1 (gj T )/n, where n is the the count of the n-gram in G . P r(g R ) re- ∈ n ∈ n length of g, and gj is the jth token in g. wards an n-gram for appearing as many times as it appears in the reference, not more. We combine 2. Co-occurrence Model: (Glickman and Dagan, precisions for n-gram orders 1-4 using a geometric 2005) Originally proposed for the RTE task, this model computes the probability of a term 5Glickman and Dagan(2005) used a product instead of geometric mean. Here we use a geometric mean to ensure gj in the n-gram being entailed by the table as that n-grams of different lengths have comparable probabili- the maximum of its probabilities of being en- ties of being entailed. 6It is unlikely that an automated system produces the same 4BLEU is usually computed at the corpus-level, however extra n-gram as present in the reference, thus a match with here we show its value for a single sentence purely for illus- the reference n-gram is considered positive. For example, in tration purposes. The remaining BLEU scores in this paper Figure1, it is highly unlikely that a system would produce are all at the corpus-level. “Silkworm” when it is not present in the table. average, similar to BLEU: model M is the average of instance level PARENT scores across the evaluation set: ! 4 1 X n N Ep = exp log Ep (3) 4 1 X i i i n=1 P ARENT (M) = P ARENT (G ,R ,T ) N i=1 Entailed Recall. We compute recall against (7) i both the reference (Er(R )), to ensure proper sen- tence structure in the generated text, and the table Smoothing & Multiple References. The dan- i ger with geometric averages is that if any of the (Er(T )), to ensure that texts which mention more information from the table get higher scores (e.g. components being averaged become 0, the average candidate 3 in Figure1). These are combined us- will also be 0. Hence, we adopt a smoothing tech- nique from Chen and Cherry(2014) that assigns ing a geometric average: n n i a small positive value to any of Ep , Er (R ) and i (1−λ) i λ E (T i) 0 Er = Er(R ) Er(T ) (4) r which are . When multiple references are available for a table, we compute PARENT against The parameter λ trades-off how much the gener- each reference and take the maximum as its overall ated text should match the reference, versus how score, similar to METEOR (Denkowski and Lavie, much it should cover information from the table. 2014). The geometric average, which acts as an AND op- Choosing λ and . To set the value of λ we can eration, ensures that the overall recall is high only tune it to maximize the correlation of the metric when both the components are high. We found with human judgments, when such data is avail- this necessary to assign low scores to bad systems able. When such data is not available, we can use which, for example, copy values from the table the recall of the reference against the table, using without phrasing them in natural language. Eq.6, as the value of 1 λ. The intuition here is When computing E (Ri), divergent references r that if the recall of the reference− against the table will have n-grams with low w(g). We want to ex- is high, it already covers most of the information, clude these from the computation of recall, and and we can assign it a high weight in Eq.4. This hence their contributions are weighted by w(g): leads to a separate value of λ automatically set for P 7 −5 i # i i (g)w(g) each instance. is set to 10 for all experiments. n i g∈Rn Gn,Rn Er (R ) = P . (5) i # i (g)w(g) g∈Rn Rn 4 Evaluation via Information Extraction Similar to precision, we combine recalls for n = Wiseman et al.(2017) proposed to use an auxil- i 1-4 using a geometric average to get Er(R ). iary model, trained to extract structured records i For computing Er(T ), note that a table is a from text, for evaluation. However, the extrac- i K set of records T = rk . For a record rk, tion model presented in that work is limited to the { }k=1 let r¯k denote its string value (such as “Michael closed-domain setting of basketball game tables Dahlquist” or “December 22 1965”). Then: and summaries. In particular, they assume that each table has exactly the same set of attributes for K i 1 X 1 i each entity, and that the entities can be identified Er(T ) = LCS(¯rk,G ), (6) K r¯k in the text via string matching. These assumptions k=1 | | are not valid for the open-domain WikiBio dataset, where r¯k denotes the number of tokens in the value and hence we train our own extraction model to string, and LCS(x, y) is the length of the longest replicate their evaluation scheme. common subsequence between x and y. The LCS Our extraction system is a pointer-generator function, borrowed from ROUGE, ensures that en- network (See et al., 2017), which learns to produce 8 tity names in r¯k appear in the same order in the text a linearized version of the table from the text. The i as the table. Higher values of Er(T ) denote that network learns which attributes need to be popu- more records are likely to be mentioned in Gi. lated in the output table, along with their values. The entailed precision and recall are combined 7For WikiBio, on average λ = 0.6 using this heuristic. into an F-score to give the PARENT metric for one 8 All (attribute, value) pairs are merged into 1 long string instance. The system-level PARENT score for a using special separator tokens between them. It is trained on the training set of WikiBio. At test Beam Length Beam Human Name time we parsed the output strings into a set of (at- Size Penalty Rescoring Eval tribute, value) tuples and compare it to the ground References – – – 0.20 ± 0.03 truth table. The F-score of this text-to-table system Template – – – -0.19 ± 0.04 was 35.1%, which is comparable to other chal- Seq2Seq 1 0 No -0.28 ± 0.03 Seq2Seq + Att 1 0 No -0.12 ± 0.03 lenging open-domain settings (Huang et al., 2017). PG-Net 1,4,8 0,1,2,3 No,Yes 0.40 ± 0.03 More details are included in the Appendix A.1. Given this information extraction system, we Table 1: Models used for WikiBio, with the human consider the following metrics for evaluation, evaluation scores for these model outputs and the ref- along the lines of Wiseman et al.(2017). Con- erence texts. PG-Net: Pointer-Generator network. tent Selection (CS): F-score for the (attribute, Human scores computed using Thurstone’s method (Tsukida and Gupta, 2011). value) pairs extracted from the generated text compared to those extracted from the reference. Relation Generation (RG): Precision for the (at- methods since these are most widely used, but we tribute, value) pairs extracted from the generated also include a template-based baseline. All neural text compared to those in the ground truth table. models were trained on the WikiBio training set. RG-F: Since our task emphasizes the recall of in- Training details and sample outputs are included formation from the table as well, we consider an- in Appendices A.2& A.3. other variant which computes the F-score of the We divide these models into two categories and extracted pairs to those in the table. We omit the measure correlation separately for both the cat- content ordering metric, since our extraction sys- egories. The first category, WikiBio-Systems, tem does not align records to the input text. includes one model each from the four families 5 Experiments & Results listed in Table1. This category tests whether a metric can be used to compare different model In this section we compare several automatic eval- families with a large variation in the quality of uation metrics by checking their correlation with their outputs. The second category, WikiBio- the scores assigned by humans to table-to-text Hyperparams, includes 13 different hyperparam- models. Specifically, given l models M1,...,Ml, eter settings of PG-Net (See et al., 2017), which and their outputs on an evaluation set, we show was the best performing system overall. 9 of these these generated texts to humans to judge their were obtained by varying the beam size and length quality, and obtain aggregated human evaluation normalization penalty of the decoder network (Wu ¯ scores for all the models, h = (h1, . . . , hl) (§5.2). et al., 2016), and the remaining 4 were obtained Next, to evaluate an automatic metric, we com- by re-scoring beams of size 8 with the information pute the scores it assigns to each model, a¯ = extraction model described in §4. All the models (a1, . . . , al), and check the Pearson correlation be- in this category produce high quality fluent texts, tween h¯ and a¯ (Graham and Baldwin, 2014).9 and differ primarily on the quantity and accuracy of the information they express. Here we are test- 5.1 Data & Models ing whether a metric can be used to compare simi- Our main experiments are on the WikiBio dataset lar systems with a small variation in performance. (Lebret et al., 2016), which is automatically con- This is an important use-case as metrics are often structed and contains many divergent references. used to tune hyperparameters of a model. In §5.6 we also present results on the data released as part of the WebNLG challenge. 5.2 Human Evaluation We developed several models of varying qual- We collected human judgments on the quality of ity for generating text from the tables in WikiBio. the 16 models trained for WikiBio, plus the refer- This gives us a diverse set of outputs to evaluate ence texts. Workers on a crowd-sourcing platform, the automatic metrics on. Table1 lists the mod- proficient in English, were shown a table with els along with their hyperparameter settings and pairs of generated texts, or a generated text and the their scores from the human evaluation (§5.2). Our reference, and asked to select the one they prefer. focus is primarily on neural sequence-to-sequence Figure2 shows the instructions they were given. 9We observed similar trends for Spearman correlation. Paired comparisons have been shown to be supe- Figure 2: Instructions to crowd-workers for comparing two generated texts. rior to rating scales for comparing generated texts WikiBio WikiBio Average (Callison-Burch et al., 2007). However, for mea- Metric Systems Hyperparams suring correlation the comparisons need to be ag- ROUGE 0.518±0.07C,W -0.585±0.15C,W -0.034 ¯ CIDEr 0.674±0.06C,W -0.516±0.15C,W 0.079 gregated into real-valued scores, h = (h1, . . . , hl), CIDEr-D 0.646±0.06C,W -0.372±0.16C,W 0.137 l = 16 C,W C,W for each of the models. For this, we use METEOR 0.697±0.06 -0.079±0.24 0.309 Thurstone’s method (Tsukida and Gupta, 2011), BLEU 0.548±0.07C,W 0.407±0.15C,W 0.478 which assigns a score to each model based on how CS 0.735±0.06W -0.604±0.16C,W 0.066 many times it was preferred over an alternative. BLEU-T 0.688±0.11W 0.587±0.14C,W 0.638 The data collection was performed separately RG 0.645±0.07C,W 0.749±0.12 0.697 W for models in the WikiBio-Systems and WikiBio- RG-F 0.753±0.06 0.763±0.12 0.758 Hyperparams categories. 1100 tables were sam- PARENT-C 0.776±0.05W 0.755±0.12 0.766 pled from the development set, and for each table PARENT-W 0.912±0.03 0.763±0.12 0.838 we got 8 different sentence pairs annotated across PARENT*-C 0.976±0.01 0.793±0.11 0.885 PARENT*-W 0.982±0.01 0.844±0.10 0.913 the two categories, resulting in a total of 8800 pair- wise comparisons. Each pair was judged by one Table 2: Correlation of metrics with human judgments worker only which means there may be noise at on WikiBio. A superscript of C/W indicates that the the instance-level, but the aggregated system-level correlation is significantly lower than that of PARENT- scores had low variance (cf. Table1). In total C/W using a bootstrap confidence test for α = 0.1. around 500 different workers were involved in the annotation. References were also included in the judgments, denoted as PARENT*-W/C. evaluation, and they received a lower score than PG-Net, highlighting the divergence in WikiBio. 5.4 Correlation Comparison 500 5.3 Compared Metrics We use bootstrap sampling ( iterations) over the 1100 tables for which we collected human an- Text only: We compare BLEU (Papineni notations to get an idea of how the correlation of et al., 2002), ROUGE (Lin, 2004), METEOR each metric varies with the underlying data. In (Denkowski and Lavie, 2014), CIDEr and CIDEr- each iteration, we sample with replacement, tables D(Vedantam et al., 2015) using their publicly along with their references and all the generated available implementations. texts for that table. Then we compute aggregated Information Extraction based: We compare the human evaluation and metric scores for each of the CS, RG and RG-F metrics discussed in §4. models and compute the correlation between the Text & Table: We compare a variant of BLEU, two. We report the average correlation across all denoted as BLEU-T, where the values from the bootstrap samples for each metric in Table2. The table are used as additional references. BLEU- distribution of correlations for the best performing T draws inspiration from iBLEU (Sun and Zhou, metrics are shown in Figure3. 2012) but instead rewards n-grams which match Table2 also indicates whether PARENT is sig- the table rather than penalizing them. For nificantly better than a baseline metric. Graham PARENT, we compare both the word-overlap and Baldwin(2014) suggest using the William’s model (PARENT-W) and the co-occurrence model test for this purpose, but since we are computing (PARENT-C) for determining entailment. We also correlations between only 4/13 systems at a time, compare versions where a single λ is tuned on the this test has very weak power in our case. Hence, entire dataset to maximize correlation with human we use the bootstrap samples to obtain a 1 α − WikiBio-Systems WikiBio-Hyperparams WikiBio-Systems WikiBio-Hyperparams 1.0 1.0 1.0 1.0
0.8 0.8 0.5 0.8 0.6 0.6 0.0 0.6 0.4 0.4 BLEU BLEU BLEU-T BLEU-T 0.2 0.5 0.4 0.2 RG-F RG-F 0.0 PARENT-W PARENT-W 0.0 1.0 0 20 40 60 80 100 0 20 40 60 80 100 BLEU RG-F BLEU RG-F % Entailed % Entailed BLEU-T PRT-CPRT-WPRT*-W BLEU-T PRT-CPRT-WPRT*-W
Figure 3: Distribution of metric correlations across 500 Figure 4: Correlation of the metrics to human judgment bootstrap samples. PRT = PARENT. as the percentage of entailed examples in WikiBio is varied. confidence interval of the difference in correlation between PARENT and any other metric and check cantly higher correlations (PARENT*-W/C). whether this is above 0 (Wilcox, 2016). 5.5 Analysis Correlations are higher for the systems cate- gory than the hyperparams category. The latter In this section we further analyze the performance is a more difficult setting since very similar mod- of PARENT-W10 under different conditions, and els are compared, and hence the variance of the compare to the other best metrics from Table2. correlations is also high. Commonly used metrics which only rely on the reference (BLEU, ROUGE, Effect of Divergence. To study the correlation METEOR, CIDEr) have only weak correlations as we vary the number of divergent references, with human judgments. In the hyperparams cat- we also collected binary labels from workers for egory, these are often negative, implying that tun- whether a reference is entailed by the correspond- ing models based on these may lead to selecting ing table. We define a reference as entailed when it worse models. BLEU performs the best among mentions only information which can be inferred these, and adding n-grams from the table as refer- from the table. Each table and reference pair was ences improves this further (BLEU-T). judged by 3 independent workers, and we used the majority vote as the label for that pair. Overall, Among the extractive evaluation metrics, CS, only 38% of the references were labeled as en- which also only relies on the reference, has poor tailed by the table. Fleiss’ κ was 0.30, which in- correlation in the hyperparams category. RG-F, dicates a fair agreement. We found the workers and both variants of the PARENT metric achieve sometimes disagreed on what information can be the highest correlation for both settings. There is reasonably entailed by the table. no significant difference among these for the hy- perparams category, but for systems, PARENT-W Figure4 shows the correlations as we vary the is significantly better than the other two. While percent of entailed examples in the evaluation set RG-F needs a full information extraction pipeline of WikiBio. Each point is obtained by fixing the in its implementation, PARENT-C only relies on desired proportion of entailed examples, and sam- co-occurrence counts, and PARENT-W can be pling subsets from the full set which satisfy this used out-of-the-box for any dataset. To our knowl- proportion. PARENT and RG-F remain stable and edge, this is the first rigorous evaluation of using show a high correlation across the entire range, information extraction for generation evaluation. whereas BLEU and BLEU-T vary a lot. In the hy- perparams category, the latter two have the worst On this dataset, the word-overlap model showed correlation when the evaluation set contains only higher correlation than the co-occurrence model entailed examples, which may seem surprising. for entailment. In §5.6 we will show that for However, on closer examination we found that this the WebNLG dataset, where more paraphrasing subset tends to omit a lot of information from the is involved between the table and text, the oppo- tables. Systems which produce more information site is true. Lastly, we note that the heuristic for than these references are penalized by BLEU, but selecting λ is sufficient to produce high correla- not in the human evaluation. PARENT overcomes tions for PARENT, however, if human annotations are available, this can be tuned to produce signifi- 10The trends were similar for PARENT-C. BLEU BLEU-T RG-F PARENT-W PARENT-C Metric Grammar Fluency Semantics Avg 0.556 0.567∗ 0.588∗ 0.598‡ 0.606† METEOR 0.788±0.04 0.792±0.04 0.576±0.06 0.719 ROUGE 0.788±0.04 0.792±0.04 0.576±0.06 0.719 ± ± ± Table 3: Accuracy on making the same judgments CIDEr 0.804 0.03 0.753 0.04 0.860 0.02 0.806 BLEU 0.858±0.02 0.811±0.03 0.775±0.03 0.815 as humans between pairs of generated texts. p < BLEU-T 0.849±0.02 0.801±0.03 0.816±0.02 0.822 0.01∗/0.05†/0.10‡: accuracy is significantly higher CIDErD 0.838±0.04 0.796±0.04 0.853±0.02 0.829 than the next best accuracy to the left using a paired PARENT-W 0.821±0.03 0.768±0.04 0.887±0.02 0.825 McNemar’s test. PARENT-C 0.851±0.03 0.809±0.04 0.877±0.02 0.846
Table 4: Average pearson correlation across 500 boot- this issue by measuring recall against the table in strap samples of each metric to human ratings for each addition to the reference. aspect of the generations from the WebNLG challenge. Ablation Study. We check how different com- ponents in the computation of PARENT contribute on 223 instances. These systems include a mix of to its correlation to human judgments. Specifi- pipelined, statistical and neural methods. Each in- cally, we remove the probability w(g) of an n- stance has upto 3 reference texts associated with gram g being entailed by the table from Eqs.2 the RDF triples, which we use for evaluation. and5. 11 The average correlation for PARENT-W The human ratings were collected on 3 dis- drops to 0.168 in this case. We also try a variant tinct aspects – grammaticality, fluency and seman- of PARENT with λ = 0, which removes the con- tics, where semantics corresponds to the degree to tribution of Table Recall (Eq.4). The average cor- which a generated text agrees with the meaning of relation is 0.328 in this case. With these compo- the underlying RDF triples. We report the corre- nents, the correlation is 0.838, showing that they lation of several metrics with these ratings in Ta- are crucial to the performance of PARENT. ble4. 12 Both variants of PARENT are either com- petitive or better than the other metrics in terms of Sentence Level Discrimination. Chaganty et al. the average correlation to all three aspects. This (2018) point out that hill-climbing on an automatic shows that PARENT is applicable for high quality metric is meaningless if that metric has a low references as well. instance-level correlation to human judgments. In While BLEU has the highest correlation for the Table3 we show the average accuracy of the met- grammar and fluency aspects, PARENT does best rics in making the same judgments as humans be- for semantics. This suggests that the inclusion of tween pairs of generated texts. Both variants of source tables into the evaluation orients the metric PARENT are significantly better than the other more towards measuring the fidelity of the content metrics, however the best accuracy is only 60% for of the generation. A similar trend is seen compar- the binary task. This is a challenging task, since ing BLEU and BLEU-T. As modern neural text there are typically only subtle differences between generation systems are typically very fluent, mea- the texts. Achieving higher instance-level accura- suring their fidelity is of increasing importance. cies will require more sophisticated language un- Between the two entailment models, PARENT- derstanding models for evaluation. C is better due to its higher correlation with the 5.6 WebNLG Dataset grammaticality and fluency aspects. To check how PARENT correlates with human Distribution of λ. The λ parameter in the calcu- judgments when the references are elicited from lation of PARENT decides whether to compute re- humans (and less likely to be divergent), we check call against the table or the reference (Eq.4). Fig- its correlation with the human ratings provided for ure5 shows the distribution of the values taken by the systems competing in the WebNLG challenge 1 λ using the heuristic described in §3 for in- (Gardent et al., 2017). The task is to generate text stances− in both WikiBio and WebNLG. For Wik- describing 1-5 RDF triples (e.g. John E Blaha, iBio, the recall of the references against the table birthPlace, San Antonio), and human ratings were is generally low, and hence the recall of the gener- collected for the outputs of 9 participating systems 12 We omit extractive evaluation metrics since no extrac- 11When computing precision we set w(g) = 0, and when tion systems are publicly available for this dataset, and devel- computing recall we set w(g) = 1 for all g. oping one is beyond the scope of this work. 400 and Zhou, 2012), a metric for evaluating para- WikiBio 350 WebNLG phrase generation, which compares the generated 300 text to both the source text and the reference. 250 While iBLEU penalizes texts which match the 200 source, here we reward such texts since our task Frequency 150 values accuracy of generated text more than the 100 need for paraphrasing the tabular content (Liu 50 et al., 2010). Similar to SARI for text simplifica- 0 0.0 0.2 0.4 0.6 0.8 1.0 tion (Xu et al., 2016) and Q-BLEU for question 1 λ − generation (Nema and Khapra, 2018), PARENT Figure 5: Histogram of the recall of the references falls under the category of task-specific metrics. against the table (Eq.6), which is used to set 1 λ. Lower values indicate that the metric relies more− on 7 Conclusions the table and less on the reference. We study the automatic evaluation of table-to-text systems when the references diverge from the ta- ated text relies more on the table. For WebNLG, ble. We propose a new metric, PARENT, which where the references are elicited from humans, shows the highest correlation with humans across this recall is much higher (often 1.0), and hence a range of settings with divergent references in the recall of the generated text relies more on the WikiBio. We also perform the first empirical reference. evaluation of information extraction based metrics (Wiseman et al., 2017), and find RG-F to be ef- 6 Related Work fective. Lastly, we show that PARENT is compa- rable to the best existing metrics when references Over the years several studies have evaluated au- are elicited by humans on the WebNLG data. tomatic metrics for measuring text generation per- formance (Callison-Burch et al., 2006; Stent et al., Acknowledgements 2005; Belz and Reiter, 2006; Reiter, 2018; Liu et al., 2016; Kilickaya et al., 2017; Gatt and Krah- Bhuwan Dhingra is supported by a fellowship mer, 2018). The only consensus from these studies from Siemens, and by grants from Google. We seems to be that no single metric is suitable across thank Maruan Al-Shedivat, Ian Tenney, Tom all tasks. A recurring theme is that metrics like Kwiatkowski, Michael Collins, Slav Petrov, Jason BLEU and NIST (Doddington, 2002) are not suit- Baldridge, David Reitter and other members of the able for judging content quality in NLG. Recently, Google AI Language team for helpful discussions Novikova et al.(2017a) did a comprehensive study and suggestions. We thank Sam Wiseman for shar- of several metrics on the outputs of state-of-the-art ing data for an earlier version of this paper. We NLG systems, and found that while they showed also thank the anonymous reviewers for their feed- acceptable correlation with human judgments at back. the system level, they failed to show any correla- tion at the sentence level. Ours is the first study References which checks the quality of metrics when table- Srinivas Bangalore, Owen Rambow, and Steve Whit- to-text references are divergent. We show that in taker. 2000. Evaluation metrics for generation. In this case even system level correlations can be un- Proc. of INLG. reliable. Regina Barzilay and Mirella Lapata. 2005. Collective Hallucination (Rohrbach et al., 2018; Lee et al., content selection for concept-to-text generation. In 2018) refers to when an NLG system generates Proc. of EMNLP. text which mentions extra information than what Anja Belz and Ehud Reiter. 2006. Comparing auto- is present in the source from which it is generated. matic and human evaluation of nlg systems. In Proc. Divergence can be viewed as hallucination in the of EACL. reference text itself. PARENT deals with halluci- nation by discounting n-grams which do not over- Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. lap with either the reference or the table. (meta-) evaluation of machine translation. In Proc. PARENT draws inspiration from iBLEU (Sun of WMT. Chris Callison-Burch, Miles Osborne, and Philipp Karen Kukich. 1983. Design of a knowledge-based re- Koehn. 2006. Re-evaluation the role of bleu in ma- port generator. In Proc. of ACL. chine translation research. In Proc. of EACL. Rémi Lebret, David Grangier, and Michael Auli. 2016. Marine Carpuat, Yogarshi Vyas, and Xing Niu. 2017. Neural text generation from structured data with Detecting cross-lingual semantic divergence for application to the biography domain. In Proc. of neural machine translation. In Proc.of Workshop on EMNLP. NMT. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Arun Chaganty, Stephen Mussmann, and Percy Liang. Fannjiang, and David Sussillo. 2018. Hallucinations 2018. The price of debiasing automatic metrics in in neural machine translation. natural language evalaution. In Proc. of ACL. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Boxing Chen and Colin Cherry. 2014. A systematic Learning semantic correspondences with less super- comparison of smoothing techniques for sentence- vision. In Proc. of ACL. level bleu. In Proc. of WMT. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. of Workshop on Ido Dagan, Oren Glickman, and Bernardo Magnini. Text Summarization Branches Out. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Eval- Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. uating Predictive Uncertainty, Visual Object Classi- 2010. PEM: A paraphrase evaluation metric exploit- fication, and Recognising Tectual Entailment, pages ing parallel texts. In Proc. of EMNLP. 177–190, Berlin, Heidelberg. Springer Berlin Hei- delberg. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Nose- worthy, Laurent Charlin, and Joelle Pineau. 2016. Michael Denkowski and Alon Lavie. 2014. Meteor How not to evaluate your dialogue system: An em- universal: Language specific translation evaluation pirical study of unsupervised evaluation metrics for for any target language. In Proc. of WMT. dialogue response generation. In Proc. of EMNLP. George Doddington. 2002. Automatic evaluation Kathleen R. McKeown. 1985. Text Generation: Using of machine translation quality using n-gram co- Discourse Strategies and Focus Constraints to Gen- occurrence statistics. In Proc. of HLT. erate Natural Language Text. Cambridge University Press, New York, NY, USA. Ondrejˇ Dušek, Jekaterina Novikova, and Verena Rieser. 2019. Evaluating the state-of-the-art of end-to-end Preksha Nema and Mitesh M Khapra. 2018. Towards a natural language generation: The E2E NLG Chal- better metric for evaluating question generation sys- lenge. arXiv preprint arXiv:1901.11528. tems. In Proc. of EMNLP.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, Jekaterina Novikova, Ondrejˇ Dušek, Amanda Cercas and Laura Perez-Beltrachini. 2017. Creating train- Curry, and Verena Rieser. 2017a. Why we need new ing corpora for micro-planners. In Proc. of ACL. evaluation metrics for NLG. In Proc. of EMNLP.
Albert Gatt and Emiel Krahmer. 2018. Survey of the Jekaterina Novikova, Ondrejˇ Dušek, and Verena Rieser. state of the art in natural language generation: Core 2017b. The E2E dataset: New challenges for end- tasks, applications and evaluation. Journal of Artifi- to-end generation. In Proc. of SIGDIAL. cial Intelligence Research, 61:65–170. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Oren Glickman and Ido Dagan. 2005. A probabilistic Jing Zhu. 2002. Bleu: A method for automatic eval- setting and lexical cooccurrence model for textual uation of machine translation. In Proc. of ACL. entailment. In Proceedings of the ACL Workshop on Jeffrey Pennington, Richard Socher, and Christopher Empirical Modeling of Semantic Equivalence and Manning. 2014. Glove: Global vectors for word rep- Entailment, pages 43–48. Association for Compu- resentation. In Proceedings of the 2014 conference tational Linguistics. on empirical methods in natural language process- ing (EMNLP), pages 1532–1543. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human Ehud Reiter. 2018. A structured review of the validity judgment. In Proc. of EMNLP. of bleu. Computational Linguistics, pages 1–12. Lifu Huang, Avirup Sil, Heng Ji, and Radu Florian. Ehud Reiter and Robert Dale. 1997. Building applied 2017. Improving slot filling performance with at- natural language generation systems. Nat. Lang. tentive neural networks on dependency structures. Eng., 3(1):57–87. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, and Erkut Erdem. 2017. Re-evaluating automatic Trevor Darrell, and Kate Saenko. 2018. Object hal- metrics for image captioning. In Proc. of EACL. lucination in image captioning. EMNLP. Abigail See, Peter J. Liu, and Christopher D. Manning. Text: 2017. Get to the point: Summarization with pointer- michael dahlquist ( december 22 , 1965 – july 14 , 2005 ) was a drummer in the seattle band silkworm . generator networks. In Proc. of ACL. Table: Amanda Stent, Matthew Marge, and Mohit Singhai. name
Table 6: Sample references and predictions from PG-Net with beam size 8. Information which is absent from the reference, but can be inferred from the table is in bold. Information which is present in the reference, but cannot be inferred from the table is in italics.
was set to 200. The input and output vocabularies were set to 50000 most common words in the cor- pus, with additional special symbols for table at- tribute names (such as “birth-date”). The embed- dings of the tokens in the vocabulary were initial- ized with Glove (Pennington et al., 2014). Learn- ing rate of 0.0003 was used during training, with the Adam optimizer, and a dropout of 0.2 was also applied to the outputs of the biLSTM. Mod- els were trained till the loss on the dev set stopped dropping. Maximum length of a decoded text was set to 40 tokens, and that of the tables was set to 120 tokens. Various beam sizes and length nor- malization penalties were applied for the table-to- text system, which are listed in the main paper. For the information extraction system, we found a beam size of 8 and no length penalty to produce the highest F-score on the dev set.
A.3 Sample Outputs Table6 shows some sample references and the cor- responding predictions from the best performing model, PG-Net for WikiBio.