Bootstrapping Generators from Noisy Data

Bootstrapping Generators from Noisy Data Laura Perez-Beltrachini Mirella Lapata School of Informatics University of Edinburgh {lperez,mlap}@inf.ed.ac.uk 1 / 20 Application? Automatic generation of descriptions for Amazon Approach? Neural Generator trained on Loosely Related Texts The Data to Text Generation Task Input Set of Properties Birth name Robert Joseph Flaherty Output Verbalisation Birth date February 16, 1884 Birth place Iron Mountain, Michigan, U.S. Robert Joseph Flaherty, (February 16, 1884 Death date July 23, 1951 (aged 67) July 23, 1951) was an American film-maker. Death place Dummerston, Vermont, U.S. Flaherty was married to Frances H. Flaherty Cause of Cerebral thrombosis death until his death in 1951. Occupation Filmmaker Spouse(s) Frances Johnson Hubbard 2 / 20 Approach? Neural Generator trained on Loosely Related Texts The Data to Text Generation Task Input Set of Properties Birth name Robert Joseph Flaherty Output Verbalisation Birth date February 16, 1884 Birth place Iron Mountain, Michigan, U.S. Robert Joseph Flaherty, (February 16, 1884 Death date July 23, 1951 (aged 67) July 23, 1951) was an American film-maker. Death place Dummerston, Vermont, U.S. Flaherty was married to Frances H. Flaherty Cause of Cerebral thrombosis death until his death in 1951. Occupation Filmmaker Spouse(s) Frances Johnson Hubbard Application? Automatic generation of descriptions for Amazon 2 / 20 The Data to Text Generation Task Input Set of Properties Birth name Robert Joseph Flaherty Output Verbalisation Birth date February 16, 1884 Birth place Iron Mountain, Michigan, U.S. Robert Joseph Flaherty, (February 16, 1884 Death date July 23, 1951 (aged 67) July 23, 1951) was an American film-maker. Death place Dummerston, Vermont, U.S. Flaherty was married to Frances H. Flaherty Cause of Cerebral thrombosis death until his death in 1951. Occupation Filmmaker Spouse(s) Frances Johnson Hubbard Application? Automatic generation of descriptions for Amazon Approach? Neural Generator trained on Loosely Related Texts 2 / 20 Learning Neural Generators from Loosely Related Data-Text Pairs Joseph Flaherty, (February 16, 1884 – July 23, 1951) was an American film-maker who directed and produced the first commercially successful Birth name Robert Joseph Flaherty feature-length documentary film, Nanook of the Birth date February 16, 1884 North (1922). Flaherty was married to writer Birth place Iron Mountain, Michigan, U.S. Frances H. Flaherty from 1914 until his death in Death date July 23, 1951 (aged 67) 1951. Frances worked on several of her husband’s Death place Dummerston, Vermont, U.S. films, and received an Academy Award nomination Cause of Cerebral thrombosis for Best Original Story for Louisiana Story (1948). death Occupation Filmmaker Spouse(s) Frances Johnson Hubbard [Lebret et al., 2016, Wiseman et al., 2017] 3 / 20 Distribute attention weights & Memorised high frequency sub-sequences! [Ghader and Monz, 2017] Learning Neural Generators from Loosely Related Data-Text Pairs Joseph Flaherty, (February 16, 1884 – July 23, 1951) was an American film-maker who directed Birth name Robert Joseph Flaherty and produced the first commercially successful Birth date February 16, 1884 feature-length documentary film, Nanook of the Birth place Iron Mountain, Michigan, U.S. North (1922). Flaherty was married to writer Death date July 23, 1951 (aged 67) Frances H. Flaherty from 1914 until his death in Death place Dummerston, Vermont, U.S. Cause of Cerebral thrombosis 1951. Frances worked on several of her husband’s death films, and received an Academy Award nomination Occupation Filmmaker for Best Original Story for Louisiana Story (1948). Spouse(s) Frances Johnson Hubbard 4 / 20 Learning Neural Generators from Loosely Related Data-Text Pairs Joseph Flaherty, (February 16, 1884 – July 23, 1951) was an American film-maker who directed Birth name Robert Joseph Flaherty and produced the first commercially successful Birth date February 16, 1884 feature-length documentary film, Nanook of the Birth place Iron Mountain, Michigan, U.S. North (1922). Flaherty was married to writer Death date July 23, 1951 (aged 67) Frances H. Flaherty from 1914 until his death in Death place Dummerston, Vermont, U.S. Cause of Cerebral thrombosis 1951. Frances worked on several of her husband’s death films, and received an Academy Award nomination Occupation Filmmaker for Best Original Story for Louisiana Story (1948). Spouse(s) Frances Johnson Hubbard Distribute attention weights & Memorised high frequency sub-sequences! [Ghader and Monz, 2017] 4 / 20 Our Approach, Pre-Train a Content Alignment Model Birth dateBirth name Spouse(s) Birth place Dead date February 16 1884 Robert Flaherty Frances J. Flaherty Iron Mountain july 23 1951 Flaherty was married to writer Frances Flaherty from 1914 until his death in 1951 5 / 20 Our Approach, Use the Content Alignment Information for Training 6 / 20 Property Set Bag Bag Labels Flaherty was married to writer Birth name Robert Joseph Flaherty Birth date February 16, 1884 Frances H. Flaherty from 1914 Birth place Iron Mountain, Michigan, U.S. until his death in 1951. Death date July 23, 1951 (aged 67) Death place Dummerston, Vermont, U.S. Cause of Cerebral thrombosis death Occupation Filmmaker Spouse(s) Frances Johnson Hubbard Content Alignment Intuition Multi-Instance Learning to discover Property-Word Alignments Loosely related (Property Set, Text) pairs provide high level supervision Property Set Bag Bag Labels Flaherty was married to writer Birth name Robert Joseph Flaherty Birth date February 16, 1884 Frances H. Flaherty from 1914 Birth place Iron Mountain, Michigan, U.S. until his death in 1951. Death date July 23, 1951 (aged 67) Death place Dummerston, Vermont, U.S. Cause of Cerebral thrombosis death Occupation Filmmaker Spouse(s) Frances Johnson Hubbard [Keeler and Rumelhart, 1992, Karpathy and Fei-Fei, 2015] 7 / 20 Property Set Bag Bag Labels Flaherty was married to writer Birth name Robert Joseph Flaherty Birth date February 16, 1884 Frances H. Flaherty from 1914 Birth place Iron Mountain, Michigan, U.S. until his death in 1951. Death date July 23, 1951 (aged 67) Death place Dummerston, Vermont, U.S. Cause of Cerebral thrombosis death Occupation Filmmaker Spouse(s) Frances Johnson Hubbard Content Alignment Intuition Multi-Instance Learning to discover Property-Word Alignments Loosely related (Property Set, Text) pairs provide high level supervision Property Set Bag Bag Labels Flaherty was married to writer Birth name Robert Joseph Flaherty Birth date February 16, 1884 Frances H. Flaherty from 1914 Birth place Iron Mountain, Michigan, U.S. until his death in 1951. Death date July 23, 1951 (aged 67) Death place Dummerston, Vermont, U.S. Cause of Cerebral thrombosis death Occupation Filmmaker Spouse(s) Frances Johnson Hubbard [Keeler and Rumelhart, 1992, Karpathy and Fei-Fei, 2015] 7 / 20 Content Alignment Model jsj X SPs = maxi2f1;:::;jPjg pi • wt t=1 LCA = max(0; SPs − SPs 0 + 1) + max(0; SPs − SP 0s + 1) P is a property set and s a sentence from the text each property vector pi is learned by an LSTM encoder word vectors wt are hidden states of an LSTM sentence encoder [Keeler and Rumelhart, 1992, Karpathy and Fei-Fei, 2015] 8 / 20 Multi-Task Learning 9 / 20 Predicting Alignment Labels Words from the original text are associated with Alignment Labels a indicating whether the word aligns to some property in the input Simultaneously predict words and alignments, EDMTL model LMTL = λ LwNLL + (1 − λ) Laln jY j X LwNLL = − log P(yt jy1:t−1; X) t=1 jY j X Laln = − log P(at jy1:t−1; X) t=1 [Caruana, 1993] 10 / 20 Reinforcement Learning 11 / 20 Training to Optimise Content Target Content is set of words from the original text with positive alignments Trained to maximise a Content Precision Reward, EDRL model r(Y^ ) = γpr r pr (Y^ ) r pr is the unigram precision of Y^ and Target Content LRL = − ^ ^ ∼ Pπ(·jX)[r(y^1; ··· ; y^ ^ )] E(y1;··· ;yjY^ j) jY j [Williams, 1992, Ranzato et al., 2016, Zhang and Lapata, 2017] 12 / 20 Automatic BLEU-4 on gold texts Human ranking of 4 models and gold texts; AMT judges Ranking criteria: (1) Is the text faithful to the set of properties? (2) Is the text comprehensible and fluent? Experimental Setup properties and abstracts about people (WikiBio [Lebret et al., 2016]) Encoder-Decoder baseline, ED model Hand-crafted templates for 50 most freqent relations, Templ 13 / 20 Experimental Setup properties and abstracts about people (WikiBio [Lebret et al., 2016]) Encoder-Decoder baseline, ED model Hand-crafted templates for 50 most freqent relations, Templ Automatic BLEU-4 on gold texts Human ranking of 4 models and gold texts; AMT judges Ranking criteria: (1) Is the text faithful to the set of properties? (2) Is the text comprehensible and fluent? 13 / 20 Automatic Evaluation Results 40 Templ ED 30 EDMTL EDRL 20 BLEU-4 15 10 5 Gold Texts 14 / 20 1 Human Evaluation Results significance at p < 0:05 50 40 30 20 Rankings as proportions 10 0 1 2 3 4 5 Templ ED EDMTL EDRL Gold Texts 15 / 20 1 Example Output Property Set: name= dorsey burnette, date= may 2012, bot= blevintron bot, background= solo singer, birth= december 28 , 1932, birth place= memphis, tennessee, death place= flos angeles; canoga park, californiag, death= august 19 , 1979, associated acts= the rock and roll trio, hometown= memphis, tennessee, genre= frock and roll; rockabilly; country musicg, occupation= fcomposer; singerg, instruments= frockabilly bass; vocals; acoustic guitarg, record labels= fera records; coral records; smash records; imperial records; capitol

Load more