A Simple Framework for Contrastive Learning of Abstractive Summarization
Total Page:16
File Type:pdf, Size:1020Kb
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization Yixin Liu, Pengfei Liu Carnegie Mellon University Motivation ● Training – Maximum Likelihood Estimation Source Document Probability Abstractive distribution of Model the next token Reference Summary (partial) MLE loss Next Token Motivation ● Evaluation – Autoregressive Generation Source Document Candidate Abstractive Summary Model Reference ROUGE Score Summary Motivation Training Source Document Probability Abstractive distribution of the Model Reference next token Summary (partial) MLE loss Next Token Evaluation Source Document Abstractive Candidate Model Summary Reference ROUGE Score Summary Reinforcement Learning Training Source Document Abstractive Candidate Model Summary Reference RL Reward Summary Evaluation Source Document Abstractive Candidate Model Summary Reference ROUGE Score Summary SimCLS S1 Ref-free Score 1 Doc Ref Real Score 1 Real Predicted Difference Difference Real Score 2 Ref-free Score 2 S2 Generation Reference-dependent Score Contrastive Learning SimCLS S1 Ref-free Score 1 Doc Ref Real Score 1 Real Predicted Difference Difference Real Score 2 Ref-free Score 2 S2 Generation Reference-dependent Score Contrastive Learning SimCLS S1 Ref-free Score 1 Doc Ref Real Score 1 Real Predicted Difference Difference Real Score 2 Ref-free Score 2 S2 Generation Reference-dependent Score Contrastive Learning SimCLS S1 Ref-free Score 1 Doc Ref Real Score 1 Real Predicted Difference Difference Real Score 2 Ref-free Score 2 S2 Generation Reference-dependent Score Contrastive Learning SimCLS S1 Ref-free Score 1 Doc Ref Real Score 1 Real Predicted Difference Difference Real Score 2 Ref-free Score 2 S2 Generation Reference-dependent Score Contrastive Learning Experiments on CNNDM Experiments on CNNDM Experiments on CNNDM Experiments on CNNDM Case Study on CNNDM System Summary Article chris ramsey says he has no problem shaking queens park rangers manager chris ramsey has revealed he will hands with john terry . queens park rangers host have no problem shaking john terry's hand in light of the racist comments the former england captain directed at former rs defender chelsea in the premier league on sunday . terry Ref. anton ferdinand four years ago . terry , who will line up against was once banned and fined for racist comments at ramsey's side , was banned for four games and fined \# 220,000 for loftus road . rio ferdinand , brother of anton , will the remarks made in october 2011 during chelsea's 1-0 defeat at not be fit to play against chelsea . loftus road . but ramsey , the premier league's only black manager , thinks the issue has been dealt with . ... ` i don't know what his queens park rangers host chelsea in the premier feelings are towards me . as long as there wasn't anything on the league on sunday . qpr boss chris ramsey says he field that was unprofessional by him , i would shake his hand . will have no problem shaking john terry's hand . queens park rangers manager chris ramsey speaks to the media on friday ahead of the chelsea match . chelsea captain john terry SimCLS terry was banned for four games and fined \# controls the ball during last weekend's premier league match against 220,000 for racist comments . rio ferdinand , stoke . ramsey arrives for friday's pre-match press conference as brother of anton , will not be fit for the match at qpr prepare to host chelsea at loftus road . ` the whole episode for loftus road . british society sat uncomfortably . it's not something we want to highlight in football . it happened and it's being dealt with . we have john terry was banned for four games and fined \# to move on . and hopefully everyone has learned something from it . 220,000 for the remarks made in october 2011 ' . ramsey revealed that rio ferdinand , who labelled terry an idiot for during chelsea's 1-0 defeat at loftus road . terry will the abuse aimed at his brother , won't be fit in time for a reunion with the chelsea skipper this weekend . but the 52-year-old suspects his Origin. line up against chris ramsey's side on sunday . rio player's one-time england colleague will be on the receiving end of a ferdinand , who labelled terry an idiot for the abuse hostile welcome from the home fans on his return the scene of the aimed at his brother , won't be fit in time for a unsavoury incident . ... ferdinand and terry argue during qpr's 1-0 reunion with the chelsea skipper this weekend victory against chelsea at loftus road in october 2011 . rio ferdinand , brother of anton , will not be fit for sunday's match against chelsea . Case Study on CNNDM System Summary Article chris ramsey says he has no problem shaking queens park rangers manager chris ramsey has revealed he will hands with john terry . queens park rangers host have no problem shaking john terry's hand in light of the racist comments the former england captain directed at former rs chelsea in the premier league on sunday . terry was Ref. defender anton ferdinand four years ago . terry , who will line up once banned and fined for racist comments at loftus against ramsey's side , was banned for four games and fined \# road . rio ferdinand , brother of anton , will not be fit 220,000 for the remarks made in october 2011 during chelsea's to play against chelsea . 1-0 defeat at loftus road . but ramsey , the premier league's only black manager , thinks the issue has been dealt with . ... ` i don't queens park rangers host chelsea in the premier know what his feelings are towards me . as long as there wasn't league on sunday . qpr boss chris ramsey says he anything on the field that was unprofessional by him , i would shake will have no problem shaking john terry's hand . his hand . queens park rangers manager chris ramsey speaks to the media on friday ahead of the chelsea match . chelsea SimCLS terry was banned for four games and fined \# captain john terry controls the ball during last weekend's premier 220,000 for racist comments . rio ferdinand , league match against stoke . ramsey arrives for friday's pre-match brother of anton , will not be fit for the match at press conference as qpr prepare to host chelsea at loftus road . ` the loftus road . whole episode for british society sat uncomfortably . it's not something we want to highlight in football . it happened and it's being john terry was banned for four games and fined \# dealt with . we have to move on . and hopefully everyone has learned 220,000 for the remarks made in october 2011 something from it . ' . ramsey revealed that rio ferdinand , who during chelsea's 1-0 defeat at loftus road . terry will labelled terry an idiot for the abuse aimed at his brother , won't be fit in time for a reunion with the chelsea skipper this weekend . but the line up against chris ramsey's side on sunday . rio 52-year-old suspects his player's one-time england colleague will be Origin. ferdinand , who labelled terry an idiot for the abuse on the receiving end of a hostile welcome from the home fans on his aimed at his brother , won't be fit in time for a return the scene of the unsavoury incident . ... ferdinand and terry reunion with the chelsea skipper this weekend argue during qpr's 1-0 victory against chelsea at loftus road in october 2011 . rio ferdinand , brother of anton , will not be fit for sunday's match against chelsea . Case Study on CNNDM System Summary Article chris ramsey says he has no problem shaking queens park rangers manager chris ramsey has revealed he will hands with john terry . queens park rangers host have no problem shaking john terry's hand in light of the racist comments the former england captain directed at former rs defender chelsea in the premier league on sunday . terry was Ref. anton ferdinand four years ago . terry , who will line up against once banned and fined for racist comments at loftus ramsey's side , was banned for four games and fined \# 220,000 road . rio ferdinand , brother of anton , will not be fit for the remarks made in october 2011 during chelsea's 1-0 to play against chelsea . defeat at loftus road . but ramsey , the premier league's only black manager , thinks the issue has been dealt with . ... ` i don't know queens park rangers host chelsea in the premier what his feelings are towards me . as long as there wasn't anything league on sunday . qpr boss chris ramsey says he on the field that was unprofessional by him , i would shake his hand . will have no problem shaking john terry's hand . queens park rangers manager chris ramsey speaks to the media on friday ahead of the chelsea match . chelsea captain john terry SimCLS terry was banned for four games and fined \# controls the ball during last weekend's premier league match against 220,000 for racist comments . rio ferdinand , stoke . ramsey arrives for friday's pre-match press conference as qpr brother of anton , will not be fit for the match at prepare to host chelsea at loftus road . ` the whole episode for british loftus road . society sat uncomfortably . it's not something we want to highlight in football . it happened and it's being dealt with . we have to move on . john terry was banned for four games and fined \# and hopefully everyone has learned something from it . ' . ramsey 220,000 for the remarks made in october 2011 revealed that rio ferdinand , who labelled terry an idiot for the during chelsea's 1-0 defeat at loftus road . terry will abuse aimed at his brother , won't be fit in time for a reunion with the chelsea skipper this weekend .