On Extractive and Abstractive Neural Document Summarization with Transformer Language Models Jonathan Pilault∗ 1;2;3, Raymond Li∗1, Sandeep Subramanian∗1;2;4 and Christopher Pal1;2;3;4;5 1Element AI 2Mila, 3Polytechnique Montreal, 4University of Montreal, 5Canada CIFAR AI Chair [email protected] Neural Document Summarization with Sentence Neural Document Summarization with Sentence Abstract Pointer Networks and Transformer Language Models Pointer Networks and Transformer Language Models 1,2,3, 1, 1,2,4, 1,2,3, 1, 1,2,4, Sandeep Subramanian ⇤, Raymond Li ⇤, Jonathan Pilault ⇤, Sandeep Subramanian ⇤, Raymond Li ⇤, Jonathan Pilault ⇤, 1,2,4,5 Christopher Pal Christopher Pal1,2,4,5 1Element AI, 2Montréal Institute for Learning Algorithms, 3Université de Montréal, 1Element AI, 2Montréal Institute for Learning Algorithms, 3Université de Montréal, 4École Polytechnique de Montréal, 5Canada CIFAR AI Chair 4École Polytechnique de Montréal, 5Canada CIFAR AI Chair 1{jonathan.pilault}@elementai.com 1{jonathan.pilault}@elementai.com Abstract Abstract We demonstrate that Transformer language models are extremely promising at We demonstrate that Transformer language models are extremely promising at summarizing long texts, and provide a new approach to deep summarization that summarizing long texts, and provide a new approach to deep summarization that can be used to generate more "abstractive" summaries. We show that our approach can be used to generate more "abstractive" summaries. We show that our approach produces more abstractive summaries than state-of-the-art methods without a copy produces more abstractive summaries than state-of-the-art methods without a copy mechanism. We provide an application to text summarization of the arXiv and mechanism. We provide an application to text summarization of the arXiv and PubMed datasets, and show that our model outperforms other popular summa- PubMed datasets, and show that our model outperforms other popular summa- rization techniques. We also discuss a simple neural extractive model based on rization techniques. We also discuss a simple neural extractive model based on pointers networks trained on documents and their salient sentences. We show that pointers networks trained on documents and their salient sentences. We show that this model can be used to augment Transformer language models to generate better this model can be used to augment Transformer language models to generate better summarization results. Note: The abstract above was generated by one of the summarization results. Note: The abstract above was generated by one of the We present a method to produce abstractive models presented in this paper, as a summary of this paper. models presented in this paper, as a summary of this paper. 1 Introduction 1 Introduction Language models (LMs) are trained to estimate the joint probability of an arbitrary sequence of Language models (LMs) are trained to estimate the joint probability of an arbitrary sequence of words or characters using a large corpus of text. They typically factorize the joint distribution of words or characters using a large corpus of text. They typically factorize the joint distribution of n n tokens p(x1,x2 ...xn) into a product of conditional probabilities p(xi x<i). It is possible to use tokens p(x ,x ...x ) into a product of conditional probabilities p(x x ). It is possible to use i | 1 2 n i i <i n-gram based models to estimate these conditional probabilities via counts, relying on Markovian n-gram based models to estimate these conditional probabilities via counts,| relying on Markovian assumptions. However, Markovian assumptions and the curse ofQ dimensionality make it harder for assumptions. However, Markovian assumptions and the curse ofQ dimensionality make it harder for n-gram LMs to model long range dependencies and learn smooth functions that can learn similarities n-gram LMs to model long range dependencies and learn smooth functions that can learn similarities between words in the vocabulary. This has led to a preference for recurrent or feed-forward neural between words in the vocabulary. This has led to a preference for recurrent or feed-forward neural language models (Bengio et al., 2003; Mikolov et al., 2010) in recent years due to to their ability to language models (Bengio et al., 2003; Mikolov et al., 2010) in recent years due to to their ability to summaries of long documents that exceed sev- learn expressive conditional probability distributions (Merity et al., 2017; Radford et al., 2019). learn expressive conditional probability distributions (Merity et al., 2017; Radford et al., 2019). The sequence-to-sequence (seq2seq) paradigm (Sutskever et al., 2014) uses language models that The sequence-to-sequence (seq2seq) paradigm (Sutskever et al., 2014) uses language models that learn the conditional probability of one sequence given another. Here, a language model serves learn the conditional probability of one sequence given another. Here, a language model serves as a “decoder” that is typically conditioned on a representation of an input sequence produced by as a “decoder” that is typically conditioned on a representation of an input sequence produced by an encoder neural network. These types of encoder-decoder architectures have been particularly an encoder neural network. These types of encoder-decoder architectures have been particularly successful when applied to problems such as machine translation (Bahdanau et al., 2014) and successful when applied to problems such as machine translation (Bahdanau et al., 2014) and abstractive summarization (Rush et al., 2015). The encoder and conditional decoder language models abstractive summarization (Rush et al., 2015). The encoder and conditional decoder language models are often paramaterized as recurrent neural networks (RNNs). Attention mechanisms (Bahdanau are often paramaterized as recurrent neural networks (RNNs). Attention mechanisms (Bahdanau et al., 2014) are used in the decoder to provide more informative conditioning on the representations et al., 2014) are used in the decoder to provide more informative conditioning on the representations produced by the encoder and to ease gradient flow into the encoder. RNNs however, are limited by produced by the encoder and to ease gradient flow into the encoder. RNNs however, are limited by their sequential nature, making them 1) difficult to optimize and learn for long sequences with long their sequential nature, making them 1) difficult to optimize and learn for long sequences with long range dependencies (Hochreiter, 1998; Pascanu et al., 2013), and 2) hard to parallelize on modern range dependencies (Hochreiter, 1998; Pascanu et al., 2013), and 2) hard to parallelize on modern eral thousand words via neural abstractive hardware like GPUs, limiting their scalability. hardware like GPUs, limiting their scalability. ⇤Equal contribution, order determined by coin flip ⇤Equal contribution, order determined by coin flip summarization. We perform a simple extrac- Preprint. Sent to peer review, May 2019. Preprint. Sent to peer review, May 2019. tive step before generating a summary, which is then used to condition the transformer lan- guage model on relevant information before being tasked with generating a summary. We also show that this approach produces more ab- stractive summaries compared to prior work that employs a copy mechanism while still achieving higher ROUGE scores. We pro- vide extensive comparisons with strong base- Figure 1: Our approach for abstractive summarization of a line methods, prior state of the art work as scientific article. An older version of this paper is shown as the well as multiple variants of our approach in- reference document. First, a sentence pointer network extracts important sentences from the paper. Next, these sentences are cluding those using only transformers, only provided along with the whole scientific article to be arranged extractive techniques and combinations of the in the following order: Introduction, extracted Sentences, ab- two. We examine these models using four dif- stract & the rest of the paper. A transformer language model ferent summarization tasks and datasets: arXiv is trained on articles organized in this format. During infer- papers, PubMed papers, the Newsroom and ence, the introduction and the extracted sentences are given to the language model as context to generate a summary. In BigPatent datasets. We find that transformer domains like news and patent documents, the introduction can based methods produce summaries with fewer be replaced by the entire document. n-gram copies, leading to n-gram copying techniques. Extractive summarization is the strat- statistics that are more similar to human gener- ated abstracts. We include a human evaluation, egy of selecting a subset of words, phrases or sen- finding that transformers are ranked highly for tences from the input document to form a sum- coherence and fluency, but purely extractive mary. Abstractive summarization consists of creat- methods score higher for informativeness and ing sentences summarizing content and capturing relevance. We hope that these architectures key ideas and elements of the source text, usually and experiments may serve as strong points of involving significant changes and paraphrases of 1 comparison for future work. text from the original source sentences. While ex- tractive summarization is able to preserve saliency, 1 Introduction the broader flow or coherency of the multiple sen- tences forming the summary