Controllable Text Generation and Ethical Implications
Total Page：16
File Type：pdf, Size：1020Kb
Controllable Text Generation And Ethical Implications Shrimai Prabhumoye CMU-LTI-21-001 May 2021 Language Technologies Institute School of Computer Science Carnegie Mellon University 5000 Forbes Ave., Pisburgh, PA 15213 www.lti.cs.cmu.edu esis committee: Alan W Black (co-chair), Carnegie Mellon University Ruslan Salakhutdinov (co-chair), Carnegie Mellon University Yulia Tsvetkov, Carnegie Mellon University Jason Weston, Facebook AI Research Submied in partial fulllment of the requirements for the degree of Doctor of Philosophy In Language and Information Technologies ©2021, Shrimai Prabhumoye to my late mother, Sushma, for being an embodiment of perseverence and dedication, and for endless love that will last me a lifetime. i Abstract e 21st century is witnessing a major shi in the way people interact with technology and Natural Language Generation (NLG) is playing a central role. Users of smartphones and smart home devices now expect their gadgets to be aware of their situation, and to produce natural language outputs in interactions. is thesis identies three aspects of human communication to make machines sound human-like - style, content and structure. is thesis provides deep learning solutions to controlling these variables in neural text generation. I rst outline the various modules which could be manipulated to perform eective controllable text generation. I provide two novel solutions for style transfer – using back-translation technique, and tag and generate approach. I also introduce two new tasks for style transfer and provide datasets for further exploration – political slant transfer and politeness transfer. I establish the task of document grounded generation which leverages information from unstructured documents for the generation process. I introduce two new tasks for document grounded generation – Wikipedia Update generation and Document Grounded Dialogue Response generation. Fur- thermore, I build two new extensions to pre-trained encoder-decoder models to solve this task. I also design a new elegant solution for the sentence ordering task to learn eective document structures. For all three tasks of style transfer, document grounded generation and sentence order, I add importance to the human evaluation of the models. I introduce new human eval- uation measures for understanding the notion of grounding and for understanding the quality of predictions in sentence ordering. At the end, I provide a discussion on the ethical considera- tions of the applications of controllable text generation. Specically, I use deontological ethics to evaluate NLP systems and discuss how controllable text generation techniques can be used to make these systems ethical. Acknowledgements I would like to rst thank my advisors Alan W Black and Ruslan Salakhutdinov without whom this thesis would not have been possible. Meeting Alan Black was a serendipitous and a piv- otal event in my life and I am forever grateful for receiving his insights on not only scientic and technical maers but also about aairs of life. Alan, you are an embodiment of a “guru” from Indian culture; an advisor who has guided me through various walks of life - scientic, technical, professional, spiritual, emotional and personal. I am constantly inspired by your boundless knowledge, tireless spirit for hard work, insatiable thirst to learn new things and generous kindness. Russ is one of the most brilliant researchers of our time and I am extremely grateful that our paths intertwined. Russ, you have been an amazing advisor in every aspect; your vast knowledge, incredible humility, dedication for work, fascinating organization skills and passion for research constantly inspires me. Russ, your mentorship and insights have been central for my growth as a researcher; your exibility and patience has allowed me to explore interesting problems in domains I would not have ventured otherwise. I would like to thank my commiee members - Yulia Tsvetkov and Jason Weston. Yulia Tsvetkov has provided me with immeasurable support and guidance in the early years of my PhD and has also taught me to write research papers. I would like to thank Jason Weston for detailed feeback on my thesis which greatly improved it. I would like to thank my collaborators who made projects interesting and fun - Elijah Mayeld, Aman Madaan, Tanmay Parekh, Amrith Setlur, Dirk Hovy, Dheeraj Rajagopal, Brendon Bodlt, and Kangyan Zhou. I would also like to thank Stacey Young for working tirelessly behind the scenes for seamless processes at LTI. I have been fortunate to intern at multiple research labs and my internship work has added a lot of value to this thesis. I would like to thank my intern mentors - Michel Galley, Chris irk, Jason Weston, and Kazuma Hashimoto. I have been very lucky to have a supportive cohort of friends at Pisburgh. I would like to thank my friends for sharing the burden of my failures and celebrating the joy of my successes - Dheeraj Rajagopal, Vidhisha Balachandran, Shruti Palaskar, Venkat Perumal, Chaitanya Ahuja, Bhavya Balu, Aman Madaan, Bhuwan Dhingra, Rolly Mantri, Priyank Lathwal, Harsh Jhamtani and Sai Krishna Rallabandi. Finally, I would like to thank my family - my father Laxmikant, and my aunt Purnima for supporting me throughout this endeaver. I would like to thank my dear sister Diksha for being understanding and constantly supporting me through toughest times. Last but denitely not the least, I would like to thank my partner Ankush Das for being my biggest cheerleader throughout the process. Ankush, thank you for unconditional love and incessant support in everyday life, I love you. v Contents Abstract iii Acknowledgementsv 1 Introduction1 1.1 esis Statement...................................4 1.2 Overview.......................................4 2 Controllable Text Generation Techniques6 2.1 Generation Process.................................7 2.2 External Input....................................9 2.2.1 Arithmetic or Linear Transform......................9 2.2.2 Stochastic Changes............................. 10 2.2.3 Decompose................................. 10 2.2.4 External Feedback............................. 11 2.3 Sequential Input................................... 11 2.3.1 Arithmetic or Linear Transform...................... 12 2.4 Generator Operations................................ 12 2.4.1 Recurrent Neural Networks........................ 13 2.4.2 Transformer................................. 14 2.4.3 Pre-trained models............................. 15 2.5 Output........................................ 15 2.5.1 Aention.................................. 15 2.5.2 External Feedback............................. 16 2.5.3 Arithmetic or Linear Transform...................... 17 2.6 Training Objective.................................. 17 2.6.1 General Loss Objectives.......................... 17 2.6.2 KL Divergence............................... 18 2.6.3 Classier Loss................................ 18 vii Contents viii 2.6.4 Task Specic Loss.............................. 19 2.7 Discussion...................................... 20 2.8 Conclusion...................................... 21 3 Style Transfer 22 3.1 Tasks and Datasets................................. 23 3.1.1 Gender Transfer.............................. 24 3.1.2 Political Slant Transfer........................... 25 3.1.3 Sentiment Modication.......................... 25 3.1.4 Politeness Transfer............................. 26 3.2 Methodology..................................... 28 3.2.1 Back-translation.............................. 28 3.2.2 Tag and Generate.............................. 32 3.3 Experiments..................................... 36 3.3.1 Style Transfer Accuracy.......................... 36 3.3.2 Preservation of Meaning.......................... 38 3.3.3 Fluency................................... 40 3.3.4 Manual Inspection............................. 41 3.4 Related Work..................................... 43 3.4.1 Task..................................... 43 3.4.2 Methodology................................ 43 3.5 Conclusion...................................... 44 4 Document Grounded Generation 46 4.1 Tasks and Datasets................................. 49 4.1.1 Task Denition............................... 49 4.1.2 Wikipedia Update Generation....................... 50 4.1.3 Document Grounded Dialog Generation................. 52 4.2 Methodology..................................... 57 4.2.1 Generative models............................. 57 4.2.2 Extractive models.............................. 59 4.2.3 Pre-trained Encoder-Decoder Models................... 60 4.3 Experiments..................................... 62 4.3.1 Automated Evaluation........................... 62 4.3.2 Human Evaluations............................. 65 4.3.3 Manual Inspection............................. 68 4.4 Ethical Considerations............................... 72 4.5 Related Work..................................... 74 Contents ix 4.5.1 Task..................................... 74 4.5.2 Methodology................................ 76 4.6 Conclusion...................................... 76 5 Sentence Ordering 78 5.1 Methodology..................................... 79 5.1.1 Topological Sort.............................. 80 5.1.2 Constraint Learning............................ 80 5.2 Experiments..................................... 81 5.2.1 Datasets................................... 81 5.2.2 Baselines................................... 81 5.2.3 Evaluation Metric.............................. 82 5.3 Results.......................................