Linguistically Informed Language Generation: A Multi-faceted Approach Dongyeop Kang CMU-LTI-20-003 Language Technologies Institute School of Computer Science Carnegie Mellon University 5000 Forbes Ave., Pittsburgh, PA 15213 www.lti.cs.cmu.edu Thesis Committee: Eduard Hovy (Carnegie Mellon University, Chair) Jeffrey Bigham (Carnegie Mellon University) Alan W Black (Carnegie Mellon University) Jason Weston (Facebook AI Research / New York University) Dan Jurafsky (Stanford University) Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy In Language and Information Technologies Copyright c 2020 Dongyeop Kang Keywords: Natural language generation, Systemic Functional Linguistics, knowledge aug- mentation, structure imposition, stylistic variation, neural-symbolic integration, text planning, cross-style analysis iv Abstract Natural language generation (NLG) is a key component of many language tech- nology applications such as dialogue systems, like Amazon’s Alexa; question an- swering systems, like IBM Watson; automatic email replies, like Google’s SmartRe- ply; and story generation. NLG is the process of converting computer-internal se- mantic representations of content into the correct form of a human language, such as English or Korean, so that the semantics are accurately included. One might think that the only information an NLG system would need to produce is that contained explicitly in the utterance. However, there is a multitude of implicit information not explicitly obvious on the surface. For instance, many different surface sentences have the same meaning but still have slightly different surface outputs. Several kinds of parameters seem to be reflected in variations of an utterance: external knowl- edge, goals, interpersonal information, speaker-internal information, and more. In this work, we call each individual input parameter a facet. To generate appropriate and natural utterances as humans do, appropriate modeling of these facets is neces- sary, and the system needs to be effectively guided by these facets. One of M. Halliday’s linguistic theories, called Systemic Functional Linguistics (SFL), suggests that such parameters could be categorized into three meta-functions, where each contains separate types of information relevant to aural and written com- munication. We choose three facets of interest, one for each SFL meta-function, and repackaged them into three facet groups: knowledge, structures, and styles. The knowledge facet decides the basic semantics of the topic to be communicated, while the structure facet coherently arranges information guiding the structure of the (multi-sentence) communication. Finally, the style facet represents all the other additional kinds of information that direct the formulation of interpersonal commu- nication. We assume that the three facets are, more or less, individual, and they dynami- cally interact with each other instead of being a sequential process. One can develop a human-like NLG system that effectively reflects the three facets of communica- tion and that simultaneously interact with each other, making a multifaceted system. In such systems, we believe that each facet of language has its own communicative goal, such that the knowledge facet is used to achieve factual goals, the structure facet is used to achieve coherence goals, and the style facet is used to achieve social goals. To show the necessity and effectiveness of the multifaceted system, we have developed several computing methods for each facet from the following questions: • Q1 “What” knowledge must be processed to make the model produce more factual text? • Q2 “How” can the model compose multiple sentences coherently? • Q3 How can the model produce stylistically appropriate output depending on “who” you are and “whom” you are talking to? vi Acknowledgments This thesis would not have been possible without the help and support of my advisor, thesis committee, mentors, friends, collaborators, faculty, and family. I would like to express my deepest appreciation to my advisor, Professor Eduard Hovy. When I first started graduate school at LTI, I embarrassingly thought language was textual data for testing fancy machine learning algorithms. He opened my eyes to a new world of computational linguistics, including how to understand the semantics of language, why humans communicate via language, and how to generate human- like language. Among hundreds of inspirational pieces of his advice, my favorite is “deep thinking, not only deep learning.” To me, he is the perfect example of a scholar. He thinks critically, is passionate about approaching problems, has the courage to delve into challenges, and is generous with his time. Although my time working with him resulted in solving only a tiny portion of the puzzles he set for me, I hope to continue relying on his wisdom to develop my career as a scientist. Professor Eduard Hovy, getting to know you has been the happiest and most precious part of my graduate study. Additionally, I would like to thank Professor Jeffrey Bigham, Professor Alan W Black, Jason Weston, and Professor Dan Jurafsky for agreeing to be on my thesis committee. Your valuable advice and feedback on the proposal, defense, and final version of the thesis has made me realize how truly fortunate I was to have you all on my thesis committee. Over the years of graduate school, I am grateful I could collaborate with amazing researchers at Microsoft Research, Allen Institute for AI, and Facebook AI Research. These experiences introduced me to many mentors: Patrick Pantel, Michael Gamon, Tushar Khot, Ashish Sabharwal, Peter Clark, Y-Lan Boureau, and Jason Weston. Your support and guidance helped broaden my perspective, both as it applies to re- search and the world in general. I would also like to thank my peers, including Ma- dian Khabsa, Waleed Ammar, Bhavana Dalvi, Roy Schwartz, Anusha Balakrishnan, Pararth Shah, and Paul Crook. I am extremely grateful to have made such amazing friends at CMU. Without your help, I would not have finished a single piece of work. I would particularly like to thank Hiroaki Hayashi and Dheeraj Rajagopal, who were always available whether I needed a random chat or to discuss research ideas. Whenever I felt down, you guys encouraged and motivated me again. I am also grateful to the Ed-visees members: Dheeraj Rajagopal, Varun Gangal, Naoki Otani, Evangelia Spiliopoulou, Hector Liu, Xuezhe Ma, Divyansh Kaushik, Qizhe Xie, Yohan Jo, and Maria Ryskina. I would also like to thank my friends, Hyeju Jang, Yohan Jo, Byungsoo Jeon, Kiry- ong Ha, Jay Lee, Seojin Bang, and Jinkyu Kim; you made me feel comfortable living in US. To my dear office mates, Volkan Cirik, Daniel Spokoyny, and Vidhisha Bal- achandran, thank you for sharing delicious snacks and providing helpful feedback. Yongjoon Kim and Junho Suh, my two oldest friends, thank you for treating me like a dumb high-school student. Your conversations helped me relax when I needed it the most. I would also like to thank the faculty members, collaborators, and administra- tive staff of LTI who helped me, including Professor Robert Frederking, Professor Graham Neubig, Professor Yulia Tsvetkov, Professor Teruko Mitamura, Professor Eunsu Kang, Stacey Young, Salvador Medina, Michael Miller, Shirley Hayati, Paul Michel, Artidoro Pagnoni, Shruti Palaskar, and many others whose names I do not have space to list. To my parents and sisters, thank you for your patience, endless love, and wisdom. It has been hard being apart for so long, but I sincerely appreciate everything you shared with me. I hope to soon fulfill my responsibilities as your son and brother. Finally, I would like to thank my wife, Taehee Jung. You made me realize what is most important in my life. Through the painful and hard times, you have kept me believing that going through the journey of life together will get us farther and happier than I ever could alone. Marrying you was the biggest achievement of my graduate life. viii Contents 1 Introduction 1 1.1 Natural Language Generation . 1 1.1.1 Case Studies . 2 1.1.2 Toward Human-like Language Generation: More Facets Needed . 5 1.1.3 Systemic Functional Linguistics (SFL) Theory . 7 1.2 A Multifaceted Approach . 9 1.2.1 Facets . 10 1.2.2 Facets-of-Interest: Knowledge, Structures, and Styles . 11 1.2.3 NLG Applications with the Three Facets . 12 1.2.4 NLG as a Multifaceted System . 14 1.2.5 Language generation is intended . 16 1.3 Research Questions and Technical Contributions . 17 1.3.1 Neural-symbolic integration for knowledge-augmented generation . 17 1.3.2 Text-planning for coherently-structured generation . 18 1.3.3 Cross-stylization for stylistically-appropriate generation . 19 1.4 Goal and Scope of Thesis . 20 2 Knowledge-Augmented Generation 21 2.1 Task: Open-ended Question Answering . 21 2.2 Proposed Approach: Neural-Symbolic Learning (NSL) . 23 2.3 Related Work . 25 2.4 Data-driven NSL: Adversarial Knowledge Augmentation . 26 2.4.1 Introduction . 26 2.4.2 Adversarial Example Generation . 27 2.4.3 Model Training . 33 2.4.4 Results . 34 2.4.5 Conclusion . 37 2.5 Embedding-driven NSL: Geometry Retrofitting . 38 2.5.1 Introduction . 38 2.5.2 Geometric Properties and Regularities . 39 2.5.3 Preliminary Analysis . 40 2.5.4 Proposed Method . 44 2.5.5 Results . 46 2.5.6 Conclusion . 51 ix 2.6 Model-driven NSL: Neural-Symbolic Module Integration . 51 2.6.1 Introduction . 51 2.6.2 Proposed Model . 53 2.6.3 Results . 56 2.6.4 Conclusion . 58 2.7 Conclusion . 59 3 Coherently-Structured Generation 61 3.1 Task: Multi-Sentence Text Generation . 61 3.2 Proposed Approach: Text Planning ......................... 64 3.3 Related Work . 66 3.4 Causal Planning for Causal Explanation Generation . 70 3.4.1 Introduction . 70 3.4.2 CSpikes: Temporal Causality Detection from Textual Features . 71 3.4.3 CGraph Construction . 74 3.4.4 Causal Reasoning .