Scalable Question-Answer Generation from Text Corpus
Total Page：16
File Type：pdf, Size：1020Kb
Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus Bang Liu1, Haojie Wei2, Di Niu1, Haolan Chen2, Yancheng He2 1University of Alberta, Edmonton, AB, Canada 2Platform and Content Group, Tencent, Shenzhen, China ABSTRACT The fight scene finale between Sharon and the character played by Ali Larter, The ability to ask questions is important in both human and ma- from the movie Obsessed, won the 2010 MTV Movie Award for Best Fight. chine intelligence. Learning to ask questions helps knowledge acqui- Answer: MTV Movie Award for Best Fight sition, improves question-answering and machine reading compre- Clue: from the movie Obsessed Style: Which hension tasks, and helps a chatbot to keep the conversation flowing Q: A fight scene from the movie, Obsessed, won which award? with a human. Existing question generation models are ineffective Answer: MTV Movie Award for Best Fight at generating a large amount of high-quality question-answer pairs Clue: The flight scene finale between Sharon and the character played by from unstructured text, since given an answer and an input passage, Ali Larter question generation is inherently a one-to-many mapping. In this Style: Which Q: Which award did the fight scene between Sharon and the role of Ali paper, we propose Answer-Clue-Style-aware Question Generation Larter win? (ACS-QG), which aims at automatically generating high-quality and diverse question-answer pairs from unlabeled text corpus at scale Answer: Obsessed Clue: won the 2010 MTV Movie Award for Best Fight by imitating the way a human asks questions. Our system consists Style: What of: i) an information extractor, which samples from the text multiple Q: What is the name of the movie that won the 2010 MTV Movie Award types of assistive information to guide question generation; ii) neu- for Best Fight? ral question generators, which generate diverse and controllable Figure 1: Given the same input sentence, we can ask diverse questions, leveraging the extracted assistive information; and iii) questions based on the different choices about i) what the a neural quality controller, which removes low-quality generated target answer is; ii) which answer-related chunk is used as a data based on text entailment. We compare our question generation clue, and iii) what type of questions is asked. models with existing approaches and resort to voluntary human evaluation to assess the quality of the generated question-answer pairs. The evaluation results suggest that our system dramatically 1 INTRODUCTION outperforms state-of-the-art neural question generation models in Automatically generating question-answer pairs from unlabeled terms of the generation quality, while being scalable in the mean- text passages is of great value to many applications, such as as- time. With models trained on a relatively smaller amount of data, sisting the training of machine reading comprehension systems we can generate 2.8 million quality-assured question-answer pairs [10, 44, 45], generating queries/questions from documents to im- from a million sentences found in Wikipedia. prove search engines [17], training chatbots to get and keep a conversation going [40], generating exercises for educational pur- CCS CONCEPTS poses [7, 18, 19], and generating FAQs for web documents [25]. • Computing methodologies → Natural language process- Many question-answering tasks such as machine reading compre- ing; Natural language generation; Machine translation. hension and chatbots require a large amount of labeled samples KEYWORDS for supervised training, acquiring which is time-consuming and costly. Automatic question-answer generation makes it possible to Question Generation, Sequence-to-Sequence, Machine Reading arXiv:2002.00748v2 [cs.CL] 5 Mar 2020 provide these systems with scalable training data and to transfer Comprehension a pre-trained model to new domains that lack manually labeled ACM Reference Format: training samples. Bang Liu1, Haojie Wei2, Di Niu1, Haolan Chen2, Yancheng He2. 2020. Asking Despite a large number of studies on Neural Question Generation, Questions the Human Way: Scalable Question-Answer Generation from it remains a significant challenge to generate high-quality QA pairs Text Corpus. In Proceedings of The Web Conference 2020 (WWW ’20), April from unstructured text at large quantities. Most existing neural 20–24, 2020, Taipei, Taiwan. ACM, New York, NY, USA, 12 pages. https: question generation approaches try to solve the answer-aware //doi.org/10.1145/3366423.3380270 question generation problem, where an answer chunk and the surrounding passage are provided as an input to the model while This paper is published under the Creative Commons Attribution 4.0 International the output is the question to be generated. They formulate the (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution. task as a Sequence-to-Sequence (Seq2Seq) problem, and design WWW ’20, April 20–24, 2020, Taipei, Taiwan various encoder, decoder, and input features to improve the quality © 2020 IW3C2 (International World Wide Web Conference Committee), published of generated questions [10, 11, 22, 27, 39, 41, 53]. However, answer- under Creative Commons CC-BY 4.0 License. ACM ISBN 978-1-4503-7023-3/20/04. aware question generation models are far from sufficient, since https://doi.org/10.1145/3366423.3380270 question generation from a passage is inherently a one-to-many WWW ’20, April 20–24, 2020, Taipei, Taiwan Bang Liu1, Haojie Wei2, Di Niu1, Haolan Chen2, Yancheng He2 mapping. Figure 1 shows an example of this phenomenon. Given type mismatches and avoid meaningless combinations of assistive the same input text “The fight scene finale between Sharon and information. the character played by Ali Larter, from the movie Obsessed, won How to learn a model to ask ACS-aware questions? Most the 2010 MTV Movie Award for Best Fight.”, we can ask a variety existing neural approaches are designed for answer-aware question of questions based on it. If we select the text chunk “MTV Movie generation, while there is no training data available for the ACS- Award for Best Fight” as the answer, we can still ask different aware question generation task. We propose effective strategies questions such as “A fight scene from the movie, Obsessed, won to automatically construct training samples from existing reading which award?” or “Which award did the fight scene between Sharon comprehension datasets without any human labeling effort. We and the role of Ali Larter win?”. define “clue” as a semantic chunk in an input passage thatwill We argue that when a human asks a question based on a pas- be included (or rephrased) in the target question. Based on this sage, she will consider various factors. First, she will still select definition, we perform syntactic parsing and chunking on input text, an answer as a target that her question points to. Second, she will and select the chunk which is most relevant to the target question as decide which piece of information will be present (or rephrased) the clue. Furthermore, we categorize different questions into 9 styles, in her question to set constraints or context for the question. We including “what”, “how”, “yes-no” and so forth, In this manner, we call this piece of information as the clue. The target answer may have leveraged the abundance of reading comprehension datasets be related to different clues in the passage. Third, even the same to automatically construct training data for ACS-aware question question may be expressed in different styles (e.g., “what”, “who”, generation models. “why”, etc.). For example, one can ask “which award” or “what is We propose two deep neural network models for ACS-aware the name of the award” to express the same meaning. Once the an- question generation, and show their superior performance in gener- swer, clue, and question style are selected, the question generation ating diverse and high-quality questions. The first model employs process will be narrowed down and become closer to a one-to-one sequence-to-sequence framework with copy and attention mecha- mapping problem, essentially mimicking the human way of asking nism [1, 3, 43], incorporating the information of answer, clue and questions. In other words, introducing these pieces of information style into the encoder and decoder. Furthermore, it discriminates into question-answer generation can help reduce the difficulty of between content words and function words in the input, and uti- the task. lizes vocabulary reduction (which downsizes the vocabularies for In this paper, we propose Answer-Clue-Style-aware Question both the encoder and decoder) to encourage aggressive copying. Generation (ACS-QG) designed for scalable generation of high- In the second model, we fine-tune a GPT2-small model [34]. We quality question-answer pairs from unlabeled text corpus. Just as a train our ACS-aware QG models using the input passage, answer, human will ask a question with clue and style in mind, our system clue, and question style as the language modeling context. As a re- first automatically extracts multiple types of information froman sult, we reduce the phenomenon of repeating output words, which input passage to assist question generation. Based on the multi- usually exists with sequence-to-sequence models, and can gener- aspect information extracted, we design neural network models to ate questions with better readability. With multi-aspect assistive generate diverse questions in a controllable way. Compared with information, our models are able to ask a variety of high-quality existing answer-aware question generation, our approach essen- questions based on an input passage, while making the generation tially converts the one-to-many mapping problem into a one-to-one process controllable. mapping problem, and is thus scalable by varying the assistive infor- How to ensure the quality of generated QA pairs? We con- mation fed to the neural network while in the meantime ensuring struct a data filter, which consists of an entailment model anda generation quality.