Interrogative-Word-Aware Question Generation

Let Me Know What to Ask: Interrogative-Word-Aware Question Generation Junmo Kang∗ Haritz Puerto San Roman∗ Sung-Hyon Myaeng School of Computing, KAIST Daejeon, Republic of Korea fjunmo.kang, haritzpuerto94, [email protected] Abstract Question Generation (QG) is a Natural Lan- guage Processing (NLP) task that aids advances in Question Answering (QA) and conversational assistants. Existing models focus on generating a question based on a text and possibly the answer to the generated question. They need to determine the type of interrogative word to be generated while having to pay attention to the grammar and vocabulary Figure 1: High-level overview of the proposed model. of the question. In this work, we propose Interrogative-Word-Aware Question Genera- ingful QG can play a key role in the advances of tion (IWAQG), a pipelined system composed of two modules: an interrogative word classi- QA (Lewis et al., 2019). fier and a QG model. The first module pre- QG is a difficult task due to the need for un- dicts the interrogative word that is provided derstanding of the text to ask about and generat- to the second module to create the question. ing a question that is grammatically correct and Owing to an increased recall of deciding the semantically adequate according to the given text. interrogative words to be used for the gener- This task is considered to have two parts: what ated questions, the proposed model achieves to ask and how to ask. The first one refers to new state-of-the-art results on the task of QG in SQuAD, improving from 46.58 to 47.69 in the identification of relevant portions of the text BLEU-1, 17.55 to 18.53 in BLEU-4, 21.24 to to ask about. This requires machine reading com- 22.33 in METEOR, and from 44.53 to 46.94 prehension since the system has to understand the in ROUGE-L. text. The latter refers to the creation of a natural language question that is grammatically cor- 1 Introduction rect and semantically precise. Most of the current approaches utilize sequence-to-sequence models, Question Generation (QG) is the task of creating composed of an encoder model that first trans- questions about a text in natural language. This forms a passage into a vector and a decoder model is an important task for Question Answering (QA) that given this vector, generates a question about since it can help create QA datasets. It is also use- the passage (Liu et al., 2019; Sun et al., 2018; ful for conversational systems like Amazon Alexa. Zhao et al., 2018; Pan et al., 2019). Due to the surge of interests in these systems, QG There are different settings for QG. Some au- is also drawing the attention of the research com- thors like (Subramanian et al., 2018) assumes that munity. One of the reasons for the fast advances only a passage is given, attempts to find candidate in QA capabilities is the creation of large datasets key phrases that represent the core of the questions like SQuAD (Rajpurkar et al., 2016) and TriviaQA to be created. Others follow an answer-aware set- (Joshi et al., 2017). Since the creation of such ting, where the input is a passage and the answer datasets is either costly if done manually or prone to the question to create (Zhao et al., 2018). We to error if done automatically, reliable and mean- assume this setting and consider that the answer ∗Equal contribution. is a span of the passage, as in SQuAD. Follow- 163 Proceedings of the Second Workshop on Machine Reading for Question Answering, pages 163–171 Hong Kong, China, November 4, 2019. c 2019 Association for Computational Linguistics ing this approach, the decoder of the sequence-to- is the first one to propose an sequence-to-sequence sequence model has to learn to generate both the model to tackle the QG problem and outperformed interrogative word (i.e., wh-word) and the rest of the previous state-of-the-art model using human the question simultaneously. and automatic evaluations. The main claim of our work is that separating Sun et al.(2018) proposed a similar approach to the two tasks (i.e., interrogative-word classifica- us, an answer-aware sequence-to-sequence model tion and question generation) can lead to a bet- with a special decoding mode in charge of only ter performance. We posit that the interrogative the interrogative word. However, we propose to word must be predicted by a well-trained classi- predict the interrogative word before the encoding fier. We consider that selecting the right inter- stage, so that the decoder can focus more on the rogative word is the key to generate high-quality rest of the question rather than on the interrogative questions. For example, a question with a wrong word. Besides, they cannot train the interrogative- interrogative word for the answer “the owner” word classifier using golden labels because it is is: “what produces a list of requirements for a learned implicitly inside the decoder. Duan et al. project?”. However, with the right interrogative (2017) proposed, in a similar way to us, a pipeline word, who, the question would be: “who produces approach. First, the authors create a long list of a list of requirements for a project?”, which is question templates like “who is author of”, and clear that is more adequate regarding the answer “who is wife of”. Then, when generating the ques- than the first one. According to our claim, the tion, they select first the question template and independent classification model can improve the next, they fill it in. To select the question template, recall of interrogative words of a QG model be- they proposed two approaches. One is a retrieval- cause 1) the interrogative word classification task based question pattern prediction, and the second is easier to solve than generating the interroga- one is a generation-based question pattern predic- tive word along with the full question in the QG tion. The first one has the problem that is com- model and 2) the QG model would be able to gen- putationally expensive when the question pattern erate the interrogative word easily by using the size is large, and the second one, although it yields copy mechanism, which can copy parts of the in- to better results, it is a generative approach and put of the encoder. With these hypotheses, we we argue that just modeling the interrogative word propose Interrogative-Word-Aware Question Gen- prediction as a classification task is easier and can eration (IWAQG), a pipelined system composed of lead to better results. As far as we know, we are two modules: an interrogative-word classifier that the first one to propose an explicit interrogative- predicts the interrogative word and a QG model word classifier that provides the interrogative word that generates a question conditioned on the pre- to the question generator. dicted interrogative word. Figure1 shows a high- level overview of our approach. 3 Interrogative-Word-Aware Question The proposed model achieves new state-of-the- Generation art results on the task of QG in SQuAD, improving 3.1 Problem Statement from 46.58 to 47.69 in BLEU-1, 17.55 to 18.53 in P A BLEU-4, 21.24 to 22.33 in METEOR, and from Given a passage , and an answer , we want to Q A 44.53 to 46.94 in ROUGE-L. find a question , whose answer is . More for- mally: 2 Related Work Q = arg max P rob(QjP; A) Question Generation (QG) problem has been ap- Q proached in two ways. One is based on heuristics, We assume that P is a paragraph composed of M templates and syntactic rules (Heilman and Smith, a list of words: P = fxtgt=1, and the answer is a 2010; Mazidi and Nielsen, 2014; Labutov et al., subspan of P . 2015). This type of approach requires a heavy hu- We model this problem with a pipelined ap- man effort, so they do not scale well. The other proach. First, given P and A, we predict the in- approach is based on neural networks and it is be- terrogative word Iw, and then, we input into QG coming popular due to the recent progress of deep module P , A, and Iw. The overall architecture of learning in NLP (Pan et al., 2019). Du et al.(2017) our model is shown in2. 164 Figure 2: Overall architecture of IWAQG. 3.2 Interrogative-Word Classifier are using [CLS] token embedding as a represen- tation of the context and the answer, we consider As discussed in section 5.2, any model can be used that using an explicit entity type embedding of the to predict interrogative words if its accuracy is answer could help the system. high enough. Our interrogative-word classifier is based on BERT, a state-of-the-art model in many 3.3 Question Generator NLP tasks that can successfully utilize the context For the QG module, we employ one of the current to grasp the semantics of the words inside a sen- state-of-the-art QG models (Zhao et al., 2018). tence (Devlin et al., 2018). We input a passage This model is a sequence-to-sequence neural net- that contains the answer of the question we want work that uses a gated self-attention in the encoder to build and add the special token [ANS] to let and an attention mechanism with maxout pointer BERT knows that the answer span has a special in the decoder.

Load more