Generating Question-Answer Hierarchies
Kalpesh Krishna & Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst {kalpesh,miyyer}@cs.umass.edu
http://squash.cs.umass.edu/
Abstract Massive Attack (band) Q. What was the iPhone application Fantom? On 21 January 2016, the iPhone A. The app... let users hear parts of ... real time, The process of knowledge acquisition can be application "Fantom" was released. The application was developed by Q. Who created it? a team including Massive Attack's A. ... team including ... Robert Del Naja viewed as a question-answer game between a Robert Del Naja and let users hear student and a teacher in which the student typi- parts of four new songs by Q. What is Ritual Spirit? remixing them in real time, using A. On ... Attack released a new EP Ritual Spirit cally starts by asking broad, open-ended ques- the phone's location, movement, clock, heartbeat, and camera. On Q. When did they release a song with tions before drilling down into specifics (Hin- 28 January 2016, Massive Attack "Fantom''? released a new EP, Ritual Spirit, A. On 28 January 2016, tikka, 1981; Hakkarainen and Sintonen, 2002). which includes the four songs ... This pedagogical perspective motivates a new released on Fantom. On 26 July 2016, Massive Attack way of representing documents. In this paper, previewed three new songs: "Come Q. What did they do in 2016? Near Me", "The Spoils", and "Dear A. On ... 2016 ... three new songs: "Come Near we present SQUASH (Specificity-controlled Friend" on Fantom, an iPhone Me", "The Spoils", and "Dear Friend" on Fantom, application on which they Question-Answer Hierarchies), a novel and previously previewed the four Q. Who was in the video? challenging text generation task that con- songs from the Ritual Spirit EP ... A. The video ... featuring Cate Blanchett, The video for "The Spoils", Q. Who was the Australian director? verts an input document into a hierarchy of featuring Cate Blanchett, and A. John Hillcoat, directed by Australian director John ... question-answer pairs. Users can click on Hillcoat, ... high-level questions (e.g., “Why did Frodo leave the Fellowship?”) to reveal related but Figure 1: A subset of the QA hierarchy generated by more specific questions (e.g., “Who did Frodo our SQUASH system that consists of GENERAL and leave with?”). Using a question taxonomy SPECIFIC questions with extractive answers. loosely based on Lehnert(1978), we classify questions in existing reading comprehension to reveal more fine-grained related information. datasets as either GENERAL or SPECIFIC. We We call our task SQUASH (Specificity-controlled then use these labels as input to a pipelined Question Answer Hierarchies). system centered around a conditional neu- ral language model. We extensively evaluate Q: Why represent a document with QA pairs?1 the quality of the generated QA hierarchies A: Questions and answers (QA) play a critical through crowdsourced experiments and report role in scientific inquiry, information-seeking dia- strong empirical results. logue and knowledge acquisition (Hintikka, 1981, 1 Introduction 1988; Stede and Schlangen, 2004). For example, web users often use QA pairs to manage and share Q: What is this paper about? knowledge (Wagner, 2004; Wagner and Bolloju, arXiv:1906.02622v2 [cs.CL] 21 Jul 2019 A: We present a novel text generation task 2005; Gruber, 2008). Additionally, unstructured which converts an input document into a model- lists of “frequently asked questions” (FAQs) are generated hierarchy of question-answer (QA) regularly deployed at scale to present information. pairs arranged in a top-down tree structure (Fig- Industry studies have demonstrated their effective- ure 1). Questions at higher levels of the tree ness at cutting costs associated with answering are broad and open-ended while questions at customer calls or hiring technical experts (Daven- lower levels ask about more specific factoids. An port et al., 1998). Automating the generation of entire document has multiple root nodes (“key QA pairs can thus be of immense value to compa- ideas”) that unfold into a forest of question trees. nies and web communities. While readers are initially shown only the root 1Our introduction is itself an example of the QA format. nodes of the question trees, they can “browse” the Other academic papers such as Henderson et al.(2018) have document by clicking on root nodes of interest also used this format to effectively present information. Q: Why add hierarchical structure to QA pairs? Q: How can the community build on this work? A: While unstructured FAQs are useful, ped- A: We have released our codebase, dataset and agogical applications benefit from additional a live demonstration of our system at http:// hierarchical organization. Hakkarainen and squash.cs.umass.edu/. Additionally, we Sintonen(2002) show that students learn concepts outline guidelines for future work in Section 7. effectively by first asking general, explanation- seeking questions before drilling down into more 2 Obtaining training data for SQUASH specific questions. More generally, hierarchies The proliferation of reading comprehension break up content into smaller, more digestable datasets like SQuAD (Rajpurkar et al., 2016, chunks. User studies demonstrate a strong 2018) has enabled state-of-the-art neural question preference for hierarchies in document summa- generation systems (Du et al., 2017; Kim et al., rization (Buyukkokten et al., 2001; Christensen 2018). However, these systems are trained for et al., 2014) since they help readers easily identify individual question generation, while the goal of and explore key topics (Zhang et al., 2017). SQUASH is to produce a general-to-specific hier- Q: How do we build systems for SQUASH? archy of QA pairs. Recently-released conversa- A: We leverage the abundance of reading compre- tional QA datasets like QuAC (Choi et al., 2018) hension QA datasets to train a pipelined system and CoQA (Reddy et al., 2018) contain a sequen- for . One major challenge is the lack tial arrangement of QA pairs, but question speci- SQUASH 2 of labeled hierarchical structure within existing ficity is not explicitly marked. Motivated by the QA datasets; we tackle this issue in Section 2 by lack of hierarchical QA datasets, we automatically using the question taxonomy of Lehnert(1978) classify questions in SQuAD, QuAC and CoQA to classify questions in these datasets as either according to their specificity using a combination GENERAL or SPECIFIC. We then condition a of rule-based and automatic approaches. neural question generation system on these two 2.1 Rules for specificity classification classes, which enables us to generate both types of questions from a paragraph. We filter and What makes one question more specific than structure these outputs using the techniques another? Our scheme for classifying question described in Section 3. specificity maps each of the 13 conceptual ques- tion categories defined by Lehnert(1978) to three Q: How do we evaluate our SQUASH pipeline? coarser labels: GENERAL, SPECIFIC, or YES-NO.3 A: Our crowdsourced evaluation (Section 4) As a result of this mapping, SPECIFIC questions focuses on fundamental properties of the gener- usually ask for low-level information (e.g., en- ated output such as QA quality, relevance, and tities or numerics), while GENERAL questions hierarchical correctness. Our work is a first step ask for broader overviews (e.g., “what happened towards integrating QA generation into document in 1999?”) or causal information (e.g, “why understanding; as such, we do not directly evalu- did...”). Many question categories can be reliably ate how useful SQUASH output is for downstream identified using simple templates and rules; A pedagogical applications. Instead, a detailed qual- complete list is provided in Table 1.4 itative analysis (Section 5) identifies challenges Classifying questions not covered by templates: that need to be addressed before SQUASH can be deployed to real users. If a question does not satisfy any template or rule, how do we assign it a label? We manage to clas- Q: What are our main contributions? sify roughly half of all questions with our tem- A1: A method to classify questions according to 2“Teachers” in the QuAC set-up can encourage “students” their specificity based on Lehnert(1978). to ask a follow-up question, but we cannot use these annota- A2: A model controlling specificity of generated tions to infer a hierarchy because students are not required to questions, unlike prior work on QA generation. actually follow their teachers’ directions. 3We add a third category for YES-NO questions as they A3: A novel text generation task (SQUASH), are difficult to classify as either GENERAL or SPECIFIC. which converts documents into specificity-based 4Questions in Lehnert(1978) were classified using a con- hierarchies of QA pairs. ceptual dependency parser (Schank, 1972). We could not find a modern implementation of this parser and thus decided to A4: A pipelined system to tackle SQUASH along use a rule-based approach that relies on spaCy 2.0 (Honnibal with crowdsourced methods to evaluate it. and Montani, 2017) for all preprocessing. Conceptual class Specificity Question asks for... Sample templates
Causal Antecedent, Goal Oriented, GENERAL the reason for occurrence Why ..., What happened after / be- Enablement, Causal Consequent, of an event and the conse- fore ..., What was the cause / rea- Expectational quences of it son / purpose ..., What led to ...
Instrumental GENERAL a procedure / mechanism How question with VERB parent for How in dependency tree
Judgemental GENERAL a listener’s opinion Words like you, your present
Concept Completion, Feature Speci- GENERAL or fill-in-the-blank informa- Where / When / Who ... fication SPECIFIC tion (“SPECIFIC” templates)
Quantification SPECIFIC an amount How many / long ...
Verification, Disjunctive YES-NO Yes-No answers first word is VERB Request N/A an act to be performed (absent in datasets)
Table 1: The 13 conceptual categories of Lehnert(1978) and some templates to identify them and their specificity. plates and rules (Table A1); for the remaining a high level, the pipeline consists of five steps: half, we resort to a data-driven approach. First, (1) answer span selection, (2) question genera- we manually label 1000 questions in QuAC5 us- tion conditioned on answer spans and specificity ing our specificity labels. This annotated data labels, (3) extractively answering generated ques- is then fed to a single-layer CNN binary classi- tions, (4) filtering out bad QA pairs, and (5) struc- fier (Kim, 2014) using ELMo contextualized em- turing the remaining pairs into a GENERAL-to- beddings (Peters et al., 2018).6 On a 85%-15% SPECIFIC hierarchy. The remainder of this sec- train-validation split, we achieve a high classifica- tion describes each step in more detail and after- tion accuracy of 91%. The classifier also trans- wards explains how we leverage pretrained lan- fers to other datasets: on 100 manually labeled guage models to improve individual components CoQA questions, we achieve a classification accu- of the pipeline. racy of 80%. To obtain our final dataset (Table 2), we run our rule-based approach on all questions 3.1 Answer span selection in SQuAD 2.0, QuAC, and CoQA and apply our Our pipeline begins by selecting an answer span classifier to label questions that were not covered from which to generate a question. To train the by the rules. We further evaluate the specificity of system, we can use ground-truth answer spans the questions generated by our final system using from our labeled datasets, but at test time how do a crowdsourced study in Section 4.3. we select answer spans? Our solution is to con- sider all individual sentences in the input para- Dataset Size GENERAL SPECIFIC YES-NO graph as potential answer spans (to generate GEN- SQuAD 86.8k 28.2% 69.7% 2.1% ERAL and SPECIFIC questions), along with all en- QuAC 65.2k 34.9% 33.5% 31.6% tities and numerics (for just SPECIFIC questions). CoQA 105.6k 23.6% 54.9% 21.5% All 257.6k 28.0% 54.5% 17.5% We did not use data-driven sequence tagging ap- proaches like previous work (Du and Cardie, 2017, Table 2: Distribution of classes in the final datasets. We 2018), since our preliminary experiments with add some analysis on this distribution in Appendix A. such approaches yielded poor results on QuAC.7 More details are provided in Appendix C. 3 A pipeline for SQUASHing documents 3.2 Conditional question generation To SQUASH documents, we build a pipelined sys- tem (Figure 2) that takes a single paragraph as in- Given a paragraph, answer span, and desired put and produces a hierarchy of QA pairs as out- specificity label, we train a neural encoder- put; for multi-paragraph documents, we SQUASH decoder model on all three reading comprehension each paragraph independently of the rest. At datasets (SQuAD, QuAC and CoQA) to generate an appropriate question. 5We use QuAC because its design encourages a higher percentage of GENERAL questions than other datasets, as the question-asker was unable to read the document to formulate 7We hypothesize that answer span identification on QuAC more specific questions. is difficult because the task design encouraged “teachers” to 6Implemented in AllenNLP (Gardner et al., 2018). provide more information than just the minimal answer span. Span x13 x13 Span Question questions Question QA QA (Similar Document Selection Class Generation Answering Filtering Pipeline)
Yes Training Data GENERAL SPECIFIC SQUASH RC Does Q match No Specificity Building QA Hierarchy Datasets template? Classifier Output
Figure 2: An overview of the process by which we generate a pair of GENERAL-SPECIFIC questions , which consists of feeding input data( “RC” is Reading Comprehension) through various modules, including a question classifier and a multi-stage pipeline for question generation, answering, and filtering.
Data preprocessing: At training time, we use Test time usage: At test time, the question gen- the ground-truth answer spans from these datasets eration module is supplied with answer spans and as input to the question generator. To improve class labels as described in Section 3.1. To pro- the quality of SPECIFIC questions generated from mote diversity, we over-generate prospective can- sentence spans, we use the extractive evidence didates (Heilman and Smith, 2010) for every an- spans for CoQA instances (Reddy et al., 2018) swer span and later prune them. Specifically, we instead of the shorter, partially abstractive answer use beam search with a beam size of 3 to gener- spans (Yatskar, 2019). In all datasets, we remove ate three highly-probable question candidates. As unanswerable questions and questions whose these candidates are often generic, we additionally answers span multiple paragraphs. A few very use top-k random sampling (Fan et al., 2018) with generic questions (e.g. “what happened in this k = 10, a recently-proposed diversity-promoting article?”) were manually identified removed from decoding algorithm, to generate ten more ques- the training dataset. Some other questions (e.g., tion candidates per answer span. Hence, for every “where was he born?”) are duplicated many times answer span we generate 13 question candidates. in the dataset; we downsample such questions to We discuss issues with using just standard beam a maximum limit of 10. Finally, we preprocess search for question generation in Section 5.1. both paragraphs and questions using byte-pair encoding (Sennrich et al., 2016). 3.3 Answering generated questions Architecture details: We use a two-layer biL- While we condition our question generation model STM encoder and a single-layer LSTM (Hochre- on pre-selected answer spans, the generated ques- iter and Schmidhuber, 1997) decoder with soft tions may not always correspond to these input attention (Bahdanau et al., 2015) to generate spans. Sometimes, the generated questions are ei- questions, similar to Du et al.(2017). Our ther unanswerable or answered by a different span architecture is augmented with a copy mecha- in the paragraph. By running a pretrained QA nism (See et al., 2017) over the encoded paragraph model over the generated questions, we can detect representations. Answer spans are marked with questions whose answers do not match their orig-
Table 3: Human evaluations demonstrate the high individual QA quality of our pipeline’s outputs. All interanno- tator agreement scores (Fleiss κ) show “fair” to “substantial” agreement (Landis and Koch, 1977).
4 Evaluation (with an equal split between GENERAL and SPE- CIFIC questions). The first row of Table 3 shows We evaluate our SQUASH pipeline on documents that 85.8% of generated questions satisfy this cri- from the QuAC development set using a vari- terion with a high agreement across workers. ety of crowdsourced13 experiments. Concretely, Question relevance: How many generated ques- we evaluate the quality and relevance of individ- tions are actually relevant to the input paragraph? ual questions, the relationship between generated While the percentage of unanswerable questions questions and predicted answers, and the struc- that were generated offers some insight into this tural properties of the QA hierarchy. We empha- question, we removed all of them during the filter- size that our experiments examine only the quality ing pipeline (Section 3.4). Hence, we display an of a ed document, not its actual useful- SQUASH input paragraph and generated question to crowd ness to downstream users. Evaluating usefulness workers (using the same data as the previous well- (e.g., measuring if is more helpful than SQUASH formedness evaluation) and ask whether or not the input document) requires systematic and tar- the paragraph contains the answer to the question. geted human studies (Buyukkokten et al., 2001) The second row of Table 3 shows that 78.7% of that are beyond the scope of this work. our questions are relevant to the paragraph, com- 4.1 Individual question quality and relevance pared to 83.3% of gold questions.
Our first evaluation measures whether questions 4.2 Individual answer validity generated by our system are well-formed (i.e., grammatical and pragmatic). We ask crowd work- Is the predicted answer actually a valid answer ers whether or not a given question is both gram- to the generated question? In our filtering pro- matical and meaningful.14 For this evaluation, we cess, we automatically measured answer overlap acquire judgments for 200 generated QA pairs and between the input answer span and the predicted 100 gold QA pairs15 from the QuAC validation set answer span and used the results to remove low- overlap QA pairs. To evaluate answer recall af- 13All our crowdsourced experiments were conducted on ter filtering, we perform a crowdsourced evalua- the Figure Eight platform with three annotators per example (scores calculated by counting examples with two or more tion on the same 300 QA pairs as above by asking correct judgments). We hired annotators from predominantly crowdworkers whether or not a predicted answer English-speaking countries with a rating of at least Level 2, span contains the answer to the question. We also and we paid them between 3 and 4 cents per judgment. 14As “meaningful” is potentially a confusing term for experiment with a more relaxed variant (partially crowd workers, we ran another experiment asking only for contains instead of completely contains) and re- grammatical correctness and achieved very similar results. port results for both task designs in the third and 15Results on this experiment were computed after remov- ing 3 duplicate generated questions and 10 duplicate gold fourth rows of Table 3. Over 85% of predicted questions. spans partially contain the answer to the gener- Cowell formed a new company Returning home to Brantford After five years, however, Tan Dun earned widespread From 1969 to 1971, Cash Syco, which is divided into three after six months abroad, Bell Limon would return to Broadway attention after composing the starred in his own television units - Syco Music, Syco TV and continued experiments with his to star as a featured dancer in score for Ang Lee's Crouching show, The Johnny Cash Show, Syco Film. Cowell returned to "harmonic telegraph". The basic Keep Off the Grass under the Tiger, Hidden Dragon (2000), for on the ABC network. The show music with his latest brainchild concept behind his device choreographer George which he won an Academy was performed at the Ryman signed to Syco ... was that messages could ... Balanchine. Award, a Grammy Award .... Auditorium in Nashville. ...
What was Bell's Why did he return to How was Tan Dun What is Syco? What did he do in 1969? telegraph? Broadway? received? How many units Where did he take Who did he work What award did he What network was does Syco have? his experiments? with? win? he in?
Figure 4: SQUASH question hierarchies generated by our system with reference snippets . Questions in the hier- archy are of the correct specificity class (i.e., GENERAL , SPECIFIC ). ated question, and this number increases if we con- together as described in Section 3.5. For each sider only questions that were previously labeled pair of questions (without showing answers), we as well-formed and relevant. The lower gold per- ask crowd workers to choose the question which, formance is due to the contextual nature of the if answered, would give them more information gold QA pairs in QuAC, which causes some ques- about the paragraph. As shown in Table 4, tions to be meaningless in isolation (e.g.“What did in 89.5% instances the GENERAL question is she do next?” has unresolvable coreferences). preferred over the SPECIFIC one, which confirms the strength of our specificity-controlled question Experiment Score Fleiss κ generation system.18 Which question type asks for 89.5% 0.57 more information? How related are SPECIFIC questions to their
Which SPECIFIC question is parent GENERAL question? Finally, we inves- closer to GENERAL QA? tigate the effectiveness of our question grouping different paragraph 77.0% 0.47 strategy, which bins multiple SPECIFIC QA pairs same paragraph 64.0% 0.30 under a single GENERAL QA pair. We show crowd Table 4: Human evaluation of the structural correctness workers a reference GENERAL QA pair and ask of our system. The labels “different / same paragraph” them to choose the most related SPECIFIC ques- refer to the location of the intruder question. The re- tion given two choices, one of which is the sys- sults show the accuracy of specificity and hierarchies. tem’s output and the other an intruder question. 4.3 Structural correctness We randomly select intruder SPECIFIC questions from either a different paragraph within the same To examine the hierachical structure of SQUASH document or a different group within the same ed documents, we conduct three experiments. paragraph. As shown in Table 4, crowd workers How faithful are output questions to input prefer the system’s generated SPECIFIC question specificity? First, we investigate whether our with higher than random chance (50%) regardless model is actually generating questions with the of where the intruder comes from. As expected, correct specificity label. We run our specificity the preference and agreement is higher when in- classifier (Section 2) over 400 randomly sampled truder questions come from different paragraphs, questions (50% GENERAL, 50% SPECIFIC) and since groups within the same paragraph often con- obtain a high classification accuracy of 91%.16 tain related information (Section 5.2). This automatic evaluation suggests the model is capable of generating different types of questions. 5 Qualitative Analysis Are GENERAL questions more representative of a paragraph than SPECIFIC questions? In this section we analyze outputs (Figure 4, Fig- To see if GENERAL questions really do provide ure 5) of our pipeline and identify its strengths and more high-level information, we sample 200 weaknesses. We additionally provide more exam- GENERAL-SPECIFIC question pairs17 grouped ples in the appendix (Figure A1).
16Accuracy computed after removing 19 duplicates. led to sparse hierarchical structures which were not favored 17We avoid gold-standard control experiments for struc- by our crowd workers. tural correctness tests since questions in the QuAC dataset 18We also ran a pilot study asking workers “Which ques- were not generated with a hierarchical structure in mind. Pi- tion has a longer answer?” and observed a higher preference lot studies using our question grouping module on gold data of 98.6% for GENERAL questions. William Bryan Paul Weston 5.2 What kind of mistakes does it make? Weston was born Paul Wetstein in Springfield, ...The treaty granted the United States control of Massachusetts, to Paul Wetstein, a teacher, and Puerto Rico, Guam, Cuba, the Philippines, and Anna "Annie" Grady. The family moved to parts of the West Indies. Many of Bryan's Pittsfield when Weston was two, and he spent We describe the various types of errors our model supporters were opposed to what they perceived his formative years in the town. His parents were as Republican aspirations of turning the country both interested in music, and when Paul Sr into an imperial power ... However, when the makes in this section, using the Paul Weston taught at a private girls' school, he was allowed Bacon Resolution (a proposed supplement to the to bring the school's gramophone ... Treaty of Paris which would allow the Filipinos a SQUASH output in Figure 5 as a running example. "stable and independent government") failed to Q. What are his parents like? pass, Bryan began publicly speaking out against the Republicans' imperial aspirations. A. Paul Wetstein, a teacher, and Anna Additionally, we list some modeling approaches "Annie" Grady. Q. What was the treaty? Q. Who was born in Springfield? we tried that did not work in Appendix C. A. The treaty granted the United States A. Weston was born Paul Wetstein control of Puerto Rico, Guam, Cuba, the in Springfield, Massachusetts, to Philippines, and parts of the West Indies. Paul Wetstein, a teacher, and Reliance on a flawed answering system: Our Anna "Annie" Grady. Q. Where did the Treaty of Paris pipeline’s output is tied to the quality of the pre- come from? Q. Where was Weston born? A. The treaty granted the United A. Springfield, Massachusetts, States control of Puerto Rico, Q. Who were his parents? trained answering module, which both filters out Guam, Cuba, the Philippines, and A. Paul Wetstein, a teacher, and parts of the West Indies. Anna "Annie" Grady. questions and produces final answers. QuAC has Q. Why was this bad? Q. Where did he move to? long answer spans (Choi et al., 2018) that cause A. Many of Bryan's supporters were A. The family moved to Pittsfield opposed to what they perceived as Q. How old was Weston when he low-precision predictions with extra information Republican aspirations of turning the was born? country into an imperial power A. two (e.g., “Who was born in Springfield?”). Addi- Q. What was a result of the resolution? Q. How did he get into music? A. failed to pass, Bryan began publicly A. His parents were both interested in tionally, the answering module occasionally swaps speaking out against the Republicans' music, and when Paul Sr taught at a private 19 imperial aspirations. girls' school, two named entities present in the paragraph. Q. Where did he go to school? A. Paul Sr taught at a private girls' school, Redundant information and lack of discourse: In our system, each QA pair is generated indepen- Figure 5: Two SQUASH outputs generated by our dently of all the others. Hence, our outputs lack system. The William Bryan example has interest- an inter-question discourse structure. Our system ing GENERAL questions. The Paul Weston example often produces a pair of redundant SPECIFIC ques- showcases several mistakes our model makes. tions where the text of one question answers the other (e.g., “Who was born in Springfield?” vs. “In 1942, Dodds enlisted in the US army and served as “Where was Weston born?”). These errors can an anti aircraft gunner during World War II.” likely be corrected by conditioning the generation In what year did the US army take place? module on previously-produced questions (or ad- B In what year did the US army take over? In what year did the US army take place in the US? ditional filtering); we leave this to future work. What year was he enlisted? Lack of world knowledge: Our models lack T When did he go to war? When did he play as anti aircraft? commonsense knowledge (“How old was Weston when he was born?”) and can misinterpret polyse- Table 5: Beam Search (B) vs Top-k sampling (T) mous words. Integrating pretrained contextualized for SPECIFIC question generation. Top-k candidates embeddings (Peters et al., 2018) into our pipeline tend to be more diverse. is one potential solution. Multiple GENERAL QA per paragraph: Our system often produces more than one tree per 5.1 What is our pipeline good at? paragraph, which is undesirable for short, focused paragraphs with a single topic sentence. To im- Meaningful hierarchies: Our method of group- prove the user experience, it might be ideal to re- ing the generated questions (Section 3.5) produces strict the number of GENERAL questions we show hierarchies that clearly distinguish between GEN- per paragraph. While we found it difficult to ERAL and SPECIFIC questions; Figure 4 contains generate GENERAL questions representative of en- some hierarchies that support the positive results tire paragraphs (Appendix C), a potential solution of our crowdsourced evaluation. could involve identifying and generating questions from topic sentences. Top-k sampling: Similar to prior work (Fan Coreferences in GENERAL questions: Many et al., 2018; Holtzman et al., 2019), we notice generated GENERAL questions contain corefer- that beam search often produces generic or repet- ences due to contextual nature of the QuAC itive beams (Table 5). Even though the top-k scheme always produces lower-probable questions 19For instance in the sentence “The Carpenter siblings were born in New Haven, to Harold B. and Agnes R.” the than beam search, our filtering system prefers a model incorrectly answers the question “Who was born in top-k question 49.5% of the time. New Haven?” as “Harold B. and Agnes R.” and CoQA training data (“How did he get into Evaluation of the SQUASH format: As dis- music?”). Potential solutions could involve ei- cussed in Section 1, previous research shows sup- ther constrained decoding to avoid beams with port for the usefulness of hierarchies and QA in anaphoric expressions or using the CorefNQG pedagogical applications. We did not directly model of Du and Cardie(2018). evaluate this claim in the context of SQUASH, fo- cusing instead on evaluating the quality of QA 5.3 Which models did not work? pairs and their hierarchies. Moving forward, care- We present modelling approaches which did not ful user studies are needed to evaluate the efficacy work in Appendix C. This includes, i) end-to- of the SQUASH format in pedagogical applica- end modelling to generate sequences of questions tions, which might be heavily domain-dependent; using QuAC, ii) span selection NER system, iii) for example, a QA hierarchy for a research paper generation of GENERAL questions representative is likely to be more useful to an end user than a of entire paragraphs, iv) answering system trained QA hierarchy for an online blog. An important on the combination of QuAC, CoQA and SQuAD. caveat is the imperfection of modern text gener- ation systems, which might cause users to pre- 6 Related Work fer the original human-written document over a generated SQUASH output. One possible solu- Our work on SQUASH is related to research in tion is a three-way comparison between the origi- three broad areas: question generation, informa- nal document, a human-written ed doc- tion retrieval and summarization. SQUASH ument, and a system-generated output. For fair Question Generation: Our work builds upon comparison, care should be taken to prevent exper- neural question generation systems (Du et al., imenter bias while crowdsourcing QA hierarchies 2017; Du and Cardie, 2018). Our work conditions (e.g., by maintaining similar text complexity in the generation on specificity, similar to difficulty- two human-written formats). conditioned question generation (Gao et al., 2018). QA pair generation has previously been used for dataset creation (Serban et al., 2016; Du and Collection of a SQUASH dataset: Besides mea- Cardie, 2018). Joint modeling of question genera- suring the usefulness of the QA hierarchies, a tion and answering has improved the performance large dedicated dataset can help to facilitate end- of individual components (Tang et al., 2017; Wang to-end modeling. While asking human annotators et al., 2017; Sachan and Xing, 2018) and enabled to write full SQUASHed documents will be ex- visual dialog generation (Jain et al., 2018). pensive, a more practical option is to ask them to pair GENERAL and SPECIFIC questions in our Information Retrieval: Our hierarchies are re- dataset to form meaningful hierarchies and write lated to interactive retrieval setting (Hardtke et al., extra questions whenever no such pair exists. 2009; Brandt et al., 2011) where similar webpages are grouped together. SQUASH is also related to exploratory (Marchionini, 2006) and faceted QA budget and deeper specificity hierarchies: search (Yee et al., 2003). In our work, we generate questions for every sen- tence and filter bad questions with fixed thresh- Summarization: Our work is related to query- olds. An alternative formulation is an adaptive focused summarization (Dang, 2005; Baumel model dependent on a user-specified QA budget, et al., 2018) which conditions an output summary akin to “target length” in summarization systems, on an input query. Hierarchies have also been ap- which would allow end users to balance cover- plied to summarization (Christensen et al., 2014; age and brevity themselves. A related modifi- Zhang et al., 2017; Tauchmann et al., 2018). cation is increasing the depth of the hierarchies. 7 Future Work While two-level QA trees are likely sufficient for documents structured into short and focused para- While Section 5.2 focused on shortcomings in our graphs, deeper hierarchies can be useful for long modeling process and steps to fix them, this sec- unstructured chunks of text. Users can control this tion focuses on broader guidelines for future work property via a “maximum children per QA node” involving the SQUASH format and its associated hyperparameter, which along with the QA budget text generation task. will determine the final depth of the hierarchy. 8 Conclusion Janara Christensen, Stephen Soderland, Gagan Bansal, et al. 2014. Hierarchical summarization: Scaling up We propose SQUASH, a novel text generation task multi-document summarization. In Proc. Associa- which converts a document into a hierarchy of QA tion for Computational Linguistics (ACL). pairs. We present and evaluate a system which Hoa Trang Dang. 2005. Overview of duc 2005. In leverages existing reading comprehension datasets Document Understanding Conferences. to attempt solving this task. We believe SQUASH is a challenging text generation task and we hope Thomas H Davenport, David W De Long, and Michael C Beers. 1998. Successful knowledge the community finds it useful to benchmark sys- management projects. Sloan management review, tems built for document understanding, question 39(2):43–57. generation and question answering. Additionally, Jacob Devlin, Ming-Wei Chang, Kenton Lee, and we hope that our specificity-labeled reading com- Kristina Toutanova. 2019. Bert: Pre-training of deep prehension dataset is useful in other applications bidirectional transformers for language understand- such as 1) finer control over question generation ing. In Proc. Conference of the North American systems used in education applications, curiosity- Chapter of the Association for Computational Lin- guistics – Human Language Technologies (NAACL driven chatbots and healthcare (Du et al., 2017). HLT). Acknowledgements Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural ques- We thank the anonymous reviewers for their in- tion generation. In Proc. Conference on Em- sightful comments. In addition, we thank Nader pirical Methods in Natural Language Processing (EMNLP) Akoury, Ari Kobren, Tu Vu and the other members . of the UMass NLP group for helpful comments on Xinya Du and Claire Cardie. 2018. Harvest- earlier drafts of the paper and suggestions on the ing paragraph-level question-answer pairs from paper’s presentation. This work was supported in wikipedia. In Proc. Association for Computational Linguistics (ACL). part by research awards from the Allen Institute for Artificial Intelligence and Adobe Research. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing to ask: Neural question generation for reading comprehension. In Proc. Association for Computa- References tional Linguistics (ACL). Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hi- gio. 2015. Neural machine translation by jointly erarchical neural story generation. In Proc. Associ- learning to align and translate. In Proc. Inter- ation for Computational Linguistics (ACL). national Conference on Learning Representations Yifan Gao, Jianan Wang, Lidong Bing, Irwin King, and (ICLR). Michael R Lyu. 2018. Difficulty controllable ques- tion generation for reading comprehension. arXiv Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. preprint arXiv:1807.03586. Query focused abstractive summarization: Incorpo- rating query relevance, multi-document coverage, Matt Gardner, Joel Grus, Mark Neumann, Oyvind and summary length constraints into seq2seq mod- Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Pe- els. arXiv preprint arXiv:1801.07704. ters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language pro- Christina Brandt, Thorsten Joachims, Yisong Yue, and cessing platform. arXiv preprint arXiv:1803.07640. Jacob Bank. 2011. Dynamic ranked retrieval. In Proceedings of the fourth ACM international con- Tom Gruber. 2008. Collective knowledge systems: ference on Web search and data mining. Where the social web meets the semantic web. Web semantics: science, services and agents on the Orkut Buyukkokten, Hector Garcia-Molina, and An- World Wide Web, 6(1). dreas Paepcke. 2001. Seeing the whole in parts: text summarization for web browsing on handheld de- Kai Hakkarainen and Matti Sintonen. 2002. The inter- vices. In Proceedings of the 10th international con- rogative model of inquiry and computer-supported ference on World Wide Web. collaborative learning. Science & Education, 11(1):25–43. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- David Hardtke, Mike Wertheim, and Mark Cramer. moyer. 2018. Quac: Question answering in context. 2009. Demonstration of improved search result rel- In Proc. Conference on Empirical Methods in Natu- evancy using real-time implicit relevance feedback. ral Language Processing (EMNLP). Understanding the User-Logging and Interpreting User Interactions in Information Search and Re- Wendy G Lehnert. 1978. The process of question an- trieval (UIIR-2009). swering: A computer simulation of cognition, vol- ume 978. Lawrence Erlbaum Hillsdale, NJ. Michael Heilman and Noah A Smith. 2010. Good question! statistical ranking for question generation. Thang Luong, Hieu Pham, and Christopher D Man- In Proc. Conference of the North American Chapter ning. 2015. Effective approaches to attention-based of the Association for Computational Linguistics – neural machine translation. In Proc. Conference on Human Language Technologies (NAACL HLT). Empirical Methods in Natural Language Processing (EMNLP). Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Gary Marchionini. 2006. Exploratory search: from 2018. Deep reinforcement learning that matters. In finding to understanding. Communications of the Proc. Association for the Advancement of Artificial ACM, 49(4):41–46. Intelligence (AAAI). Adam Paszke, Sam Gross, Soumith Chintala, Gre- Jaakko Hintikka. 1981. The logic of information- gory Chanan, Edward Yang, Zachary DeVito, Zem- seeking dialogues: A model. Werner Becker and ing Lin, Alban Desmaison, Luca Antiga, and Adam Wilhelm K. Essler Konzepte der Dialektik, pages Lerer. 2017. Automatic differentiation in pytorch. 212–231. In NIPS 2017 Autodiff Workshop.
Jaakko Hintikka. 1988. What is the logic of experi- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt mental inquiry? Synthese, 74(2):173–190. Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long resentations. In Proc. Conference of the North short-term memory. Neural computation. American Chapter of the Association for Computa- tional Linguistics – Human Language Technologies Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine (NAACL HLT). Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. Alec Radford, Jeff Wu, Rewon Child, David Luan, In Proc. Association for Computational Linguistics Dario Amodei, and Ilya Sutskever. 2019. Language (ACL). models are unsupervised multitask learners. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Know what you don’t know: Unanswerable ques- Choi. 2019. The curious case of neural text degen- tions for squad. In Proc. Association for Computa- eration. arXiv preprint arXiv:1904.09751. tional Linguistics (ACL).
Matthew Honnibal and Ines Montani. 2017. spacy 2: Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Natural language understanding with bloom embed- Percy Liang. 2016. Squad: 100,000+ questions for dings, convolutional neural networks and incremen- machine comprehension of text. In Empirical Meth- To appear tal parsing. . ods in Natural Language Processing, pages 2383– 2392. Unnat Jain, Svetlana Lazebnik, and Alexander G Schwing. 2018. Two can play this game: visual Siva Reddy, Danqi Chen, and Christopher D. Manning. dialog with discriminative question generation and 2018. Coqa: A conversational question answering answering. In Proc. IEEE Computer Society Conf. challenge. arXiv preprint arXiv:1808.07042. Computer Vision and Pattern Recognition (CVPR). Mrinmaya Sachan and Eric Xing. 2018. Self-training Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Ky- for jointly learning to ask and answer questions. In omin Jung. 2018. Improving neural question gen- Proc. Conference of the North American Chapter of eration using answer separation. arXiv preprint the Association for Computational Linguistics – Hu- arXiv:1809.02393. man Language Technologies (NAACL HLT).
Yoon Kim. 2014. Convolutional neural networks for Roger C Schank. 1972. Conceptual dependency: A sentence classification. In Proc. Conference on Em- theory of natural language understanding. Cognitive pirical Methods in Natural Language Processing psychology, 3(4):552–631. (EMNLP). Abigail See, Peter J Liu, and Christopher D Manning. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A 2017. Get to the point: Summarization with pointer- method for stochastic optimization. In Proc. Inter- generator networks. In Proc. Association for Com- national Conference on Learning Representations putational Linguistics (ACL). (ICLR). Rico Sennrich, Barry Haddow, and Alexandra Birch. J Richard Landis and Gary G Koch. 1977. The mea- 2016. Neural machine translation of rare words with surement of observer agreement for categorical data. subword units. In Proc. Association for Computa- biometrics, pages 159–174. tional Linguistics (ACL). Iulian Vlad Serban, Alberto García-Durán, Caglar Amy X. Zhang, Lea Verou, and David Karger. 2017. Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Wikum: Bridging discussion forums and wikis using Courville, and Yoshua Bengio. 2016. Generat- recursive summarization. In Conference on Com- ing factoid questions with recurrent neural net- puter Supported Cooperative Work and Social Com- works: The 30m factoid question-answer corpus. puting (CSCW). In Proc. Association for Computational Linguistics (ACL).
Manfred Stede and David Schlangen. 2004. Information-seeking chat: Dialogues driven by topic-structure. In Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04).
Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017. Question answering and ques- tion generation as dual tasks. arXiv preprint arXiv:1706.02027.
Christopher Tauchmann, Thomas Arnold, Andreas Hanselowski, Christian M Meyer, and Margot Mieskes. 2018. Beyond generic summarization: A multi-faceted hierarchical summarization corpus of large heterogeneous data. In Proceedings of the Eleventh International Conference on Language Re- sources and Evaluation (LREC).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. Neural Information Processing Systems (NIPS).
Christian Wagner. 2004. Wiki: A technology for con- versational knowledge management and group col- laboration. Communications of the association for information systems, 13(1):19.
Christian Wagner and Narasimha Bolloju. 2005. Sup- porting knowledge management in organizations with conversational technologies: Discussion fo- rums, weblogs, and wikis. Journal of Database Management, 16(2).
Tong Wang, Xingdi Yuan, and Adam Trischler. 2017. A joint model for question answering and question generation. arXiv preprint arXiv:1706.01450.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A trans- fer learning approach for neural network based con- versational agents. CoRR, abs/1901.08149.
Mark Yatskar. 2019. A qualitative comparison of coqa, squad 2.0 and quac. Proc. Human Language Tech- nology/Conference of the North American Chap- ter of the Association for Computational Linguistics (HLT/NAACL).
Ka-Ping Yee, Kirsten Swearingen, Kevin Li, and Marti Hearst. 2003. Faceted metadata for image search and browsing. In Proceedings of the SIGCHI con- ference on Human factors in computing systems. Appendix frame answer span selection as a sequence la- belling problem. We experimented with the NER A Question Classification Details system in AllenNLP (with ELMo embeddings) on Confirming our intuition, Table 2 shows us the QuAC dataset, after the ground truth answer that QuAC has the highest percentage of GEN- spans marked with BIO tags, after overlapping an- ERAL questions. On the other hand CoQA and swers were merged together. We recorded low SQuAD, which allowed the question-asker to look F1 scores of 33.3 and 15.6 on sentence-level and at the passage, are dominated by SPECIFIC ques- paragraph-level input respectively. tions. These findings are consistent with a Paragraph-level question generation. Our comparison across the three datasets in Yatskar question generation model rarely generated (2019). Interestingly, the average answer length GENERAL questions representative of the entire for SPECIFIC questions in QuAC is 12 tokens, paragraph, even when we fed the entire paragraph compared to 17 tokens for GENERAL questions. as the answer span. We noticed that most GEN- We provide the exact distribution of rule-labeled, ERAL questions in our dataset were answered by hand-labeled and classifier-labeled questions in one or two sentences in the paragraph. Table A1. Answering system trained on all datasets. Re- cently, Yatskar(2019) reported small improve- B Hyperparameters for Question ments on the QuAC validation set by pre-training Generation the BiDAF++ model on SQuAD 2.0 or CoQA. We tried combining the training data in all three Our question generation system consists of a two datasets but achieved a validation F1 score of just layer bidirectional LSTM encoder and a unidirec- 29.3 (compared to 50.2 after using just QuAC tional LSTM decoder respectively. The LSTM training data). hidden unit size in each direction and token em- bedding size is each set to 512. The class speci- Dataset Size Rule Hand CNN ficity embeddings size is 16. Embeddings are shared between the paragraph encoder and ques- SQuAD 86.8k 30.5% 0.0% 69.5% tion decoder. All attention computations use a QuAC 65.2k 59.3% 1.5% 39.2% bilinear product (Luong et al., 2015). A dropout CoQA 105.6k 57.1% 0.1% 42.8% of 0.5 is used between LSTM layers. Models are All 257.6k 48.7% 0.4% 50.9% trained using Adam (Kingma and Ba, 2014) with a learning rate of 10−3, with a gradient clipping of Table A1: Distribution of scheme adopted to classify questions in different datasets. “CNN” refers to the 5.0 and minibatch size 32. Early stopping on val- data-driven classifier. Roughly half the questions were idation perplexity is used to choose the best ques- classified using the rules described in Table 1. tion generation model.
C What did not work? End-to-End Sequential Generation. We ex- perimented with an end-to-end neural model which generated a sequence of questions given a sequence of answer spans. As training data, we leveraged the sequence IDs and follow-up infor- mation in the QuAC dataset, without specificity la- bels. We noticed that during decoding the model rarely attended over the history and often pro- duced questions irrelevant to the context. A poten- tial future direction would involve using the speci- ficity labels for an end-to-end model.
Span Selection NER system. As discussed in Section 3.1 and Du and Cardie(2017), we could Orson Welles Michael Phelps Converge (band)
Breaking with the Federal Theatre Project in 1937, Welles and Houseman founded their own repertory Before the final of the 100-meter butterfly, US born On February 25, 2003 Converge released their first company, which they called the Mercury Theatre. The Serbian swimmer Milorad Cavic caused a minor stir official DVD, The Long Road Home. The DVD is name was inspired by the title of the iconoclastic when he said it would be "good" if Phelps lost. "It'd be modeled after band home videos such as Metallica's magazine, The American Mercury. Welles was good for him if he loses. It would be nice if historians Cliff Em' All release. Deathwish Inc describes the DVD executive producer, and the original company included talk about Michael Phelps winning seven gold medals as a "two disc collection that is as energetic and such actors as Joseph Cotten, George Coulouris, and losing the eighth to 'some guy.' I'd like to be that exciting as the moments the release captures". The Geraldine Fitzgerald, Arlene Francis, Martin Gabel, guy", Cavic said. Phelps responded, "When people say DVD also comes with a bonus disk that included three John Hoyt, Norman Lloyd, Vincent Price, Stefan things like that, it fires me up more than anything." On full live sets from the band. Schnabel and Hiram Sherman. August 16, Phelps won his seventh gold medal of the Games in the men's 100-meter butterfly, setting an Q. What did they do in 2003? Q. What is Mercury Theatre? Olympic record for the event with a time of 50.58 A. On February 25, 2003 Converge seconds and edging out his nearest competitor Cavic, A. Breaking with the Federal Theatre by one hundredth (0.01) of a second. released their first official DVD, The Long Project in 1937, Welles and Houseman Road Home. founded their own repertory company, Q. Why was he lost? Q. What was their first DVD? which they called the Mercury Theatre. A. "It'd be good for him if he loses A. On February 25, 2003 Converge released their first Q. What company did they Q. What did Phelps do on August 16? official DVD, The Long Road form? A. On August 16, Phelps won his seventh Home. A. Federal Theatre Project gold medal of the Games in the men's 100- Q. When did they release this? Q. When was the Federal meter butterfly, A. On February 25, 2003 Theatre Project founded? A. 1937, Q.Who did he win against? Q. Where were the release? A. 100-meter butterfly, A. The Long Road Home. Q. Who started it? A. Welles and Houseman founded Q. Who is the Serbian swimmer? Q.What was the DVD about? their own repertory company, A. US born Serbian swimmer A. The DVD is modeled after band home which they called the Mercury Milorad Cavic videos such as Metallica's Cliff Em' All Theatre. Q. Who did he lose? release. Q. Who was the producer? A. Milorad Cavic A. Welles was executive producer, Q. What other videos did they Q. When did he win a medal? and the original company included have? A. On August 16 such actors as Joseph Cotten, A. Metallica's Cliff Em' All release. George Coulouris, Geraldine Q. How many gold medals did Fitzgerald, Arlene Francis, Martin he win? Q. How many sets were from the Gabel, A. Phelps won his seventh gold band? medal of the Games in the men's A. three full live sets from the bad Q. Why was it called the Federal 100-meter butterfly, Theatre? Q. What is Deathwise Inc? Q. Who did he beat? A. The name was inspired by the title of the A. Deathwish Inc describes the DVD as a A. edging out his nearest iconoclastic magazine, The American "two disc collection that is as energetic and competitor Cavic, by one Mercury. exciting as the moments the release hundredth (0.01) of a second. Q. What was the name of the captures". iconoclastic magazine? A. The American Mercury
Figure A1: Three SQUASH outputs generated by our system, showcasing the strengths and weaknesses described in Section 5. D Technical Note on Modified System GPT-2, Causal Transformer Language Model This technical note describes the modifications made to the originally published system to make Word a faster and more accurate system. We have in- Embeddings corporated language modelling pre-training in our modules using GPT-2 small (Radford et al., 2019) Positional for question generation and BERT (Devlin et al., Embeddings 2019) for question answering. The official code and demo uses the modified version of the system. Segment P P P SA P P SA SQ SQ Embeddings D.1 Dataset Input Paragraph Ans Question
The primary modification in the dataset tackles the P paragraph segment problem of coreferences in GENERAL questions, SA SPECIFIC answer segment as described in Section 5.2. This is a common problem in QuAC and CoQA due to their con- SQ SPECIFIC question segment textual setup. We pass every question through the spaCy pipeline extension neuralcoref20 using Figure A2: An illustration of the model used for gener- the paragraph context to resolve co-references. We ating a SPECIFIC question. A paragraph (context), an- swer and question and concatenated and the model is have also black-listed a few more question tem- optimized to generate the question. Separate segment plates (such as “What happened in
20https://github.com/huggingface/ neuralcoref/ 21https://github.com/huggingface/ 22https://github.com/huggingface/ transfer-learning-conv-ai pytorch-pretrained-BERT