Towards Inference-Oriented Reading Comprehension: Parallelqa
Total Page：16
File Type：pdf, Size：1020Kb
Towards Inference-Oriented Reading Comprehension: ParallelQA Soumya Wadhwa∗ Varsha Embar∗ Matthias Grabmair Eric Nyberg Carnegie Mellon University soumyaw, vembar, mgrabmai, en09 @andrew.cmu.edu { } Abstract help perform such reasoning. Although state-of- the-art deep learning models for machine reading In this paper, we investigate the tendency of are believed to have such reasoning capabilities, end-to-end neural Machine Reading Compre- the limited ability of these models to generalize hension (MRC) models to match shallow pat- terns rather than perform inference-oriented indicates certain shortcomings. We believe that it reasoning on RC benchmarks. We aim to test is important to develop benchmarks which give a the ability of these systems to answer ques- realistic sense of a system’s RC capabilities. Thus, tions which focus on referential inference. We our goal in this paper is two-fold: propose ParallelQA, a strategy to formulate Proof of Concept: We propose a method to create such questions using parallel passages. We an RC dataset that assesses a model’s ability to: also demonstrate that existing neural models move beyond lexical pattern matching be- fail to generalize well to this setting. • tween the question and passage, infer the correct answers to questions which 1 Introduction • contain referring expressions, and Reading Comprehension (RC) is the task of read- generalize to different language styles. • ing a body of text and answering questions about Analysis of Existing Models: We test three it. It requires a deep understanding of the infor- end-to-end neural MRC models, which perform mation presented in order to reason about entities, well on SQuAD (Rajpurkar et al., 2016), on a actions, events, and their interrelationships. This few question-answer pairs generated using our necessitates language understanding skills as well methodology. We demonstrate that it is indeed dif- as the cognitive ability to draw inferences. ficult for these systems to answer such questions, Recent efforts in creating large-scale datasets also indicating their tendencies to resort to shallow have triggered a renewed interest in the RC task, pattern matching and overfit to training data. with subsequent development of complex end- to-end solutions featuring neural models. While 2 Existing Datasets these models do exceedingly well on the specific In this work, we focus on datasets with multi- datasets they are developed for (some reaching word spans as answers rather than cloze-style RC or even surpassing human performance), they do datasets like MCTest (Richardson et al., 2013), not perform proportionally across datasets. Weis- CNN / Daily Mail (Hermann et al., 2015) and senborn et al.(2017) have shown that using a con- Children’s Book Test (Weston et al., 2015). text or type matching heuristic to derive simple The Stanford Question Answering Dataset neural baseline architectures can achieve compa- (SQuAD) (Rajpurkar et al., 2016) was one of the rable results. Our experiments also indicate that first large scale RC datasets (over 100k QA pairs), pattern matching can work well on these datasets. where the answer to each question is a span in the Inference, an important RC skill (Spearritt, given passage. For its collection, different sets 1972; Strange, 1980), is the ability to understand of crowd-workers were asked to formulate ques- the meaning of text without all the information be- tions and answers using passages obtained from ing stated explicitly. Table5, SectionA describes 500 Wikipedia articles. However, this resulted ∼ the types of inference that we may encounter while in the questions having similar word patterns to comprehending a passage along with the cues that the sentences containing the answers. We empir- ∗Equal Contribution ically demonstrate this in Table1, where we ob- 1 Proceedings of the Workshop on Generalization in the Age of Deep Learning, pages 1–7 New Orleans, Louisiana, June 5, 2018. c 2018 Association for Computational Linguistics served that the sentence in the passage with the eral important aspects of RC that remain untested. highest lexical similarity to the question contained the answer 80% of the time. Final answers tend 3 ParallelQA ∼ to be short, with an average span length of around 3 tokens, and are largely entities (40.88%). Subra- In an RC task, there is a need to incorporate ques- manian et al.(2017) and Yang et al.(2017) provide tions that require not just lexical and syntactic evidence for regular patterns in candidate answers prowess, but reference resolution, multiple steps that neural models can exploit. We show in subse- of reasoning, and use of world knowledge. These quent sections that models which perform well on capabilities ultimately lead to global rather than SQuAD rely on lexical pattern matching, and are sentence-level understanding of text. The con- also not robust to variance in language style. struction of a large-scale dataset of this nature is a challenging task. We take a small step in this direction by focusing on referential inference.1 Metric SQuAD NewsQA ParallelQA WikiHop (Welbl et al., 2017) is an interesting Jaccard 79.28% 38.11% 27.45% multi-hop inference-focused dataset created using TF-IDF 81.32% 51.86% 31.37% entity-relation pairs for queries spanning different BM25 74.26% 43.45% 27.45% Wikipedia passages. While the focus of our pi- lot study is similar to theirs, we believe that our Table 1: Sentence Retrieval Performance using Jaccard similarity (Jaccard, 1912), TF-IDF overlap method can easily be extended to other inference (Sparck Jones, 1972) and BM-25 overlap (Robertson types. Also, identifying the correct span is more et al., 1994) scoring metrics challenging than choosing an answer from a list. We aim to incorporate multiple language styles, To alleviate the lack of topic diversity in making it hard for the system to memorize linguis- SQuAD, NewsQA (Trischler et al., 2016) was tic patterns (Williams et al., 2017). We achieve created from 12,744 news articles sampled from this by using two parallel passages that talk about CNN/Daily Mail. To ensure lexical diversity, one the same or related subject(s) but are obtained set of crowd-workers generated questions using from different sources. This helps in formulating only an abstractive summary, while the answer referential inference questions because there exists spans were marked in the full article by another set no single sentence in the passage which matches a of crowd-workers. However, news articles tend to paraphrase of the question, and necessitates that encourage questions that point to entities, and the inference (which goes beyond co-reference) be dataset does not specifically focus on inference. performed across both passages. Evaluation is Determining the exact answer span is harder, but easy and objective because answers are still spans this may be due to the use of only news highlights within the passages. Questions can be answered to generate questions; this may induce noise in the solely on the basis of the information provided in answer spans marked in the news articles since the their accompanying passages. question might not be exactly apt. For example, to answer Question 1 in Table2, To prevent annotation bias, SearchQA (Dunn the system will have to infer from passage 1 that et al., 2017) starts with question-answer pairs President Kamazu Banda belongs to the MCP and from Jeopardy! and adds documents retrieved was defeated in the elections. The equivalence of by a search engine for each question as its con- this event and the election in passage 2 must be es- text. However, the questions are mostly factoid. tablished, while comprehending that the “favored Kociskˇ y` et al.(2017) found that 80% of answers challenger” Bakili Muluzi is the one Banda lost are bigrams or unigrams, and 99% contain 5 or the elections to, and who belonged to the UDP, fewer tokens, with many answers being named making it the correct answer. entities. TriviaQA (Joshi et al., 2017) similarly Given that the information is spatially scattered includes question-answer pairs authored by trivia across the two passages, this method would ensure enthusiasts along with independently-gathered ev- that the parallel passages have to be understood in idence documents which provide distant supervi- combination to answer the question. sion for answering the questions. 1Referential inference is the process of identifying the dis- These datasets have facilitated the development course and/or real-world entity referred to by a linguistic ex- of new QA models, but we believe there are sev- pression (name, noun, pronoun, etc.). 2 Hastings Kamuzu Banda was the leader of Malawi from 1961 to 1994. In 1963 he was formally appointed prime minister of Nyasaland and, a year later, led the country to independence as Malawi. Two years later he proclaimed Malawi a republic with himself as president. He declared Malawi a one-party state under the Malawi Congress Party (MCP) and became President of MCP as well as President for Life of Malawi in 1971. A referendum ended his one-party state and a special assembly ended his life-term presidency, stripping him of most of his powers. Banda ran for president in the democratic elections which followed and was defeated. He died in South Africa in 1997. Malawians Saturday wound up an historic election campaign bringing multiparty politics to a country ruled for the past three decades by President Hastings Kamuzu Banda. The ailing president inspected troops from an open truck as some 20,000 people turned up at a stadium here to celebrate his official birthday ahead of elections on May 17. Reading a prepared speech with some difficulty, Banda appealed to Malawians to conduct themselves ”as ladies and gentlemen” during the elections, which should be ”free and fair.” Meanwhile, the bigger opposition rally was addressed by the presidential challenger favored to win the elections, Bakili Muluzi of the United Democratic Front (UDF).