Active Reading Comprehension: A dataset for learning the Question-Answer Relationship strategy Diana Galvan1 ([email protected]) 1Graduate School of Information Sciences, Tohoku University

Overview Background Current Reading Comprehension (RC) setting ü Goal: Create a general reading comprehension model. RC Dataset 1 .86 acc Uneven performance ML ü How: Importing the QAR strategy from the Model RC Dataset 1 algorithm Model’s performance drops when educational literature to the NLP domain. tested on a different RC dataset. RC Dataset 2 .55 acc ü Contribution: Novel evaluation method. Q1: xxxxxx A1: xxxxx Accuracy Product-based evaluation ü Plan: Rely on previous RC work to create a Q2: xxxxxx A2: xxxxx 0.5 new dataset. ・Focus on the output of the activity. ・Can result in the false impression of My answer was A2: xxxxx ◯◯◯ incomplete weak understanding.

Research proposal Key Idea: Evaluate the overall comprehension process Word match (question & text)

Interviewer: Mrs. Choy, would you like to tell us something about your background before the Japanese invasion? Q2: What were the consequences of Elizabeth’s parents and grandparents being “more advanced for their times” ? 1 Elizabeth Choy: 1. Oh, it will go back quite a long way, you know, because I came to A2: They wanted their daughters to be educated so they sent them to Singapore in December 1929 for higher education. 2. I was born in which schools away from home. is now. 3. My ancestors were from China. 4. They went to Hong Kong, and from Hong Kong, they came to . 5. They started plantations, coconut plantations,

rubber plantations. 6. My parents and grandparents were more advanced for their times 2 and when they could get on a bit, they wanted their daughters to be educated too. 7. Answer location (sentence 6) So, we were sent to schools away from home. 8. First, we went to Jesselton which is Missing piece now. 9. There was a girls’ school run by English missionaries. of information

Question-answer relationship strategy

u Proposed by Raphael (1982) as a way of improving children Step 2 Identify the information source reading performance across grades and subject areas.

u Emphasizes the relationship between the text, Q2 Both sentence 6 and sentence 7 questions and the reader’s prior knowledge. answer the question.

In the text In my head I should combine the answers found in these two sentences. Right there Question Author and me Step 1 classification Think and search On my own Step 3 Identify the type of reasoning

Research plan 1 Create a new dataset for a QAR classification task New annotations Classify RC questions into one of the Task: QAR categories. Sentence Answer Sentence Sentence Answer QAR 1 Mary moved to the bathroom Needed 1 Mary moved to the bathroom category 2 John went to the kitchen 2 John went to the kitchen 3 Mary took the football there 3 Mary took the football there 4 How many objects is Mary carrying? one 3 4 How many objects is Mary carrying? one 1 5 John went back to the kitchen 5 John went back to the kitchen Where n is the number of sentences 2 6 Mary dropped the ball 6 Mary dropped the ball Evaluate state-of-the-art RC models on 7 How many objects is Marry carrying? none 3, 6 7 How many objects is Marry carrying? none 2 the QAR classification task

bAbI HotpotQA q Knowledgeable reader (Mihaylov and Frank, 2018) Weston et al., 2015 et al., 2018 QAR MCScript Dataset q Knowledge-enhanced reader (Mihaylov et al., 2018) SQuAD MultiRC Ostermann et al., 2018 Rajpurkar et al., 2016 Khashabi et al., 2018 3) Author and me q TriAN (Wang et al., 2018) … 1) Right there 2) Think and search 4) On my own

The 2019 ACL Student Workshop (July 29, 2019)