Active Reading Comprehension: A dataset for learning the Question-Answer Relationship strategy Diana Galvan Tohoku University Graduate School of Information Sciences Sendai, Japan
[email protected] Abstract evaluation can result in the false impression of weak understanding (i.e., misunderstanding of the Reading comprehension (RC) through ques- text, the question, or both) or the absence of re- tion answering is a useful method for evaluat- quired knowledge. However, the system could ing if a reader understands a text. Standard ac- curacy metrics are used for evaluation, where a have failed to arrive at the correct answer for some high accuracy is taken as indicative of a good other reasons. For example, consider the reading understanding. However, literature in quality comprehension task shown in Figure1. For the learning suggests that task performance should question “What were the consequences of Eliza- also be evaluated on the undergone process to beth Choy's parents and grandparents being ‘more answer. The Question-Answer Relationship advanced for their times’?” the correct answer (QAR) is one of the strategies for evaluating is in the text but it is located in different sen- a reader’s understanding based on their ability to select different sources of information de- tences. If the system only identifies “They wanted pending on the question type. We propose the their daughters to be educated” as an answer, it creation of a dataset to learn the QAR strategy would be judged to be incorrect when it did not with weak supervision. We expect to comple- fail at finding the answer, it failed at connecting ment current work on reading comprehension it with the fact “we were sent to schools away by introducing a new setup for evaluation.