IIRC: A Dataset of Incomplete Information Reading Comprehension Questions James Ferguson}∗ Matt Gardner| Hannaneh Hajishirzi}| Tushar Khot| Pradeep Dasigi| }University of Washington |Allen Institute for AI fjfferg,
[email protected] fmattg,tushark,
[email protected] Abstract Humans often have to read multiple doc- uments to address their information needs. However, most existing reading comprehen- sion (RC) tasks only focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a poten- tial lack of sufficient information and locat- ing sources for that information. To fill this gap, we present a dataset, IIRC, with more than 13K questions over paragraphs from En- glish Wikipedia that provide only partial in- formation to answer them, with the missing information occurring in one or more linked documents. The questions were written by crowd workers who did not have access to any Figure 1: An example from IIRC. At the top is a con- of the linked documents, leading to questions text paragraph which provides only partial information that have little lexical overlap with the con- required to answer the question. The bold spans in the texts where the answers appear. This process context indicate links to other Wikipedia pages. The also gave many questions without answers, colored boxes below the question show snippets from and those that require discrete reasoning, in- four of these pages that provide the missing informa- creasing the difficulty of the task. We fol- tion for answering the question. The answer is the un- low recent modeling work on various reading derlined span.