R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason Naoya Inoue1;2 Pontus Stenetorp2;3 Kentaro Inui1;2 1Tohoku University 2RIKEN 3University College London fnaoya-i,
[email protected] [email protected] Abstract Question What was the former band of the member of Mother Recent studies have revealed that reading com- Love Bone who died just before the release of “Apple”? prehension (RC) systems learn to exploit an- Articles notation artifacts and other biases in current Title: Return to Olympus [1] Return to Olympus is the datasets. This prevents the community from only album by the alternative rock band Malfunkshun. reliably measuring the progress of RC systems. [2] It was released after the band had broken up and 4 after lead singer Andrew Wood (later of Mother Love To address this issue, we introduce R C, a Input Bone) had died... [3] Stone Gossard had compiled… new task for evaluating RC systems’ internal Title: Mother Love Bone [4] Mother Love Bone was 4 reasoning. R C requires giving not only an- an American rock band that… [5] The band was active swers but also derivations: explanations that from… [6] Frontman Andrew Wood’s personality and justify predicted answers. We present a reli- compositions helped to catapult the group to... [7] Wood died only days before the scheduled release able, crowdsourced framework for scalably an- of the band’s debut album, “Apple”, thus ending the… notating RC datasets with derivations. We cre- ate and publicly release the R4C dataset, the Explanation Answer first, quality-assured dataset consisting of 4.6k Supporting facts (SFs): Malfunkshun questions, each of which is annotated with 3 [1], [2], [4], [6], [7] reference derivations (i.e.