R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason
R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason
复制标题
DOI：
10.18653/v1/2020.acl-main.602
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Naoya Inoue;Pontus Stenetorp;Kentaro Inui
中科院分区：
文献类型：
--
作者：
Naoya Inoue;Pontus Stenetorp;Kentaro Inui
最近的研究表明，阅读理解（RC）系统学会利用当前数据集中的注释伪像和其他偏见。 RC系统的内部推理不仅需要答案，而且还需要派生：证明我们的答案是合理的。用衍生的统一注释RC数据集。使用多个参考推导的评估指标是可靠的，R4C评估与现有基准的不同技能不同。
Recent studies have revealed that reading comprehension (RC) systems learn to exploit annotation artifacts and other biases in current datasets. This prevents the community from reliably measuring the progress of RC systems. To address this issue, we introduce R4C, a new task for evaluating RC systems’ internal reasoning. R4C requires giving not only answers but also derivations: explanations that justify predicted answers. We present a reliable, crowdsourced framework for scalably annotating RC datasets with derivations. We create and publicly release the R4C dataset, the first, quality-assured dataset consisting of 4.6k questions, each of which is annotated with 3 reference derivations (i.e. 13.8k derivations). Experiments show that our automatic evaluation metrics using multiple reference derivations are reliable, and that R4C assesses different skills from an existing benchmark.