STARC: Structured Annotations for Reading Comprehension
STARC: Structured Annotations for Reading Comprehension
复制标题
DOI：
10.18653/v1/2020.acl-main.507
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Yevgeni Berzak;J. Malmaud;R. Levy
中科院分区：
文献类型：
--
作者：
Yevgeni Berzak;J. Malmaud;R. Levy
本文提出了一种新的阅读理解能力测试的注释框架STARC（Structured Annotations for阅读Comprehension）。我们的框架引入了一个原则性的结构，答案的选择，并将它们绑定到文本跨度注释。该框架在OneStopQA中实现，OneStopQA是一个新的高质量数据集，用于评估和分析英语阅读理解。我们使用这个数据集来证明，STARC可以利用一个关键的新的应用程序的SAT样阅读理解材料的发展：自动注释质量探测通过跨度消融实验。我们进一步表明，它可以深入分析和比较机器和人类的阅读理解行为，包括错误分布和猜测能力。我们的实验还表明，标准的多项选择数据集在NLP，RACE，是有限的，在其能力来衡量阅读理解。47%的问题可以被机器猜到，而不需要访问文章，18%的问题被人类一致认为没有唯一的正确答案。OneStopQA为阅读理解提供了一个替代测试集，它弥补了这些缺点，并具有显著更高的人类上限性能。
We present STARC (Structured Annotations for Reading Comprehension), a new annotation framework for assessing reading comprehension with multiple choice questions. Our framework introduces a principled structure for the answer choices and ties them to textual span annotations. The framework is implemented in OneStopQA, a new high-quality dataset for evaluation and analysis of reading comprehension in English. We use this dataset to demonstrate that STARC can be leveraged for a key new application for the development of SAT-like reading comprehension materials: automatic annotation quality probing via span ablation experiments. We further show that it enables in-depth analyses and comparisons between machine and human reading comprehension behavior, including error distributions and guessing ability. Our experiments also reveal that the standard multiple choice dataset in NLP, RACE, is limited in its ability to measure reading comprehension. 47% of its questions can be guessed by machines without accessing the passage, and 18% are unanimously judged by humans as not having a unique correct answer. OneStopQA provides an alternative test set for reading comprehension which alleviates these shortcomings and has a substantially higher human ceiling performance.