Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks
复制标题
DOI：
10.1609/aaai.v34i05.6398
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Anna Rogers;Olga Kovaleva;Matthew Downey;Anna Rumshisky
中科院分区：
文献类型：
--
作者：
Anna Rogers;Olga Kovaleva;Matthew Downey;Anna Rumshisky
最近问答研究的爆炸产生了大量的事实阅读理解（RC）和常识推理数据集。把它们结合起来就提出了一种不同的任务：不仅要决定文本中是否存在信息，还要决定是否可以对缺失的信息进行可靠的猜测。我们提出了QuAIL，第一个RC数据集结合联合收割机基于文本的，世界知识和无法回答的问题，并提供问题类型的注释，使诊断的推理策略由一个给定的QA系统。QuAIL包含4个领域800个文本的15K多项选择题。至关重要的是，它提供了一般和文本特定的问题，不太可能在预训练数据中找到。我们发现，QuAIL对当前最先进的系统构成了重大挑战，与最相似的现有数据集相比，准确性下降了30%。
The recent explosion in question answering research produced a wealth of both factoid reading comprehension (RC) and commonsense reasoning datasets. Combining them presents a different kind of task: deciding not simply whether information is present in the text, but also whether a confident guess could be made for the missing information. We present QuAIL, the first RC dataset to combine text-based, world knowledge and unanswerable questions, and to provide question type annotation that would enable diagnostics of the reasoning strategies by a given QA system. QuAIL contains 15K multi-choice questions for 800 texts in 4 domains. Crucially, it offers both general and text-specific questions, unlikely to be found in pretraining data. We show that QuAIL poses substantial challenges to the current state-of-the-art systems, with a 30% drop in accuracy compared to the most similar existing dataset.