What Makes Reading Comprehension Questions Easier?
What Makes Reading Comprehension Questions Easier?
复制标题
DOI：
10.18653/v1/d18-1453
复制
发表时间：
2018-08
期刊：
影响因子：
--
通讯作者：
Saku Sugawara;Kentaro Inui;S. Sekine;Akiko Aizawa
中科院分区：
文献类型：
--
作者：
Saku Sugawara;Kentaro Inui;S. Sekine;Akiko Aizawa
创建用于机器阅读理解的数据集（MRC）的挑战是收集需要对语言有复杂理解的问题，以回答不使用浅表提示。在这项工作中，我们调查了什么使最近的12个MRC数据集具有三种问题样式（答案提取，描述和多项选择）更容易。我们建议采用简单的启发式方法将每个数据集拆分为简单而硬的子集，并检查每个子集的两个基线模型的性能。然后，我们以有效性和必要的推理技能来手动注释从每个子集中抽样的问题，以调查哪些技能解释简易问题和硬性问题之间的差异。从这项研究中，我们观察到（i）与整个数据集相比，硬性子集的基线性能显着降低，（ii）硬性问题需要知识推断和与简单问题相比，以及（iii）多重句子推理 - 选择问题往往需要更广泛的推理技能，而不是回答提取和描述问题。这些结果表明，可能高估了MRC最近的进步。
A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investigate what makes questions easier across recent 12 MRC datasets with three question styles (answer extraction, description, and multiple choice). We propose to employ simple heuristics to split each dataset into easy and hard subsets and examine the performance of two baseline models for each of the subsets. We then manually annotate questions sampled from each subset with both validity and requisite reasoning skills to investigate which skills explain the difference between easy and hard questions. From this study, we observed that (i) the baseline performances for the hard subsets remarkably degrade compared to those of entire datasets, (ii) hard questions require knowledge inference and multiple-sentence reasoning in comparison with easy questions, and (iii) multiple-choice questions tend to require a broader range of reasoning skills than answer extraction and description questions. These results suggest that one might overestimate recent advances in MRC.