Prerequisite Skills for Reading Comprehension: Multi-Perspective Analysis of MCTest Datasets and Systems
Prerequisite Skills for Reading Comprehension: Multi-Perspective Analysis of MCTest Datasets and Systems
复制标题
DOI：
10.1609/aaai.v31i1.10957
复制
发表时间：
2017-02
期刊：
影响因子：
6.7
通讯作者：
Saku Sugawara;Hikaru Yokono;Akiko Aizawa
中科院分区：
文献类型：
--
作者：
Saku Sugawara;Hikaru Yokono;Akiko Aizawa
自然语言处理的主要目标之一是对自然语言文档的综合理解，特别是阅读理解。RC系统的进一步发展的一个障碍是缺乏一个综合的方法来分析其性能。由于自然语言理解的过程是复杂的，因此很难仅仅根据任务的结果来检查系统的性能。为了解决这个问题，我们在本文中提出了一种方法的启发，在软件工程中的单元测试，使检查RC系统从多个方面。我们的方法包括三个步骤。首先，我们根据现有的NLP任务定义了一组RC的先决条件技能。我们假设RC能力可以分为这些技能。其次，我们手动注释RC任务的数据集，其中包含有关回答每个问题所需技能的信息。最后，我们分析了RC系统的性能为每个技能的基础上的注释。最后两个步骤突出了两个方面：数据集的特征，以及RC系统的弱点和差异。我们通过注释机器理解测试（MCTest）数据集并分析四个现有系统（包括神经系统）来测试我们方法的有效性。注释的结果表明，回答问题需要综合技能，并阐明了系统理解自然语言所需的能力。我们的结论是，我们定义的先决条件的技能是有前途的RC的分解和分析。
One of the main goals of natural language processing (NLP) is synthetic understanding of natural language documents, especially reading comprehension (RC). An obstacle to the further development of RC systems is the absence of a synthetic methodology to analyze their performance. It is difficult to examine the performance of systems based solely on their results for tasks because the process of natural language understanding is complex. In order to tackle this problem, we propose in this paper a methodology inspired by unit testing in software engineering that enables the examination of RC systems from multiple aspects. Our methodology consists of three steps. First, we define a set of prerequisite skills for RC based on existing NLP tasks. We assume that RC capability can be divided into these skills. Second, we manually annotate a dataset for an RC task with information regarding the skills needed to answer each question. Finally, we analyze the performance of RC systems for each skill based on the annotation. The last two steps highlight two aspects: the characteristics of the dataset, and the weaknesses in and differences among RC systems. We tested the effectiveness of our methodology by annotating the Machine Comprehension Test (MCTest) dataset and analyzing four existing systems (including a neural system) on it. The results of the annotations showed that answering questions requires a combination of skills, and clarified the kinds of capabilities that systems need to understand natural language. We conclude that the set of prerequisite skills we define are promising for the decomposition and analysis of RC.