Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future
Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future
复制标题
DOI：
10.1162/coli_a_00464
复制
发表时间：
2022-06
影响因子：
9.3
通讯作者：
Jan-Christoph Klie;B. Webber;Iryna Gurevych
中科院分区：
文献类型：
--
作者：
Jan-Christoph Klie;B. Webber;Iryna Gurevych
在训练和评估机器学习模型的自然语言处理中，带注释的数据是必不可少的组成部分。因此，高质量的注释是非常可取的。然而，最近的研究表明，一些流行的数据集包含了数量惊人的注释错误或不一致。为了缓解这个问题，多年来已经设计了许多注释错误检测方法。虽然研究人员表明他们的方法在新引入的数据集上工作得很好，但他们很少将他们的方法与以前的工作或相同的数据集进行比较。这引起了对方法一般性能的强烈关注，并使评估它们的优缺点变得困难。因此，我们重新实现了18种方法来检测潜在的注释错误，并在9个英语数据集上对它们进行评估，用于文本分类以及标记和span标记。此外，我们定义了一个统一的评估设置，包括注释错误检测任务的新形式化、评估协议和通用最佳实践。为了促进未来的研究和可重复性，我们在一个易于使用的开源软件包中发布了我们的数据集和实现
Annotated data is an essential ingredient in natural language processing for training and evaluating machine learning models. It is therefore very desirable for the annotations to be of high quality. Recent work, however, has shown that several popular datasets contain a surprising number of annotation errors or inconsistencies. To alleviate this issue, many methods for annotation error detection have been devised over the years. While researchers show that their approaches work well on their newly introduced datasets, they rarely compare their methods to previous work or on the same datasets. This raises strong concerns on methods’ general performance and makes it difficult to assess their strengths and weaknesses. We therefore reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets for text classification as well as token and span labeling. In addition, we define a uniform evaluation setup including a new formalization of the annotation error detection task, evaluation protocol, and general best practices. To facilitate future research and reproducibility, we release our datasets and implementations in an easy-to-use and open source software package.1