Label noise correction and application in crowdsourcing
Label noise correction and application in crowdsourcing
复制标题
DOI：
10.1016/j.eswa.2016.09.003
复制
发表时间：
2016-12
期刊：
影响因子：
--
通讯作者：
B. Nicholson;Victor S. Sheng;J. Zhang
中科院分区：
文献类型：
--
作者：
B. Nicholson;Victor S. Sheng;J. Zhang
校正标签噪声的重要任务在文献中很少提到。开发鲁棒的标签校正算法的困难导致了关于标签校正的沉默。为了打破沉默，我们提出了两个算法来纠正标签噪声。一种是利用自训练来重新标记噪声，称为自训练校正（STC）。另一种是基于聚类的方法，将实例分组在一起以推断其地面实况标签，称为基于聚类的校正（CC）。我们还从以前的工作中调整了一种算法，一种基于共识的方法，称为抛光，它与分类器的集合进行协商，以改变属性和标签的值。我们简化抛光，使其只改变实例的标签，并称之为抛光标签（PL）。我们通过实验将我们的新方法与抛光标签进行比较，检查它们在不同噪声水平下对二进制和多类数据集的标签质量、模型质量和AUC指标的改进。我们的实验结果表明，CC显着提高标签质量，模型质量和AUC指标一致。我们进一步研究了这三种噪声校正算法如何在众包图像标记的背景下提高数据质量（标签准确性）。首先，我们研究了三种共识方法，用于从众包获得的多个噪声标签中推断出真实标签，即，多数投票（MV），Dawid Skene（DS）和科斯。然后，我们应用三种噪声校正方法来校正由这些共识方法推断的标签。实验结果表明，噪声校正方法显著提高了标注质量。作为我们实验的总体结果，我们得出结论，CC执行最好的。我们的研究已经说明了实施噪声校正作为防止标签错误的另一条防线的可行性，特别是在众包环境中。此外，它提出了自动化的可行性，否则手动过程分析数据集，并纠正和清理的实例，一个昂贵的和耗时的任务。
The important task of correcting label noise is addressed infrequently in literature. The difficulty of developing a robust label correction algorithm leads to this silence concerning label correction. To break the silence, we propose two algorithms to correct label noise. One utilizes self-training to re-label noise, called Self-Training Correction (STC). Another is a clustering-based method, which groups instances together to infer their ground-truth labels, called Cluster-based Correction (CC). We also adapt an algorithm from previous work, a consensus-based method called Polishing that consults with an ensemble of classifiers to change the values of attributes and labels. We simplify Polishing such that it only alters labels of instances, and call it Polishing Labels (PL). We experimentally compare our novel methods with Polishing Labels by examining their improvements on the label qualities, model qualities, and AUC metrics of binary and multi-class data sets under different noise levels. Our experimental results demonstrate that CC significantly improves label qualities, model qualities, and AUC metrics consistently. We further investigate how these three noise correction algorithms improve the data quality, in terms of label accuracy, in the context of image labeling in crowdsourcing. First, we look at three consensus methods for inferring a ground-truth label from the multiple noisy labels obtained from crowdsourcing, i.e., Majority Voting (MV), Dawid Skene (DS), and KOS. We then apply the three noise correction methods to correct labels inferred by these consensus methods. Our experimental results show that the noise correction methods improve the labeling quality significantly. As an overall result of our experiments, we conclude that CC performs the best. Our research has illustrated the viability of implementing noise correction as another line of defense against labeling error, especially in a crowdsourcing setting. Furthermore, it presents the feasibility of the automation of an otherwise manual process of analyzing a data set, and correcting and cleaning the instances, an expensive and time-consuming task.