Coresets for Robust Training of Neural Networks against Noisy Labels
Coresets for Robust Training of Neural Networks against Noisy Labels
复制标题
DOI：
--
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Baharan Mirzasoleiman;Kaidi Cao;J. Leskovec
中科院分区：
文献类型：
--
作者：
Baharan Mirzasoleiman;Kaidi Cao;J. Leskovec
现代神经网络具有过拟合在现实世界数据集中经常发现的噪声标签的能力。虽然已经取得了很大的进步，但现有的技术在为用噪声标签训练的神经网络的性能提供理论保证方面是有限的。在这里，我们提出了一种新颖的方法，为用噪声标签训练的深度网络的鲁棒训练提供了强有力的理论保证。我们的方法背后的关键思想是选择提供近似低秩雅可比矩阵的干净数据点的加权子集（核心集）。然后，我们证明了应用于子集的梯度下降不会过拟合噪声标签。我们的大量实验证实了我们的理论，并证明了在我们的子集上训练的深度网络与最先进的网络相比取得了显著的卓越性能，例如，在CIFAR-10上使用80%的噪声标签提高了6%的准确性，在mini Webvision上提高了7%的准确性。
Modern neural networks have the capacity to overfit noisy labels frequently found in real-world datasets. Although great progress has been made, existing techniques are limited in providing theoretical guarantees for the performance of the neural networks trained with noisy labels. Here we propose a novel approach with strong theoretical guarantees for robust training of deep networks trained with noisy labels. The key idea behind our method is to select weighted subsets (coresets) of clean data points that provide an approximately low-rank Jacobian matrix. We then prove that gradient descent applied to the subsets do not overfit the noisy labels. Our extensive experiments corroborate our theory and demonstrate that deep networks trained on our subsets achieve a significantly superior performance compared to state-of-the art, e.g., 6% increase in accuracy on CIFAR-10 with 80% noisy labels, and 7% increase in accuracy on mini Webvision.