Robust Training under Label Noise by Over-parameterization
Robust Training under Label Noise by Over-parameterization
复制标题
DOI：
--
复制
发表时间：
2022-02
期刊：
影响因子：
--
通讯作者：
Sheng Liu;Zhihui Zhu;Qing Qu;Chong You
中科院分区：
文献类型：
--
作者：
Sheng Liu;Zhihui Zhu;Qing Qu;Chong You
最近，过度参数化的深度网络，其网络参数比训练样本越来越多，已经主导了现代机器学习的性能。然而，当训练数据被破坏时，众所周知，过度参数化的网络往往会过度拟合而不能泛化。在这项工作中，我们提出了一种原则方法，用于在分类任务中对部分训练标签损坏的过参数化深度网络进行鲁棒训练。主要思想仍然很简单：标签噪声是稀疏的，与从干净数据中学习到的网络不一致，所以我们对噪声建模并学习将其从数据中分离出来。具体来说，我们通过另一个稀疏过参数化项对标签噪声进行建模，并利用隐式算法正则化来恢复和分离潜在的损坏。值得注意的是，当在实践中使用如此简单的方法进行训练时，我们在各种真实数据集上展示了针对标签噪声的最先进的测试准确性。此外，我们的实验结果得到了简化线性模型理论的证实，表明在非相干条件下可以实现稀疏噪声和低秩数据的精确分离。这项工作为利用稀疏过参数化和隐式正则化改进过参数化模型开辟了许多有趣的方向。
Recently, over-parameterized deep networks, with increasingly more network parameters than training samples, have dominated the performances of modern machine learning. However, when the training data is corrupted, it has been well-known that over-parameterized networks tend to overfit and do not generalize. In this work, we propose a principled approach for robust training of over-parameterized deep networks in classification tasks where a proportion of training labels are corrupted. The main idea is yet very simple: label noise is sparse and incoherent with the network learned from clean data, so we model the noise and learn to separate it from the data. Specifically, we model the label noise via another sparse over-parameterization term, and exploit implicit algorithmic regularizations to recover and separate the underlying corruptions. Remarkably, when trained using such a simple method in practice, we demonstrate state-of-the-art test accuracy against label noise on a variety of real datasets. Furthermore, our experimental results are corroborated by theory on simplified linear models, showing that exact separation between sparse noise and low-rank data can be achieved under incoherent conditions. The work opens many interesting directions for improving over-parameterized models by using sparse over-parameterization and implicit regularization.