To Smooth or Not? When Label Smoothing Meets Noisy Labels
To Smooth or Not? When Label Smoothing Meets Noisy Labels
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Jiaheng Wei;Hangyu Liu;Tongliang Liu;Gang Niu;Yang Liu
中科院分区：
文献类型：
--
作者：
Jiaheng Wei;Hangyu Liu;Tongliang Liu;Gang Niu;Yang Liu
标签平滑（LS）是一种新兴的学习范式，它使用硬训练标签和均匀分布的软标签的正加权平均值。结果表明，LS作为一个正则化的训练数据与硬标签，因此提高了模型的推广。后来有报道称，LS甚至有助于提高使用噪声标签学习时的鲁棒性。然而，我们观察到，当我们在高标签噪声状态下操作时，LS的优势消失。直观地说，这是由于$\mathbb{P}（\text{noisy label}）的熵增加|当噪声率高时，在这种情况下，进一步应用LS倾向于“过度平滑“估计的后验。我们继续发现，文献中的几种带噪声标签的学习解决方案与负/非标签平滑（NLS）更密切相关，NLS与LS相反，并定义为使用负权重来联合收割机组合硬标签和软标签！我们提供的LS和NLS的属性的理解时，学习与噪声标签。在其他既定的属性，我们理论上表明NLS被认为是更有益的标签噪声率高时。我们也提供了多个基准的广泛实验结果来支持我们的发现。代码可在https://github.com/UCSC-REAL/negative-label-smoothing上公开获取。
Label smoothing (LS) is an arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels. It was shown that LS serves as a regularizer for training data with hard labels and therefore improves the generalization of the model. Later it was reported LS even helps with improving robustness when learning with noisy labels. However, we observed that the advantage of LS vanishes when we operate in a high label noise regime. Intuitively speaking, this is due to the increased entropy of $\mathbb{P}(\text{noisy label}|X)$ when the noise rate is high, in which case, further applying LS tends to"over-smooth"the estimated posterior. We proceeded to discover that several learning-with-noisy-labels solutions in the literature instead relate more closely to negative/not label smoothing (NLS), which acts counter to LS and defines as using a negative weight to combine the hard and soft labels! We provide understandings for the properties of LS and NLS when learning with noisy labels. Among other established properties, we theoretically show NLS is considered more beneficial when the label noise rates are high. We provide extensive experimental results on multiple benchmarks to support our findings too. Code is publicly available at https://github.com/UCSC-REAL/negative-label-smoothing.