NoiLin: Improving adversarial training and correcting stereotype of noisy labels
NoiLin: Improving adversarial training and correcting stereotype of noisy labels
复制标题
DOI：
--
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
Jingfeng Zhang;Xilie Xu;Bo Han;Tongliang Liu;Gang Niu;Li-zhen Cui;Masashi Sugiyama
中科院分区：
文献类型：
--
作者：
Jingfeng Zhang;Xilie Xu;Bo Han;Tongliang Liu;Gang Niu;Li-zhen Cui;Masashi Sugiyama
将对抗性训练（AT）表述为极大极小优化问题，可以有效提高模型对对抗性攻击的鲁棒性。现有的AT方法主要集中在操纵内部最大化以生成高质量的对抗变量或操纵外部最小化以设计有效的学习目标。然而，AT的实证结果总是表现出鲁棒性与准确性不一致，并且存在交叉混合问题，这促使我们研究一些标签随机性以使AT受益。首先，我们深入研究了噪声标签（NLs）注入到AT的内部最大化和外部最小化中，并获得了NL注入何时有利于AT的观察结果。其次，基于观察结果，我们提出了一种简单而有效的方法——NoiLIn，该方法在每个训练历元随机注入NL到训练数据中，并在发生鲁棒过拟合时动态增加NL注入速率。从经验上看，NoiLIn可以显著缓解AT的鲁棒过拟合的不良问题，甚至进一步提高最先进的AT方法的泛化。从哲学上讲，NoiLIn揭示了使用nl学习的新视角：nl不应该总是被认为是有害的，即使在训练集中没有nl的情况下，我们也可以考虑故意注入它们。代码可在https://github.com/zjfheart/NoiLIn上获得。
Adversarial training (AT) formulated as the minimax optimization problem can effectively enhance the model's robustness against adversarial attacks. The existing AT methods mainly focused on manipulating the inner maximization for generating quality adversarial variants or manipulating the outer minimization for designing effective learning objectives. However, empirical results of AT always exhibit the robustness at odds with accuracy and the existence of the cross-over mixture problem, which motivates us to study some label randomness for benefiting the AT. First, we thoroughly investigate noisy labels (NLs) injection into AT's inner maximization and outer minimization, respectively and obtain the observations on when NL injection benefits AT. Second, based on the observations, we propose a simple but effective method -- NoiLIn that randomly injects NLs into training data at each training epoch and dynamically increases the NL injection rate once robust overfitting occurs. Empirically, NoiLIn can significantly mitigate the AT's undesirable issue of robust overfitting and even further improve the generalization of the state-of-the-art AT methods. Philosophically, NoiLIn sheds light on a new perspective of learning with NLs: NLs should not always be deemed detrimental, and even in the absence of NLs in the training set, we may consider injecting them deliberately. Codes are available in https://github.com/zjfheart/NoiLIn.