M ITIGATING A DVERSARIAL T RAINING I NSTABILITY WITH B ATCH N ORMALIZATION
M ITIGATING A DVERSARIAL T RAINING I NSTABILITY WITH B ATCH N ORMALIZATION
复制标题
DOI：
--
复制
发表时间：
2021
影响因子：
--
通讯作者：
A. Sridhar;Chawin Sitawarin;David A. Wagner
中科院分区：
文献类型：
--
作者：
A. Sridhar;Chawin Sitawarin;David A. Wagner
对抗训练范式已经成为深度神经网络鲁棒性训练的标准。然而，它仍然不稳定，导致这种不稳定的机制知之甚少。在本研究中，我们发现这种不稳定性主要是由非平滑优化环境和内部协变量移位现象驱动的，并表明批归一化（Batch Normalization， BN）可以有效地缓解这两个问题。此外，我们证明了BN普遍提高了各种防御、数据集和模型类型的干净和健壮的性能，在更困难的任务上有更大的改进。最后，我们用混合批训练验证了BN的异构分布问题，并提出了解决方案。
The adversarial training paradigm has become the standard in training deep neural networks for robustness. Yet, it remains unstable, with the mechanisms driving this instability poorly understood. In this study, we discover that this instability is primarily driven by a non-smooth optimization landscape and an internal covariate shift phenomenon, and show that Batch Normalization (BN) can effectively mitigate both these issues. Further, we demonstrate that BN universally improves clean and robust performance across various defenses, datasets, and model types, with greater improvement on more difficult tasks. Finally, we confirm BN’s heterogeneous distribution issue with mixed-batch training and propose a solution.