RUSH: Robust Contrastive Learning via Randomized Smoothing
RUSH: Robust Contrastive Learning via Randomized Smoothing
复制标题
DOI：
10.48550/arxiv.2207.05127
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Yijiang Pang;Boyang Liu;Jiayu Zhou
中科院分区：
文献类型：
--
作者：
Yijiang Pang;Boyang Liu;Jiayu Zhou
最近，对抗性训练已被纳入自监督对比预训练中，以通过令人兴奋的对抗性鲁棒性来提高标签效率。然而，这种鲁棒性是以昂贵的对抗性训练为代价的。在本文中，我们展示了一个令人惊讶的事实，即对比预训练与鲁棒性有着有趣但隐含的联系，预训练表示中的这种自然鲁棒性使我们能够设计一种强大的鲁棒算法来对抗对抗性攻击，RUSH，它结合了标准对比预训练和随机平滑。与对抗性训练相比，它提高了标准精度和鲁棒精度，并显着降低了训练成本。我们使用广泛的实证研究表明，所提出的 RUSH 在一阶攻击下的通用基准（CIFAR-10、CIFAR-100 和 STL-10）上明显优于对抗性训练的鲁棒分类器。特别是，在对 CIFAR-10 进行大小为 8/255 PGD 攻击的 $\ell_{\infty}$-范数扰动下，我们使用 ResNet-18 作为主干的模型达到了 77.8% 的鲁棒精度和 87.9% 的标准精度。与最先进的技术相比，我们的工作在鲁棒精度方面提高了 15% 以上，在标准精度方面略有提高。
Recently, adversarial training has been incorporated in self-supervised contrastive pre-training to augment label efficiency with exciting adversarial robustness. However, the robustness came at a cost of expensive adversarial training. In this paper, we show a surprising fact that contrastive pre-training has an interesting yet implicit connection with robustness, and such natural robustness in the pre trained representation enables us to design a powerful robust algorithm against adversarial attacks, RUSH, that combines the standard contrastive pre-training and randomized smoothing. It boosts both standard accuracy and robust accuracy, and significantly reduces training costs as compared with adversarial training. We use extensive empirical studies to show that the proposed RUSH outperforms robust classifiers from adversarial training, by a significant margin on common benchmarks (CIFAR-10, CIFAR-100, and STL-10) under first-order attacks. In particular, under $\ell_{\infty}$-norm perturbations of size 8/255 PGD attack on CIFAR-10, our model using ResNet-18 as backbone reached 77.8% robust accuracy and 87.9% standard accuracy. Our work has an improvement of over 15% in robust accuracy and a slight improvement in standard accuracy, compared to the state-of-the-arts.