CARTL: Cooperative Adversarially-Robust Transfer Learning
CARTL: Cooperative Adversarially-Robust Transfer Learning
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Dian Chen;Hongxin Hu;Qian Wang;Yinli Li;Cong Wang;Chao Shen;Qi Li
中科院分区：
文献类型：
--
作者：
Dian Chen;Hongxin Hu;Qian Wang;Yinli Li;Cong Wang;Chao Shen;Qi Li
迁移学习减轻了从头开始训练性能良好的模型的负担，特别是在训练数据稀缺且计算能力有限的情况下。在深度学习中，迁移学习的典型策略是冻结预训练模型的早期层，并在目标域上微调其其余层。以前的工作重点关注转移模型的准确性，但忽略了对抗性鲁棒性的转移。在这项工作中，我们首先证明迁移学习提高了目标域的准确性，但降低了目标模型继承的鲁棒性。为了解决这个问题，我们提出了一种新颖的协作对抗鲁棒迁移学习（CARTL），通过特征距离最小化来预训练模型，并通过针对目标域任务的非扩展微调来微调预训练模型。经验结果表明，在同等准确度下，CARTL相比基线最多提高了约28%的遗传鲁棒性。此外，我们研究了迁移学习背景下批量归一化（BN）层与鲁棒性之间的关系，我们发现冻结 BN 层可以进一步提高鲁棒性迁移。
Transfer learning eases the burden of training a well-performed model from scratch, especially when training data is scarce and computation power is limited. In deep learning, a typical strategy for transfer learning is to freeze the early layers of a pre-trained model and fine-tune the rest of its layers on the target domain. Previous work focuses on the accuracy of the transferred model but neglects the transfer of adversarial robustness. In this work, we first show that transfer learning improves the accuracy on the target domain but degrades the inherited robustness of the target model. To address such a problem, we propose a novel cooperative adversarially-robust transfer learning (CARTL) by pre-training the model via feature distance minimization and fine-tuning the pre-trained model with non-expansive fine-tuning for target domain tasks. Empirical results show that CARTL improves the inherited robustness by about 28% at most compared with the baseline with the same degree of accuracy. Furthermore, we study the relationship between the batch normalization (BN) layers and the robustness in the context of transfer learning, and we reveal that freezing BN layers can further boost the robustness transfer.