Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization
Adversarially Robust Models may not Transfer Better: Sufficient Conditions for Domain Transferability from the View of Regularization
复制标题
DOI：
--
复制
发表时间：
2022-02
期刊：
影响因子：
--
通讯作者：
Xiaojun Xu;Jacky Y. Zhang;Evelyn Ma;Danny Son;Oluwasanmi Koyejo;Bo Li
中科院分区：
文献类型：
--
作者：
Xiaojun Xu;Jacky Y. Zhang;Evelyn Ma;Danny Son;Oluwasanmi Koyejo;Bo Li
机器学习(ML)的稳健性和领域泛化从根本上是相关的：它们本质上分别涉及对抗性和自然环境下的数据分布变化。一方面，最近的研究表明，更健壮的(对抗性训练的)模型更具普遍性。另一方面，对它们之间的基本联系缺乏理论上的理解。在本文中，我们探讨了正则化和域可转移性之间的关系，考虑了不同的因素，如范数正则化和数据扩充(DA)。我们提出了一个一般的理论框架，证明了涉及模型函数类正则化的因素是相对区域可转移的充分条件。我们的分析表明，稳健性既不是可转移性的必要条件，也不是可转移性的充分条件；相反，正则化是理解领域可转移性的更基本的角度。然后，我们讨论了流行的DA协议(包括对抗性训练)，并展示了它们在特定条件下何时可以被视为函数类的正则化，从而提高了泛化能力。我们通过大量的实验来验证我们的理论发现，并给出了在不同数据集上健壮性和泛化负相关的反例。
Machine learning (ML) robustness and domain generalization are fundamentally correlated: they essentially concern data distribution shifts under adversarial and natural settings, respectively. On one hand, recent studies show that more robust (adversarially trained) models are more generalizable. On the other hand, there is a lack of theoretical understanding of their fundamental connections. In this paper, we explore the relationship between regularization and domain transferability considering different factors such as norm regularization and data augmentations (DA). We propose a general theoretical framework proving that factors involving the model function class regularization are sufficient conditions for relative domain transferability. Our analysis implies that ``robustness"is neither necessary nor sufficient for transferability; rather, regularization is a more fundamental perspective for understanding domain transferability. We then discuss popular DA protocols (including adversarial training) and show when they can be viewed as the function class regularization under certain conditions and therefore improve generalization. We conduct extensive experiments to verify our theoretical findings and show several counterexamples where robustness and generalization are negatively correlated on different datasets.