On the regularized risk of distributionally robust learning over deep neural networks
On the regularized risk of distributionally robust learning over deep neural networks
复制标题
DOI：
10.1007/s40687-022-00349-9
复制
发表时间：
2021-09
影响因子：
1.2
通讯作者：
Camilo A. Garcia Trillos;N. G. Trillos
中科院分区：
文献类型：
--
作者：
Camilo A. Garcia Trillos;N. G. Trillos
在本文中，我们探讨了分布鲁棒学习和不同形式的正则化之间的关系，以增强深度神经网络的鲁棒性。特别地，我们从一个具体的最小-最大分布鲁棒问题出发，利用最优输运理论的工具，用适当的正则化风险最小化问题推导出分布鲁棒问题的一阶和二阶逼近。在深度ResNet模型的背景下，我们将产生的正则化问题的结构识别为平均场最优控制问题，其中状态变量的数量和维数在原始非鲁棒问题维数的无维因子内。利用与这些问题相关的庞特里亚金最大值原理，我们激发了一系列可扩展的算法来训练鲁棒神经网络。我们的分析恢复了文献中已知的一些结果和算法（在整篇论文中解释的设置中），并提供了许多其他理论和算法的见解，据我们所知是新颖的。在我们的分析中，我们使用了我们认为对未来分析更一般的对抗性学习问题有用的工具。
In this paper, we explore the relation between distributionally robust learning and different forms of regularization to enforce robustness of deep neural networks. In particular, starting from a concrete min-max distributionally robust problem, and using tools from optimal transport theory, we derive first-order and second-order approximations to the distributionally robust problem in terms of appropriate regularized risk minimization problems. In the context of deep ResNet models, we identify the structure of the resulting regularization problems as mean-field optimal control problems where the number and dimension of state variables are within a dimension-free factor of the dimension of the original unrobust problem. Using the Pontryagin maximum principles associated with these problems, we motivate a family of scalable algorithms for the training of robust neural networks. Our analysis recovers some results and algorithms known in the literature (in settings explained throughout the paper) and provides many other theoretical and algorithmic insights that to our knowledge are novel. In our analysis, we employ tools that we deem useful for a future analysis of more general adversarial learning problems.