An Online Method for Distributionally Deep Robust Optimization
An Online Method for Distributionally Deep Robust Optimization
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Qi Qi-Qi;Zhishuai Guo;Yi Xu;Rong Jin;Tianbao Yang
中科院分区：
文献类型：
--
作者：
Qi Qi-Qi;Zhishuai Guo;Yi Xu;Rong Jin;Tianbao Yang
本文提出了一种实用的在线求解深度学习分布稳健优化问题的方法，该方法对于提高神经网络的鲁棒性在机器学习中具有重要的应用。在文献中，大多数求解DRO的方法都是基于随机原-对偶方法。然而，用于深度DRO的原始-对偶方法有几个缺点：(1)处理与数据大小对应的高维对偶变量的时间开销很大；(2)它们不利于数据按顺序到来的在线学习。为了解决这些问题，我们将最小-最大公式转化为最小化公式，并提出了一种实用的KL散度正则化求解深度DRO的无对偶在线随机方法。所提出的在线随机方法在几个方面类似于广泛用于学习深度神经网络的实用随机Nester-ovs方法。在Polyak-Lojasiewicz(PL)条件下，我们证明了该方法可以在不要求大批量的情况下获得最优的样本复杂度。该方法具有独立意义，也可用于求解一类随机合成问题。
In this paper, we propose a practical online method for solving a distributionally robust optimization (DRO) for deep learning, which has important applications in machine learning for improving the robustness of neural networks. In the literature, most methods for solving DRO are based on stochastic primal-dual methods. However, primal-dual methods for deep DRO suffer from several drawbacks: (1) manipulating a high-dimensional dual variable corresponding to the size of data is time expensive; (2) they are not friendly to online learning where data is coming sequentially. To address these issues, we transform the min-max formulation into a minimization formulation and propose a practical duality-free online stochastic method for solving deep DRO with KL divergence regularization. The proposed online stochastic method resembles the practical stochastic Nesterovs method in several perspectives that are widely used for learning deep neural networks. Under a Polyak-Lojasiewicz (PL) condition, we prove that the proposed method can enjoy an optimal sample complexity without any requirements on large batch size. Of independent interest, the proposed method can be also used for solving a family of stochastic compositional problems.