An Online Method for A Class of Distributionally Robust Optimization with Non-convex Objectives
An Online Method for A Class of Distributionally Robust Optimization with Non-convex Objectives
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Qi Qi-Qi;Zhishuai Guo;Yi Xu;Rong Jin;Tianbao Yang
中科院分区：
文献类型：
--
作者：
Qi Qi-Qi;Zhishuai Guo;Yi Xu;Rong Jin;Tianbao Yang
本文提出了一种实用的在线求解一类非凸目标分布鲁棒优化问题的方法，该方法在机器学习中提高神经网络的鲁棒性方面具有重要的应用.在文献中，大多数方法用于解决DRO是基于随机原始对偶方法。然而，DRO的原始对偶方法有几个缺点：（1）操纵对应于数据大小的高维对偶变量是时间昂贵的;（2）它们对数据顺序到来的在线学习不友好。为了解决这些问题，我们考虑了一类DRO与KL发散正则化的对偶变量，转换的最小最大问题成一个组合最小化问题，并提出了实用的对偶免费在线随机方法，而不需要一个大的小批量。我们建立了国家的最先进的复杂性所提出的方法与和没有一个Polyak-\L ojasiewicz（PL）条件的目标。对大规模深度学习任务的实证研究（i）表明，我们的方法可以将训练速度提高到基线方法的2倍以上，并在具有$\sim $265K图像的大规模数据集上节省数天的训练时间，（ii）验证DRO在不平衡数据集上优于经验风险最小化（ERM）的最高性能。独立感兴趣的是，所提出的方法还可以用于解决一类具有最先进复杂性的随机组合问题。
In this paper, we propose a practical online method for solving a class of distributionally robust optimization (DRO) with non-convex objectives, which has important applications in machine learning for improving the robustness of neural networks. In the literature, most methods for solving DRO are based on stochastic primal-dual methods. However, primal-dual methods for DRO suffer from several drawbacks: (1) manipulating a high-dimensional dual variable corresponding to the size of data is time expensive; (2) they are not friendly to online learning where data is coming sequentially. To address these issues, we consider a class of DRO with an KL divergence regularization on the dual variables, transform the min-max problem into a compositional minimization problem, and propose practical duality-free online stochastic methods without requiring a large mini-batch size. We establish the state-of-the-art complexities of the proposed methods with and without a Polyak-\L ojasiewicz (PL) condition of the objective. Empirical studies on large-scale deep learning tasks (i) demonstrate that our method can speed up the training by more than 2 times than baseline methods and save days of training time on a large-scale dataset with $\sim$ 265K images, and (ii) verify the supreme performance of DRO over Empirical Risk Minimization (ERM) on imbalanced datasets. Of independent interest, the proposed method can be also used for solving a family of stochastic compositional problems with state-of-the-art complexities.