The importance of better models in stochastic optimization
The importance of better models in stochastic optimization
复制标题
DOI：
10.1073/pnas.1908018116
复制
发表时间：
2019-11-12
影响因子：
11.1
通讯作者：
Duchi, John C.
中科院分区：
文献类型：
--
作者：
Asi, Hilal;Duchi, John C.
标准随机优化方法很脆弱，对步长选择和其他算法参数敏感，并且在表现良好的目标族之外表现出不稳定性。为了应对这些挑战，我们研究了随机优化和学习问题的模型，这些模型对问题族和算法参数表现出更好的鲁棒性。通过适当准确的模型（我们称之为 APROX 系列），随机方法可以变得稳定、可证明收敛且渐近最优；即使目标是非负的建模也足以实现这种稳定性。我们将这些结果扩展到凸性之外的弱凸目标，其中包括凸损失与现代机器学习中常见的平滑函数的组合。我们通过收敛时间和算法灵敏度的实验评估来强调稳健性和准确建模的重要性。
Standard stochastic optimization methods are brittle, sensitive to stepsize choice and other algorithmic parameters, and they exhibit instability outside of well-behaved families of objectives. To address these challenges, we investigate models for stochastic optimization and learning problems that exhibit better robustness to problem families and algorithmic parameters. With appropriately accurate models-which we call the APROX family-stochastic methods can be made stable, provably convergent, and asymptotically optimal; even modeling that the objective is non-negative is sufficient for this stability. We extend these results beyond convexity to weakly convex objectives, which include compositions of convex losses with smooth functions common in modern machine learning. We highlight the importance of robustness and accurate modeling with experimental evaluation of convergence time and algorithm sensitivity.