A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization
A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization
复制标题
在线和随机优化中自适应正则化的统一方法
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Y. Singer
中科院分区：
文献类型：
--
作者：
Vineet Gupta;Tomer Koren;Y. Singer
我们描述了一个用于导出和分析在线优化算法的框架，该算法结合了自适应、数据相关的正则化（也称为预处理）。事实证明，此类算法通过根据数据的几何形状重塑梯度，在随机优化中非常有用。我们的框架捕获并统一了许多有关自适应在线方法的现有文献，包括 AdaGrad 和在线牛顿步算法及其对角版本。因此，我们获得了这些算法的新收敛证明，这些证明比以前的分析要简单得多。我们的框架还揭示了常见随机优化方法中使用的不同预处理更新的基本原理。
We describe a framework for deriving and analyzing online optimization algorithms that incorporate adaptive, data-dependent regularization, also termed preconditioning. Such algorithms have been proven useful in stochastic optimization by reshaping the gradients according to the geometry of the data. Our framework captures and unifies much of the existing literature on adaptive online methods, including the AdaGrad and Online Newton Step algorithms as well as their diagonal versions. As a result, we obtain new convergence proofs for these algorithms that are substantially simpler than previous analyses. Our framework also exposes the rationale for the different preconditioned updates used in common stochastic optimization methods.