Stochastic Quasi-Newton Methods
Stochastic Quasi-Newton Methods
复制标题
DOI：
10.1109/jproc.2020.3023660
复制
发表时间：
2020-09
影响因子：
20.6
通讯作者：
Aryan Mokhtari;Alejandro Ribeiro
中科院分区：
文献类型：
--
作者：
Aryan Mokhtari;Alejandro Ribeiro
大规模数据科学训练包含大量样品的数据集的模型。培训通常被表达为经验风险最小化问题的解决方案，这些问题是优化程序，其复杂性与数据集中的元素数量相比。随机优化方法克服了这一挑战，但它们具有自己的局限性。本文讨论了通过二阶信息的开发来加速随机优化的收敛性的最新发展。这是通过使用随机梯度信息近似目标函数的曲率的准牛顿方法的随机变体来实现的。讨论了导致更快收敛的原因以及引入增量方法，该方法利用内存以达到超线性收敛速率。这是随机优化方法的最著名收敛率。随机准Newton方法应用于几个问题，包括预测特定访问者对特定搜索引擎查询显示的广告的点击率。实验评估显示了相对于随机梯度下降算法的总体计算时间的减少。
Large-scale data science trains models for data sets containing massive numbers of samples. Training is often formulated as the solution of empirical risk minimization problems that are optimization programs whose complexity scales with the number of elements in the data set. Stochastic optimization methods overcome this challenge, but they come with their own set of limitations. This article discusses recent developments to accelerate the convergence of stochastic optimization through the exploitation of second-order information. This is achieved with stochastic variants of quasi-Newton methods that approximate the curvature of the objective function using stochastic gradient information. The reasons for why this leads to faster convergence are discussed along with the introduction of an incremental method that exploits memory to achieve a superlinear convergence rate. This is the best-known convergence rate for a stochastic optimization method. Stochastic quasi-Newton methods are applied to several problems, including prediction of the click-through rate of an advertisement displayed in response to a specific search engine query by a specific visitor. Experimental evaluations showcase reductions in overall computation time relative to stochastic gradient descent algorithms.