Stochastic Block BFGS: Squeezing More Curvature out of Data
Stochastic Block BFGS: Squeezing More Curvature out of Data
复制标题
DOI：
--
复制
发表时间：
2016-03
期刊：
影响因子：
--
通讯作者：
R. Gower;D. Goldfarb;Peter Richtárik
中科院分区：
文献类型：
--
作者：
R. Gower;D. Goldfarb;Peter Richtárik
我们提出了一种新的有限记忆随机块BFGS更新，将丰富的曲率信息的随机逼近方法。在我们的方法中，由它维护的逆Hessian矩阵的估计在每次迭代时使用Hessian的草图更新，即，一个随机生成的压缩形式的海森。我们提出了几种草图策略，提出了一种新的拟牛顿方法，使用随机块BFGS更新结合方差减少方法SVRG计算批量随机梯度，并证明了线性收敛的方法。大规模逻辑回归问题的数值测试表明，我们的方法是更强大的，大大优于目前最先进的方法。
We propose a novel limited-memory stochastic block BFGS update for incorporating enriched curvature information in stochastic approximation methods. In our method, the estimate of the inverse Hessian matrix that is maintained by it, is updated at each iteration using a sketch of the Hessian, i.e., a randomly generated compressed form of the Hessian. We propose several sketching strategies, present a new quasi-Newton method that uses stochastic block BFGS updates combined with the variance reduction approach SVRG to compute batch stochastic gradients, and prove linear convergence of the resulting method. Numerical tests on large-scale logistic regression problems reveal that our method is more robust and substantially outperforms current state-of-the-art methods.