Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations
Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations
复制标题
DOI：
10.1080/10556788.2019.1624747
复制
发表时间：
2018-07
影响因子：
2.2
通讯作者：
Jennifer B. Erway;J. Griffin;Roummel F. Marcia;Riadh Omheni
中科院分区：
文献类型：
--
作者：
Jennifer B. Erway;J. Griffin;Roummel F. Marcia;Riadh Omheni
摘要机器学习（ML）问题通常是高度非线性、非凸的无约束优化问题。基于随机梯度下降的ML问题求解方法很容易扩展到非常大的问题，但可能涉及微调许多超参数。基于有限内存Broyden-Fletcher-Goldfarb-Shanno（BFGS）更新的拟牛顿方法通常不需要手动调整超参数，但会受到用正定矩阵近似潜在不确定Hessian的影响。Hessian-free方法利用了执行Hessian-vector乘法而不需要整个Hessian矩阵的能力，但每次迭代的复杂度明显大于拟牛顿方法。在本文中，我们提出了一种基于拟牛顿信赖域框架的解决ML问题的替代方法，用于解决允许不定Hessian近似的大规模优化问题。在标准测试数据集上的数值实验表明，在固定的计算时间预算下，所提出的方法比传统的有限内存BFGS方法和Hessian-free方法获得了更好的结果.
ABSTRACT Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may involve fine-tuning many hyper-parameters. Quasi-Newton approaches based on the limited-memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) update typically do not require manually tuning hyper-parameters but suffer from approximating a potentially indefinite Hessian with a positive-definite matrix. Hessian-free methods leverage the ability to perform Hessian-vector multiplication without needing the entire Hessian matrix, but each iteration's complexity is significantly greater than quasi-Newton methods. In this paper we propose an alternative approach for solving ML problems based on a quasi-Newton trust-region framework for solving large-scale optimization problems that allow for indefinite Hessian approximations. Numerical experiments on a standard testing data set show that with a fixed computational time budget, the proposed methods achieve better results than the traditional limited-memory BFGS and the Hessian-free methods.