A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning
A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning
复制标题
DOI：
10.7146/dpb.v19i339.6570
复制
发表时间：
1990-11
期刊：
影响因子：
--
通讯作者：
M. F. Møller
中科院分区：
文献类型：
--
作者：
M. F. Møller
介绍了一种具有超线性收敛速度的有监督学习算法（Scaled Conjugate Gradient，SCG）。该算法是基于一类众所周知的优化技术在数值分析的共轭梯度法。SCG使用来自神经网络的二阶信息，但只需要O（N）的内存使用，其中N是网络中的权重数。SCG的性能基准的标准反向传播算法（BP），共轭梯度反向传播（CGB）和一步Broyden-Fletcher-Goldfarb-Shanno无记忆拟牛顿算法（BFGS）的性能。相对于BP，SCG产生至少一个数量级的加速。加速取决于收敛标准，即，对减少误差的需求越大，加速就越大。SCG是完全自动化的，不包括用户依赖的参数，并避免了耗时的线搜索，CGB和BFGS在每次迭代中使用，以确定适当的步长。在神经网络的架构中简化问题相关的结构信息通常会降低整体复杂性。相对于问题域，神经网络的复杂度越小，权值空间包含以尖锐曲率为特征的长沟壑的可能性就越大。虽然BP是低效的，这些沟壑现象，它表明，SCG处理它们有效。
A supervised learning algorithm (Scaled Conjugate Gradient, SCG) with superlinear convergence rate is introduced. The algorithm is based upon a class of optimization techniques well known in numerical analysis as the Conjugate Gradient Methods. SCG uses second order information from the neural network but requires only O(N) memory usage, where N is the number of weights in the network. The performance of SCG is benchmarked against the performance of the standard backpropagation algorithm (BP), the conjugate gradient backpropagation (CGB) and the one-step Broyden-Fletcher-Goldfarb-Shanno memoryless quasi-Newton algorithm (BFGS). SCG yields a speed-up of at least an order of magnitude relative to BP. The speed-up depends on the convergence criterion, i.e., the bigger demand for reduction in error the bigger the speed-up. SCG is fully automated including no user dependent parameters and avoids a time consuming line-search, which CGB and BFGS use in each iteration in order to determine an appropriate step size. Incorporating problem dependent structural information in the architecture of a neural network often lowers the overall complexity. The smaller the complexity of the neural network relative to the problem domain, the bigger the possibility that the weight space contains long ravines characterized by sharp curvature. While BP is inefficient on these ravine phenomena, it is shown that SCG handles them effectively.