Low-Storage Second-Order Learning Algorithms for Neural Network Modeling

研究者将研究低存储、二阶优化方法在神经网络建模中估计问题的应用。他建议对该领域研究人员可用的数值方法做出根本性贡献，并将他的方法应用于广泛使用的神经网络模型。部分基于他之前对神经网络的研究，他提议应用、修改、发展牛顿和准牛顿方法的共轭梯度和低存储变体，并在神经网络文献中的应用和标准测试问题上测试这些方法。初步的数值结果非常令人鼓舞，表明这些技术可以显著减少大型网络中学习的数值负担。他建议采用最有前途的方法（可能是低存储准牛顿）来解决生物网络和循环网络的估计问题。他建议研究混合策略，在混合策略中，他将自己的方法与在学习过程中生成底层网络的技术相结合，并研究这些技术在学习过程中的应用循环和生物网络。此外,他建议继续他之前对容错网络的研究，进一步研究该模型的应用，并开发出有效的数值方法来计算结果尺度非线性约束优化问题。

The investigator will study the application of low-storage, second- order optimization methods for estimation problems arising in neural network modeling. He proposes to make fundamental contributions to the numerical approaches available to researchers in this field and to apply his methods to widely-used neural network models. Building in part on his previous research on neural networks, he proposes to apply, modify, and develop conjugate gradient and low-storage variants of Newton and quasi-Newton methods and to test these methods on applications and standard test problems in the neural network literature. Preliminary numerical results are very encouraging and indicate that these techniques can significantly reduce the numerical burden of learning in large networks. He proposes to adapt the most promising methods (probably low-storage quasi-Newton) to solve estimation problems for biological networks and recurrent networks. He proposes to investigate hybrid strategies in which he combines his approaches with techniques that generate the underlying network during the learning process and to investigate the application of these techniques to recurrent and biological networks. Further, he proposes to continue his previous research on fault-tolerant networks by investigation further applications of this model and developing effective numerical approaches for the resulting large- scale non-linearly constrained optimization problems.