Accelerated Vonvergence and Structure Determination of the Backpropagation Neural Network

反向传播神经网络的加速收敛和结构确定

• 批准号: 9211691
• 负责人: Luke Achenie
• 金额: $ 10万
• 依托单位: University of Connecticut
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 1992
• 资助国家: 美国
• 起止时间: 1992-08-01 至 1996-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=9211691&HistoricalAwards=false
• 关键词: Accelerated; Vonvergence; Structure; Determination; Backpropagation

Neural nets can identify and learn correlative patterns between sets of input data and corresponding target values. The most widely used neural net architecture, the back- propagation net, loosely mimics the human learning process and "learns" to recognize patterns relating input and output variables. Such nets are trained by being repeatedly fed input data together with corresponding target outcomes. After a sufficient number of training iterations, the net learns to recognize patterns in the data and, effectively, creates an internal model of the process governing the data. The net can then use this internal model to make predictions for new input conditions. Supervised training of back-propagation neural networks is usually achieved through the solution of an appropriate optimization problem. Subsequently, training times are affected by the nonlinear programming algorithms used. The training algorithm that is often used is the delta rule, which is a steepest descent derivative and as such exhibits a linear rate of convergence around a local minimum. This results in very long training time, often on the order of hours or days for practical problems. In this project the PI plans to: (1) accelerate training of the back-propagation network using Newton type algorithms, (2) determine network structure through the use of the singular value decomposition of the analytic hessian, (3) use the concept of a Minimal Spanning Network to derive a network of linear elements that will provide a performance lower bound on the neural network, and (4) impose appropriate bounds (or constraints) on design variables to enhance convergence. He hopes that the results will significantly speed up the training of neural networks. The structure of both the analytic gradients and the analytic hessian will be exploited in an implementation of the back-propagation algorithm on parallel computers, resulting in further increases in speed up. With such speed up, it will be possible to tackle difficult industrially relevant problems in a reasonable time frame.

神经网络可以识别和学习输入数据集和相应目标值之间的关联模式。最广泛使用的神经网络结构是反向传播网络，它松散地模拟人类的学习过程，并通过“学习”来识别与输入和输出变量相关的模式。这种网络是通过反复向其输入数据以及相应的目标结果来训练的。在足够数量的训练迭代之后，网络学习识别数据中的模式，并有效地创建管理数据的过程的内部模型。然后，网络可以使用这个内部模型来对新的输入条件进行预测。BP神经网络的有监督训练通常是通过求解适当的优化问题来实现的。随后，训练时间受所使用的非线性规划算法的影响。经常使用的训练算法是Delta规则，它是最陡峭的下降导数，因此在局部最小值附近表现出线性收敛速度。这导致了非常长的培训时间，对于实际问题，培训时间通常在几个小时或几天左右。在这个项目中，PI计划：(1)使用牛顿型算法加速反向传播网络的训练；(2)通过使用解析Hessian的奇异值分解来确定网络结构，(3)利用最小生成网络的概念推导出一个由线性元件组成的网络，它将提供神经网络的性能下界；(4)对设计变量施加适当的界(或约束)以增强收敛。他希望这一结果将大大加快神经网络的训练速度。在并行计算机上实现反向传播算法时，将利用解析梯度和解析赫辛的结构，从而进一步提高速度。有了这样的速度，就有可能在合理的时间框架内解决与工业有关的难题。