Analyzing the weight dynamics of recurrent learning algorithms
Analyzing the weight dynamics of recurrent learning algorithms
复制标题
DOI：
10.1016/j.neucom.2004.04.006
复制
发表时间：
2005-01-01
期刊：
影响因子：
6
通讯作者：
Steil, JJ
中科院分区：
文献类型：
--
作者：
Schiller, UD;Steil, JJ
通过比较真实的时间递归学习（RTRL）与一种新的连续时间在线算法，我们深入了解了递归在线训练算法的组织和动态。后者是根据Atiya和Parlos（IEEE Trans. Neural Networks 11（3）（2000）697）介绍的最近方法的精神导出的，其导致非梯度搜索方向。我们将这种方法称为Atiya-Parlos学习（APRL），并根据其最小化标准二次误差的策略对其进行解释。仿真结果表明，RTRL和APRL的不同方法导致定性不同的重量动态。APRL的单输出行为的正式分析进一步表明，权重动态有利于网络的功能分区为一个快速的输出层和一个较慢的动态水库，其权重变化率是紧密耦合的。(C)2004 Elsevier B. V.保留所有权利。
We provide insights into the organization and dynamics of recurrent online training algorithms by comparing real time recurrent learning (RTRL) with a new continuous-time online algorithm. The latter is derived in the spirit of a recent approach introduced by Atiya and Parlos (IEEE Trans. Neural Networks 11 (3) (2000) 697), which leads to non-gradient search directions. We refer to this approach as Atiya-Parlos learning (APRL) and interpret it with respect to its strategy to minimize the standard quadratic error. Simulations show that the different approaches of RTRL and APRL lead to qualitatively different weight dynamics. A formal analysis of the one-output behavior of APRL further reveals that the weight dynamics favor a functional partition of the network into a fast output layer and a slower dynamical reservoir, whose rates of weight change are closely coupled. (C) 2004 Elsevier B.V. All rights reserved.