Riemannian metrics for neural networks II: recurrent networks and learning symbolic data sequences
Riemannian metrics for neural networks II: recurrent networks and learning symbolic data sequences
复制标题
神经网络的黎曼度量 II：循环网络和学习符号数据序列
DOI：
--
复制
发表时间：
2013
期刊：
影响因子：
--
通讯作者：
Y. Ollivier
中科院分区：
文献类型：
--
作者：
Y. Ollivier
递归神经网络是序列数据的强大模型，能够
在序列中表示复杂的依赖关系，
马尔可夫模型无法处理。然而，他们是出了名的难以训练。
在这里，我们介绍一个训练过程，使用梯度上升，
黎曼度量：这产生一个
算法独立于设计选择，例如
参数和单元活动的编码。这个公制梯度上升是
设计成具有接近反向传播的算法成本，
时间对于稀疏连接的网络。
我们将此过程用于门控漏神经网络（GLNN），
一种递归神经网络的变体，
受到有限自动机和进化方程的启发，
连续时间网络
用黎曼梯度训练的GLNN被证明可以有效地捕获各种各样的
合成问题中的结构：基本块嵌套
与上下文无关语法（自然语言的一个重要特征，
但很难学习），多个独立的交叉点
马尔可夫型关系，或长距离关系，如
距离异或问题
该方法不需要调整网络结构或初始化
参数：使用的网络是稀疏随机图，
对于所有考虑的问题，初始化是相同的。
Recurrent neural networks are powerful models for sequential data, able
to represent complex dependencies in the sequence that simpler models such as hidden
Markov models cannot handle. Yet they are notoriously hard to train.
Here we introduce a training procedure using a gradient ascent in a
Riemannian metric: this produces an
algorithm independent from design choices such as the
encoding of parameters and unit activities. This metric gradient ascent is
designed to have an algorithmic cost close to backpropagation through
time for sparsely connected networks.
We use this procedure on emph{gated leaky neural networks} (GLNNs),
a variant of recurrent neural networks with an architecture
inspired by finite automata and an evolution equation inspired by
continuous-time networks.
GLNNs trained with a Riemannian gradient are demonstrated to effectively capture a variety of
structures in synthetic problems: basic block nesting
as in context-free grammars (an important feature of natural languages,
but difficult to learn), intersections of multiple independent
Markov-type relations, or long-distance relationships such as the
distant-XOR problem.
This method does not require adjusting the network structure or initial
parameters: the network used is a sparse random graph and the
initialization is identical for all problems considered.