Learning to predict a context-free language: analysis of dynamics in recurrent hidden units
Learning to predict a context-free language: analysis of dynamics in recurrent hidden units
复制标题
学习预测上下文无关语言：循环隐藏单元的动态分析
DOI：
10.1049/cp:19991135
复制
发表时间：
1999
期刊：
影响因子：
--
通讯作者：
A. Blair
中科院分区：
文献类型：
--
作者：
M. Bodén;Janet Wiles;Bradley Tonkes;A. Blair
正则语言的递归神经网络处理相当好理解。最近的工作研究了不太熟悉的上下文无关语言问题。先前关于语言a（n）B（n）的结果表明，虽然小的递归网络可以处理上下文无关的语言，但学习它们是困难的。本文认为这一困难的原因，考虑网络的动态和权重空间之间的关系。我们能够表明，解决方案所需的动力学在于一个区域的权重空间接近一个分叉点，在那里的权重的微小变化可能会导致从根本上不同的网络行为。此外，我们表明，在这个区域中的误差梯度信息是高度不规则的。我们的结论是，由于空间的性质，任何基于梯度的学习方法在学习语言时都会遇到困难，并且一种更有前途的提高学习性能的方法可能是以非独立的方式进行权重改变。
Recurrent neural network processing of regular languages is reasonably well understood. Recent work has examined the less familiar question of context-free languages. Previous results regarding the language a(n)b(n) suggest that while it is possible for a small recurrent network to process context-free languages, learning them is difficult. This paper considers the reasons underlying this difficulty by considering the relationship between the dynamics of the network and weightspace. We are able to show that the dynamics required for the solution lie in a region of weightspace close to a bifurcation point where small changes in weights may result in radically different network behaviour. Furthermore, we show that the error gradient information in this region is highly irregular. We conclude that any gradient-based learning method will experience difficulty in learning the language due to the nature of the space, and that a more promising approach to improving learning performance may be to make weight changes in a non-independent manner.