The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction
The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction
复制标题
DOI：
10.1162/neco.1996.8.6.1135
复制
发表时间：
1996-08
影响因子：
2.9
通讯作者：
Michael Casey
中科院分区：
文献类型：
--
作者：
Michael Casey
递归神经网络（RNN）可以学习执行有限状态计算。它表明，一个RNN执行有限状态计算必须组织其状态空间，以模仿的最小确定性有限状态机，可以执行该计算的状态，并给出了这样的系统的吸引子结构的精确描述。这些知识有效地预测了激活空间动态，这使得人们能够理解RNN计算动态，尽管激活动态很复杂。该理论为理解有限状态机（FSM）提取技术提供了一个理论框架，并可用于改进执行FSM计算的RNN的训练方法。这提供了一个成功的方法来理解尚未明确设计的复杂系统的一般类别的例子，例如，系统已经进化或学习了它们的内部结构。
Recurrent neural networks (RNNs) can learn to perform finite state computations. It is shown that an RNN performing a finite state computation must organize its state space to mimic the states in the minimal deterministic finite state machine that can perform that computation, and a precise description of the attractor structure of such systems is given. This knowledge effectively predicts activation space dynamics, which allows one to understand RNN computation dynamics in spite of complexity in activation dynamics. This theory provides a theoretical framework for understanding finite state machine (FSM) extraction techniques and can be used to improve training methods for RNNs performing FSM computations. This provides an example of a successful approach to understanding a general class of complex systems that has not been explicitly designed, e.g., systems that have evolved or learned their internal structure.