Simple Recurrent Networks Learn Context-Free and Context-Sensitive Languages by Counting
Simple Recurrent Networks Learn Context-Free and Context-Sensitive Languages by Counting
复制标题
简单的循环网络通过计数来学习上下文无关和上下文敏感的语言
作者：
P. Rodríguez
事实证明，如果循环神经网络 (RNN) 学会处理常规语言，则可以通过将相空间区域视为 FSM 状态来提取有限状态机 (FSM)。然而，也有研究表明，可以通过使用 RNN 动力学作为计数器来构建 RNN 来实现图灵机。但是网络如何学习需要计数的语言呢？ Rodriguez、Wiles 和 Elman (1999) 表明，简单的循环网络 (SRN) 可以通过向上和向下计数来学习处理简单的上下文无关语言 (CFL)。本文对此进行了扩展，展示了一系列语言任务，其中 SRN 开发的解决方案不仅可以计数，还可以复制和存储计数信息。在一种情况下，网络像显式存储机制一样存储信息。在其他情况下，网络更间接地将信息存储在对依赖于上下文的轻微位移敏感的轨迹中。从这个意义上说，SRN 可以将模拟计算学习为一组相互依赖的计数器。这证明了 SRN 如何成为语言或序列处理的替代心理模型。
It has been shown that if a recurrent neural network (RNN) learns to process a regular language, one can extract a finite-state machine (FSM) by treating regions of phase-space as FSM states. However, it has also been shown that one can construct an RNN to implement Turing machines by using RNN dynamics as counters. But how does a network learn languages that require counting? Rodriguez, Wiles, and Elman (1999) showed that a simple recurrent network (SRN) can learn to process a simple context-free language (CFL) by counting up and down. This article extends that to show a range of language tasks in which an SRN develops solutions that not only count but also copy and store counting information. In one case, the network stores information like an explicit storage mechanism. In other cases, the network stores information more indirectly in trajectories that are sensitive to slight displacements that depend on context. In this sense, an SRN can learn analog computation as a set of interdependent counters. This demonstrates how SRNs may be an alternative psychological model of language or sequence processing.