Symbol Processing System Modeled after Brains

在研究过程中，我们遇到了令人困惑的实验结果，这意味着递归神经网络（RNN）的表示能力比通常认为的要有限。这些结果是令人困惑的，因为它们表现出，例如，可学习性虽然有限和不稳定的学习结果。我们做了进一步的调查，以获得结果，以显示为什么RNN学习是可能的，以及规避能力不足的方法（a）如果要求噪声容忍，那么一般计数器是不可学习的，因此堆栈也不可学习。基于这些结果，我们提出了一种单圈计数器，当它向下计数一次时不能向上计数，并且我们建设性地表明，单圈计数器和以同样的方式有限圈计数器是可实现的，但无限圈计数器不是。因此，我们证明了RNN至多可以表示一个具有有限圈计数器的有限状态自动机，并且实验结果显示了RNN的可学习性。 关于我们 ty计数器实际上最多显示有限圈计数器的可学习性，而不是计数器的可学习性（B）理论上，如果没有合适的学习偏置，有限状态自动机不能被学习，并且在RNN的情况下，一般来说，不可能证明或反驳RNN中两个学习自动机的等价性。提出了一种新的以经典感知器为计算单元的RNN随机学习算法。由算法自然引入的偏差使得学习有限状态自动机成为可能。由于RNN所表示的状态转移是有限空间的，所以我们可以保证得到这种类型的RNN的有限状态自动机表示。如果存在解，则算法保证以概率1收敛，尽管期望的收敛时间可能是无限的。（c）我们描述了由有限状态自动机生成的语言，其中有限0或单圈计数器。这些语言形成了一种不同于乔姆斯基等级的等级结构

During the research, we encountered puzzling experimental results that would imply that the representation capability of the recurrent neural networks (RNN) is limited further than usually believed. Those results were puzzling because they exhibit, for example, learnability althogh limited and unstable learned results. We made further investigation to obtain results to show why the RNN learning is possible and methods to circumvent the insufficent capability(a) If noise-tolerance is requested, then general counters are not learnable, therefore stacks are not learnable either. Based upon the results, we proposed a single-turn counter that cannot count-up when it counts down once and we showed constructively that the single-turn counter and in the same way finite-turn counter is implementable but the infinite turn counter is not. In consequnence we showed that RNN can represent at most a finite state automaton with finite-turn counters and that the experimental results showing learnabili … More ty of counters are in fact showing at most the learnability of finite-turn counters and not that of counters(b) Theoretically a finite state automaon cannot be learned without a suitable learning bias and in RNN cases it is impossible to prove or disprove in general the equivalence of two learned automaton in the RNN. We proposed a new stochastic learning alogrithm of RNN with classical perceptrons as its computation units. A bias naturally introduced by the algorithm make it possible to learn a finite state automaton. Since the state transition represented by the RNN is of finite space, it is guranteed for us to get a finite state automaton representaion ofRNN of the type. The algorithm is guranteed to converge with probability one if a solution exists, although the expected time to convergence might be infinite(c) We characterized the languages generated by a finite state automaton with finite0 or single-turn counters. The languages form a hierarchical structure different from Chomsky's hierarchy Less