Symbol Processing System Modeled after Brains

在研究过程中，我们遇到了令人费解的实验结果，这意味着递归神经网络(RNN)的表示能力比通常认为的更有限。这些结果令人费解，因为它们展示了，例如，尽管学习结果有限且不稳定，但可学习性。我们做了进一步的研究，以显示为什么RNN学习是可能的，以及规避能力不足的方法：(A)如果要求容忍噪声，那么普通计数器是不可学习的，因此堆栈也是不可学习的。在此基础上，我们提出了一种单圈计数器，该计数器在倒计时一次时不能递增计数，并建设性地证明了单圈计数器和有限圈计数器是可以实现的，而无限圈计数器是不能实现的。在序列中，我们证明了RNN最多只能表示一个具有有限圈计数器的有限状态自动机，并且实验结果表明它是可学习的…更多类型的计数器实际上至多显示出有限圈计数器的可学习性，而不是计数器的可学习性。(B)理论上，如果没有合适的学习偏差，有限状态自动机是不能学习的，并且在RNN情况下，不可能证明或反驳RNN中两个学习自动机的等价性。提出了一种新的以经典感知器为计算单元的随机神经网络学习算法。算法自然引入的偏差使学习有限状态自动机成为可能。由于RNN所表示的状态转移是有限空间的，因此我们需要得到这类RNN的有限状态自动机表示。该算法保证在解存在的情况下以概率1收敛，尽管收敛的预期时间可能是无限的。(C)我们刻画了由有限状态自动机生成的语言。语言形成了一种不同于乔姆斯基的等级制度的等级结构

During the research, we encountered puzzling experimental results that would imply that the representation capability of the recurrent neural networks (RNN) is limited further than usually believed. Those results were puzzling because they exhibit, for example, learnability althogh limited and unstable learned results. We made further investigation to obtain results to show why the RNN learning is possible and methods to circumvent the insufficent capability(a) If noise-tolerance is requested, then general counters are not learnable, therefore stacks are not learnable either. Based upon the results, we proposed a single-turn counter that cannot count-up when it counts down once and we showed constructively that the single-turn counter and in the same way finite-turn counter is implementable but the infinite turn counter is not. In consequnence we showed that RNN can represent at most a finite state automaton with finite-turn counters and that the experimental results showing learnabili … More ty of counters are in fact showing at most the learnability of finite-turn counters and not that of counters(b) Theoretically a finite state automaon cannot be learned without a suitable learning bias and in RNN cases it is impossible to prove or disprove in general the equivalence of two learned automaton in the RNN. We proposed a new stochastic learning alogrithm of RNN with classical perceptrons as its computation units. A bias naturally introduced by the algorithm make it possible to learn a finite state automaton. Since the state transition represented by the RNN is of finite space, it is guranteed for us to get a finite state automaton representaion ofRNN of the type. The algorithm is guranteed to converge with probability one if a solution exists, although the expected time to convergence might be infinite(c) We characterized the languages generated by a finite state automaton with finite0 or single-turn counters. The languages form a hierarchical structure different from Chomsky's hierarchy Less