Self-Organization of Hierarchical Reinforcement Learning System

在此之前，我们提出了两种学习算法，标记Q学习(LQ学习)和切换Q学习(SQ学习)。前者虽然是由单个智能体组成的结构简单的算法，但在一定的POMDP环境下可以很好地学习。后者是一种层次化Q学习方法(HQ学习)，它通过使用层次化学习自动机来改变Q-模，并且在更复杂的POMDP环境中也能很好地工作。在本研究中，我们对这两种算法进行了改进，并开发出更有效的HQ学习算法。此外，为了克服更真实的环境，其中一个或两个观测和行动是连续的值，我们进行了函数逼近的神经网络的基础研究。研究结果如下：1)改进了SQ-学习算法，使其在噪声环境下能很好地工作。2)我们通过引入Kohonen的自组织映射(SOM)提高了LQ学习的性能。3)我们改进了Sun和Sessions提出的序列自分割(SSS)算法。此外，我们还开发了一种新的算法，称为SSS(λ)。4)通过将其应用于移动机器人的导航任务，我们检验了SSS(λ)的有效性。5)提出了一种同时递归神经网络的统计逼近学习方法，并证明了该方法具有较高的非线性函数逼近精度。在此基础上，提出了一种新的增量学习神经网络模型。

Previously, we proposed two learning algorithms, Labeling Q-learning(LQ-learning) and Switching Q-learning(SQ-learning). Although the former is the algorithm of simple structure which consists of a single agent, it can learn well in a certain kind of POMDP environments. The latter is a type of hierarchical Q-learning method (HQ-learning), which changes Q-modules by using a hierarchical learning automaton, and can work well also in a more complicated POMDP environment. In this study, we improved these two algorithms, and developed more effective HQ-learning algorithms. Further, in order to overcome more realistic environments where either or both of observations and actions take continuous values, we conducted a basic study about function approximations by neural networks. The results are following.1) We improved the SQ-learning so that it works well in noisy environments. We also demonstrated that the SQ-learning exhibits a better performance than Wiering's HQ-learning.2) We enhanced the performance of the LQ-leaning by introducing the Kohonen's self-organizing map(SOM).3) We improved the self-segmentation of sequence(SSS) algorithm by Sun and Sessions. Further, we also developed a new algorithm, called SSS(λ).4) We examined the effectiveness of SSS(λ) by applying it to the navigation task of a mobile robot. Here, the SOM was used for self-classification of continuous sonar observations.5) We proposed a statistical approximation learning(SAL) for the simultaneous recurrent neural networks, and demonstrated that it achieves the high accuracy of nonlinear function approximation. Further, we presented a novel neural network model for incremental learning.