Self-control of Memory Structure of Reinforcement Learning in Hidden Markov Environments

隐马尔可夫环境下强化学习记忆结构的自我控制

• 批准号: 11650441
• 负责人: ABE Kenichi
• 金额: $ 2.24万
• 依托单位: Tohoku University
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (C)
• 财政年份: 1999
• 资助国家: 日本
• 起止时间: 1999 至 2000
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-11650441/
• 关键词: Hidden MARKOV; Reinforcement Learning; Q-Learning; Labeling Q-learning; Learning Automaton; Swiching Q-Learning; Hierarchical Q-Learning; スイッチングQ-学習

Recent research on reinforcement learning (RL) algorithms has concentrated on partially observable Markov decision problems (POMDPs). A possible solution to POMDPs is to use history information to estimate state. Q values must be updated in the form reflecting past history of observation/action pairs. In this study, we developed two methods of reinforcement learning, which can solve certain types of POMDPs. The results are summarized as follows :(1) As a result of last Grant-in-Aid for Scientific Research (C)(2), we proposed Labeling Q-learning (LQ-learning), which has a new memory architecture of handling past history. In this study, we established a general framework of the LQ-learning. Various algorithms in this framework were devised, and we gave comparative study between these through simulation. The above LQ-learning, however, has the drawback that we must predefine the labeling mechanism. To overcome this drawback, we further devised a SOM (self-organizing feature map) approach of labeling, in which past history of observation/action pairs are partitioned into classes. The SOM has one-dimensional structure and the output nodes of the SOM produce labels.(2) We proposed a new type of hierarchical RL, called Switching Q-learning (SQ-learning). The basic idea of SQ-learning is that non-Markovian tasks can be automatically decomposed into subtasks solvable by memoryless policies, without any other information leading to "good" subgoals. To deal with such decomposition, SQ-learning employs ordered sequences of Q-modules in which each module discovers a local control policy. SQ-learning uses a hierarchical system of learning automata for switching module. The simulation results demonstrate that SQ-learning has the ability to quickly learn optimal or near-optimal policies without huge computational burden.It is a future work to build a unified view by which LQ-learning and SQ-learning can be dealt with systematically.

强化学习（RL）算法的最新研究集中在部分可观测马尔可夫决策问题（POMDPs）。POMDP的一个可能的解决方案是使用历史信息来估计状态。Q值必须以反映观察/行动对过去历史的形式更新。在这项研究中，我们开发了两种强化学习方法，可以解决某些类型的POMDPs。（1）作为上一个科学研究资助项目（C）（2）的结果，我们提出了标记Q学习（LQ-learning），它具有处理过去历史的新的记忆结构。在本研究中，我们建立了一个LQ学习的一般框架。在此框架下设计了各种算法，并通过仿真对这些算法进行了比较研究。然而，上面的LQ学习有一个缺点，那就是我们必须对标记机制进行改进。为了克服这个缺点，我们进一步设计了一个SOM（自组织特征映射）的标记方法，在过去的历史观察/动作对划分为类。SOM具有一维结构，SOM的输出节点产生标签。(2)我们提出了一种新型的分层RL，称为开关Q学习（SQ学习）。SQ-learning的基本思想是，非马尔科夫任务可以自动分解为可通过无记忆策略解决的子任务，而无需任何其他信息即可实现“好”的子目标。为了处理这种分解，SQ学习采用Q模块的有序序列，其中每个模块发现局部控制策略。SQ-learning使用一个分层的学习自动机系统作为切换模块。仿真结果表明，SQ-learning具有快速学习最优或接近最优策略的能力，且计算量小，因此建立统一的观点来系统地处理LQ-learning和SQ-learning是未来的工作。

项目成果

Hae Yeon Lee: "Labeling Q-learning for Maze Problems with Partially Observable States"Proc.of 15th Korea Automatic Control Conference. Vol.2. 484-487 (2000)

Hae Yeon Lee：“Labeling Q-learning for Maze Problems with Partially Observable States”，第 15 届韩国自动控制会议论文集。

Haeyon Lee: "Labeling Q-Learning for Partially Observable Markov Decision Process Environments"Proc.of Fifth Int.Symp.on Artificial Life and Robtics. 484-490 (2000)

Haeyon Lee：“Labeling Q-Learning for Partially Observable Markov Decision Process Environmentals”Proc.of Fifth Int.Symp.on Artificial Life and Robtics。

HaeYeon Lee: "Labeling Q-Learning For Non-Markovian Environments"1999 IEEE International Conference on SMC. Vol.V. 487-491 (1999)

HaeYeon Lee：“为非马尔可夫环境标记 Q 学习”1999 年 IEEE 国际 SMC 会议。

HaeYeon Lee: "Labeling Q-learning for partially observable markov decision process environments"AROB 5th '00. Vol.2. 281-284 (2000)

HaeYeon Lee：“为部分可观察的马尔可夫决策过程环境标记 Q 学习”AROB 5th 00。

Ikuo Yoshihara: "Extending prediction term of GP-based time series model"AROB 5th '00. Vol.1. 268-271 (2000)

Ikuo Yoshihara：“扩展基于 GP 的时间序列模型的预测项”AROB 5th 00。