KDI: Temporal Abstraction in Reinforcement Learning

这个项目研究了一种在多个时间抽象级别上学习、计划和表示知识的新方法。它开发了人工强化学习系统可以对持续的行动过程进行建模和推理并以相应的术语感知其环境的方法，并开发和检查了与此方法相关的动物行为模型的有效性。该项目的目标是发展该方法的数学理论，改进、扩展和进行相关动物行为模型的验证研究，检查该理论与控制理论和人工智能的关系，并在一些模拟学习任务中证明其有效性。目前的强化学习(RL)研究使用的框架是，代理必须采取一系列行动的单个固定时间步骤：行动采取一步完成，其直接结果在一步后可用(建模为马尔可夫决策过程，或MDP)。这使得在不同的时间尺度上学习和计划变得困难。一些RL研究转而使用此框架的一般化(半马尔可夫决策过程，或SMDP)，其中操作需要不同的时间来完成，并且现有理论指定如何对这些操作的结果进行建模以及如何对其进行规划。然而，这种方法是有限的，因为时间上扩展的动作被视为不可分割和未知的单元。为了获得最大的灵活性和最佳性能，有必要研究临时扩展的操作，以检查或修改它们是如何由较低级别的操作组成的，这在现有方法中没有考虑到。相比之下，该项目将扩展的操作过程建模为覆盖在基本MDP上的SMDP操作。然后，可以将这些称为选项的操作过程视为原始操作；可以几乎不变地应用现有的RL。这种方法使备选方案能够在MDP和SMDP两个级别上进行分析，并在这两个级别之间的界面上引入了新的问题。这种方法很有吸引力，因为它简单，与以前使用原始动作的方法相似，并且在MDP和SMDP理论中有坚实的数学基础。这正在进一步发展成为分级和多时间尺度规划和学习的一般方法。

This project investigates a new approach to learning, planning, and representing knowledge at multiple levels of temporal abstraction. It develops methods by which an artificial reinforcement learning system can model and reason about persistent courses of action and perceive its environment in corresponding terms, and it develops and examines the validity of models of animal behavior related to this approach. The project's objectives are to develop the mathematical theory of the approach, to refine, extend, and conduct validation studies of related models of animal behavior, to examine the theory's relationship to control theory and artificial intelligence, and to demonstrate its effectiveness in a number of simulated learning tasks.Most current reinforcement learning (RL) research uses a framework in which an agent has to take a sequence of actions paced by a single, fixed time step: actions take one step to complete, and their immediate consequences become available after one step (modeled as a Markov decision process, or MDP). This makes it difficult to learn and plan at different time scales. Some RL research instead uses a generalization of this framework (semi-Markov decision processes, or SMDPs) in which actions take varying amounts of time to complete, and the existing theory specifies how to model the results of these actions and how to plan with them. However, this approach is limited because temporally extended actions are treated as indivisible and unknown units. For the greatest flexibility and best performance, it is necessary to look inside temporally extended actions to examine or modify how they are comprised of lower-level actions, which is not considered in existing approaches.This project, by contrast, will model extended courses of action as SMDP actions overlaid upon a base MDP. These courses of action, called options, can then be treated as if they were primitive actions; existing RL can be applied almost unchanged. This approach enables options to be analyzed at both the MDP and SMDP levels and introduces new issues at the interface between the levels. This approach is appealing because of its simplicity, similarity to previous approaches using primitive actions, and its solid mathematical foundation in MDP and SMDP theory. This is being developed further into a general approach to hierarchical and multi-time-scale planning and learning.