Open-Ended Discovery of Skill Hierarchies in Artificial Intelligence

人们每天通过将复杂的任务分解成较小的子任务来解决这些任务。例如，泡茶的任务可以分解为烧壶、加糖、加茶包、抓杯等子任务。这些子问题本身可以分解成更小的子问题，一直到涉及的单个肌肉运动--形成了一个有助于解决问题的技能等级。当然，学习如何在肌肉运动的规模上泡茶将是一项不合理的巨大计算任务--我们解决问题的能力很大程度上归因于我们发现和计划使用分层组织的更高级别行为的能力。规划和学习几个高级别行为的序列显然比规划和学习可能数百万个原始动作的序列计算成本要低。两个关键的开放研究问题是，这些有用的技能应该如何描述，以及人工智能代理应该如何自主发现它们。我们希望至少部分地在这个研究项目中解决这些问题。我们在成熟的强化学习(RL)框架内进行了这项研究，该框架广泛地关注人工智能代理应该如何通过与环境的交互来学习最优的行为策略。许多RL方法，甚至是那些被认为是最先进的方法，都是使用原始的动作--可以说，他们仍然在按照肌肉运动的规模制作茶杯。RL的一个分支被称为层次强化学习，它考虑了在不同的时间步长上采取的更高级别的行为，参考了技能如何被分层组织。显然，本研究项目的主要目标是开发一个HRL算法或HRL算法集，它赋予人工智能代理通过与其环境的交互来发现有用的高级行为的层次结构的能力。在本项目的过程中开发的算法(S)应该具有几个期望的属性。首先，算法应该是发展的，随着时间的推移，更高级别的、更复杂的技能应该从较低级别的技能分层构建。其次，算法应该是与域无关的，以确保它们适用于多种类型的问题。第三，如果开发的算法在目前被认为是困难的任务(例如，像Montezuma的复仇游戏这样的“艰难探索”问题)中表现良好，这将是一个理想的结果。例如，许多现有的HRL方法只适用于离散的域，那些具有小状态空间的方法，或者其他不能很好地扩展到更大的域或具有连续状态空间的域的方法。这将它们的适用性限制在许多有趣的、复杂的问题上，这些问题最终是RR感兴趣的。开发这样的算法的好处将是广泛的-允许强化学习应用于更大、更复杂的问题，而当前的方法只是不能很好地扩展到这些问题。

People solve complex tasks every day by decomposing them into smaller sub-tasks. For instance, the task of making a cup of tea can be decomposed into the sub-tasks of boiling the kettle, adding sugar, adding a tea-bag, grasping the cup, and so on. These sub-problems can themselves be decomposed into even smaller sub-problems, all the way down to the individual muscle movements involved - forming a hierarchy of skills useful for solving the problem.Of course, learning how to make a cup of tea at the scale of muscle movements would be an unreasonably large computational undertaking - much of our problem-solving ability is attributed to our ability to discover and plan using hierarchically organised higher-level behaviours. Planning and learning a sequence of a few high-level behaviours is clearly less computationally expensive than planning and learning a sequence of perhaps millions of primitive actions.Two key open research questions are how such useful skills should be characterised, and how an artificially intelligent agent should go about autonomously discovering them. It is these questions that we hope, at least in part, to address during this research project.We frame this research within the well-developed framework of Reinforcement Learning (RL), which concerns itself broadly with how artificially intelligent agents should learn optimal behavioural policies through interaction with their environments. Many RL methods, even those considered state-of-the-art, operate using primitive actions - they are still making cups of tea operating on the scale of muscle movements, as it were. The branch of RL which considers higher-level behaviours taken over varying timesteps is known as Hierarchical Reinforcement Learning (HRL), in reference to how skills can be organised hierarchically.Explicitly, the main objective of this research project is to develop an HRL algorithm, or set of HRL algorithms, which endow artificially intelligent agents with the ability to discover a hierarchy of useful high-level behaviours through interaction with their environment.There are several desirable properties that the algorithm(s) developed over the course of this project should possess. Firstly, the algorithms should be developmental, with higher-level, more complex skills being constructed hierarchically from lower-level ones as time goes on. Secondly, the algorithms should be domain-independent to ensure their applicability to many types of problem. Thirdly, it would be a desirable outcome if the algorithms developed performed well in tasks which are currently considered difficult (e.g. "hard exploration" problems such as the game of Montezuma's Revenge).These desirable properties stem partly from various shortcomings in current HRL methods. For instance, many existing HRL methods are applicable only in discrete domains, those with small state-spaces, or otherwise would not scale well to larger domains or those with continuous state-spaces. This limits their applicability to many of the interesting, complex problems that are ultimately of interest to RL.The benefits of developing such algorithms would be wide-ranging - allowing reinforcement learning to be applied to larger, more complex problems to which current method simply do not scale well.