Compositional Causal Model-based Reinforcement Learning

人工智能最重要的、尚未解决的问题之一是建立具有类似人类的创造力、好奇心、自我评估和常识性推理的代理。最近，无模型强化学习（MFRL）在视频游戏和使用深度神经网络的运动控制方面表现出了令人印象深刻的表现。尽管它们取得了成功，但MFRL方法从根本上受到其试验和错误性质的限制，这需要数百万个训练示例来学习可靠的策略。另一方面，基于模型的强化学习（MBRL）智能体能够通过深思熟虑的推理来实现其目标。与无模型智能体不同，MBRL智能体迭代学习世界模型，并根据世界模型计划其行动。MBRL具有很大的吸引力，因为学习模型允许代理预测其未来并对其自身行为的后果进行推理。

One of the most important, unsolved problems of artificial intelligence is to build agents with human-like creativity, curiosity, self-assessment, and commonsense reasoning. Recently, model--free reinforcement learning (MFRL) has shown impressive performance in video game playing and locomotion controls using deep neural networks. Despite their success, MFRL methods are fundamentally limited by their trial--and--error nature, which requires millions of training examples to learn a reliable policy. On the other hand, a model--based reinforcement learning (MBRL) agent is capable of deliberate reasoning to achieve its goal. Unlike model--free agents, the MBRL agent iteratively learns a model of the world and plan its action according to its world model. MBRL has a great appeal because the learned model allows the agent to predict its future and reason about the consequences of its own actions. One of the ultimate goals of reinforcement learning research is to have agents acting in multiple environments and generalize previous learning experience to new situations. The ability to transfer knowledge across tasks is considered a critical aspect of any intelligent agent. The main objectives of the proposed research are to introduce a general model-based reinforcement learning algorithm that brings together three key ideas--compositionality, causality, and intrinsic curiosity--have been separately influential in machine learning over the past several decades. The objectives in this 5-year project are as follows: 1. Establish baselines for comparisons: Train and evaluate state-of-the-art model-based reinforcement learning agents in the latest locomotion control physics simulators. 2. Derive a compositional forward dynamics model, where the internal representations are object-based. 3. Explore, evaluate different types of causal inference methods in the proposed compositional model, including linear independent component analysis, mutual information-based independence tests, variational inference. 4. Develop planning-based algorithms to overcome non-stationary intrinsic rewards in exploration. 5. Answer the hypothesis that causal representations lead to simplified learning on new down-stream tasks, to help end-users in interpreting data, and to generalize to novel test examples. I anticipate that this project will benefit both deep learning and reinforcement learning community in several ways, ranging from the establishment of a new approach to actively infer causal factors, to elucidating new knowledge of exploration algorithms, to providing benchmark and open-source implementations of state-of-the-art MFRL and MBRL agents to maximally facilitate future research in the field of machine learning.