Object-Centric Visual Representation And Reinforcement Learning

我们将开发新的以对象为中心的视觉序列模型，旨在提高视频预测和基于视觉的强化学习的数据效率和对分布外环境的鲁棒性。本研究的第一阶段旨在将时间预测编码与对象中心学习相结合，并将所得模型应用于视频预测。这个阶段的目标是回答几个问题：以对象为中心的视频模型应该满足什么要求？这些要求如何体现在OCTPC的设计中？对象属性、瞬时属性变量、属性变量的随机时间演化和对象间关系应该如何表示？对象实例如何以不同的方式表示对象类型？如何精确地指定时间预测编码机制，使其能够充分学习长期依赖关系？现有的以对象为中心的学习方法之间的共同点和区别是什么，这些与这些算法的性能有什么关系？OCTPC在各种视频预测基准测试中的表现如何？哪些设计选择和超参数对性能影响最大？OCTPC的性能、行为和学习效率与不以对象为中心或不使用预测编码的模型相比如何？本研究的第二阶段旨在探索OCTPC在强化学习中可能带来的好处。这样做有两个主要动机。首先，强化学习有效地依赖于能够预测未来，因为最终目标是选择一个策略，使预期的长期未来回报最大化。使用高性能视频预测模型作为基于模型的强化学习的组成部分，将允许代理更准确地预测策略的变化如何影响未来的经验，因此应该如何改变策略以最大化未来的奖励，这是合理的。其次，有一个神经科学原理指出，“在所有的新皮层区域中，新皮层模块的处理功能在性质上是相似的……运动皮层本质上没有运动的东西，感觉皮层也没有感觉的东西。”[13]。因此，如果发现以对象为中心的归纳偏差对视频预测有用，那么类似的归纳偏差在策略表示中也可能有用。将这种策略表示与分层强化学习[17]中的现有工作进行比较，并探索这种表示是否可以提高强化学习中的样本效率，以及对分布外环境的鲁棒性，这将是一件有趣的事情。研究方法的新颖性据我们所知，将时间预测编码与以对象为中心的学习相结合的研究以前没有被探索过。探索这种结合将为机器学习提供有价值的贡献和见解，同时也通过在算法层面上更接近理解人类智能，为认知科学提供价值，这是合理的。该研究计划与人工智能和机器人主题、人工智能技术以及图像和视觉计算领域保持一致。

Abstract We will develop new object-centric sequence models for vision, with the intention to improve data-efficiencyand robustness to out-of-distribution environments for video prediction and vision-based reinforcement learning.Aims And Objectives The first stage of the proposed research aims to combine temporal predictive coding with object centric learning and apply the resulting model to video prediction. This stage will aim to answer several questions:What requirements should an object-centric video model satisfy? How can these requirements be reflected in the designof OCTPC? How should object properties, instantaneous attribute variables, stochastic temporal evolution of attributevariables, and inter-object relationships be represented? How should object instances be represented differently to objecttypes? How precisely should the temporal predictive coding mechanism be specified such that it is capable of learningsufficiently long-term dependencies? What are the commonalities and differences between existing approaches to object centric learning, and how do these relate to the performance of such algorithms? How does OCTPC perform on a variety ofvideo-prediction benchmarks? Which design choices and hyperparameters have the greatest effect on performance? Howdoes the performance, behaviour, and learning efficiency of OCTPC compare with models which aren't object-centric, orwhich don't use predictive coding?1The second stage of the proposed research aims to explore the possible benefits of OCTPC in reinforcement learning.There are two primary motivations for doing so. Firstly, reinforcement learning effectively depends on being able to predictthe future, because the ultimate objective is to choose a policy which maximises expected long-term future reward. It isplausible that a using a performant video prediction model as a component in model-based reinforcement learning wouldallow an agent to make more accurate predictions of how changes in policy would affect future experience, and thereforehow the policy should be changed in order to maximise future reward. Secondly, there is a neuroscientific principle whichstates that "the processing function of neocortical modules is qualitatively similar in all neocortical regions... there isnothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex." [13]. Therefore, if it is foundthat object-centric inductive biases are useful for video prediction, then it may be the case that similar inductive biasesare useful in policy representations as well. It would be interesting to compare such policy representations to existingwork in hierarchical reinforcement learning [17], and explore whether such representations can improve sample-efficiencyin reinforcement learning, and robustness to out-of-distribution environments.Novelty Of The Research Methodology To our knowledge, combining temporal predictive coding with object-centriclearning has not previously been explored. It is plausible that exploring this combination will provide valuable contributionsand insights to machine learning, while also providing value to the cognitive sciences by moving closer to an understandingof human intelligence on the algorithmic level.Alignment To EPSRC's Strategies And Research Areas This research proposal aligns with the areas of Artificialintelligence and robotics theme, Artificial intelligence technologies, and Image and vision computing.