E NVIRONMENT PREDICTIVE CODING FOR EMBODIED AGENTS
E NVIRONMENT PREDICTIVE CODING FOR EMBODIED AGENTS
复制标题
DOI：
--
复制
发表时间：
2020
影响因子：
12
通讯作者：
中科院分区：
文献类型：
--
作者：
我们引入环境预测编码，这是一种学习实体代理环境级表示的自监督方法。与之前的图像自监督学习工作相比，我们的目标是对代理在 3D 环境中移动时收集的一系列图像进行联合编码。我们通过区域预测任务学习这些表示，在该任务中，我们智能地屏蔽掉代理轨迹的部分内容，并根据代理的相机姿势从未屏蔽的部分中预测它们。通过学习视频集合上的此类表示，我们展示了向多个下游导航导向任务的成功迁移。我们在 Gibson 和 Matterport3D 的真实感 3D 环境中进行的实验表明，我们的方法在具有挑战性的任务上优于最先进的方法，并且仅需要有限的经验预算。
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. In contrast to prior work on self-supervised learning for images, we aim to jointly encode a series of images gathered by an agent as it moves about in 3D environments. We learn these representations via a zone prediction task, where we intelligently mask out portions of an agent’s trajectory and predict them from the unmasked portions, conditioned on the agent’s camera poses. By learning such representations on a collection of videos, we demonstrate successful transfer to multiple downstream navigationoriented tasks. Our experiments on the photorealistic 3D environments of Gibson and Matterport3D show that our method outperforms the state-of-the-art on challenging tasks with only a limited budget of experience.