One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones
One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones
复制标题
DOI：
10.1109/cvpr52688.2022.01504
复制
发表时间：
2022-02
期刊：
影响因子：
--
通讯作者：
Chan Hee Song;Jihyung Kil;Tai-Yu Pan;Brian M. Sadler;Wei-Lun Chao;Yu Su
中科院分区：
文献类型：
--
作者：
Chan Hee Song;Jihyung Kil;Tai-Yu Pan;Brian M. Sadler;Wei-Lun Chao;Yu Su
我们研究开发自主代理的问题，该代理可以遵循人类指令来推断并执行一系列操作以完成基本任务。近年来，特别是短期任务方面取得了重大进展。然而，当涉及到具有扩展动作序列的长范围任务时，代理很容易忽略某些指令或陷入长指令的中间，最终使任务失败。为了应对这一挑战，我们提出了一种与模型无关的基于里程碑的任务跟踪器（M-TRACK）来指导代理并监控其进度。具体来说，我们提出了一个里程碑构建器，它用代理需要逐步完成的导航和交互里程碑来标记指令，以及一个里程碑检查器，系统地检查代理在当前里程碑中的进度，并确定何时继续下一个里程碑。在具有挑战性的 ALFRED 数据集上，我们的 M-Track 比两个竞争基础模型的成功率显着提高了 33% 和 52%。
We study the problem of developing autonomous agents that can follow human instructions to infer and perform a sequence of actions to complete the underlying task. Significant progress has been made in recent years, especially for tasks with short horizons. However, when it comes to long-horizon tasks with extended sequences of actions, an agent can easily ignore some instructions or get stuck in the middle of the long instructions and eventually fail the task. To address this challenge, we propose a modelagnostic milestone-based task tracker (M-TRACK) to guide the agent and monitor its progress. Specifically, we propose a milestone builder that tags the instructions with navigation and interaction milestones which the agent needs to complete step by step, and a milestone checker that systemically checks the agent's progress in its current milestone and determines when to proceed to the next. On the challenging ALFRED dataset, our M-Track leads to a notable 33% and 52% relative improvement in unseen success rate over two competitive base models.