Following Natural Language Instructions for Household Tasks With Landmark Guided Search and Reinforced Pose Adjustment
Following Natural Language Instructions for Household Tasks With Landmark Guided Search and Reinforced Pose Adjustment
复制标题
DOI：
10.1109/lra.2022.3178804
复制
发表时间：
2022-07
影响因子：
5.2
通讯作者：
Michael Murray;M. Cakmak
中科院分区：
文献类型：
--
作者：
Michael Murray;M. Cakmak
我们研究在移动机械手机器人上遵循自然语言指令的挑战性问题。这项任务具有挑战性，因为它要求机器人将不受约束的自然语言指令的语义与机器人对环境的自我中心视觉观察相结合，而这些观察通常是不完整且嘈杂的。为了解决这些挑战，我们提出了一种方法，能够使用可见的地标来更有效地探索环境，以搜索自然语言指令描述的对象。此外，我们建议在操纵计划期间使用姿势调整策略来帮助机器人从嘈杂的视觉观察中恢复。我们表明，可以通过强化学习和人机反馈的经验来训练该策略。我们根据基准测试在流行的 ALFRED 指令上评估我们的方法，并表明这些方法实现了最先进的性能 (35.41%)，与之前的工作有很大的差距（绝对值 8.92%）。
We study the challenging problem of following natural language instructions on a mobile manipulator robot. This task is challenging because it requires the robot to integrate the semantics of the unconstrained natural language instructions with the robot’s egocentric visual observations of the environment which are typically incomplete and noisy. To address these challenges, we propose a method that is able to use visible landmarks to more efficiently explore the environment in search of the objects described by the natural language instructions. Additionally, we propose using a pose adjustment policy during manipulation planning to help the robot recover from noisy visual observations. We show that this policy can be trained through experience with reinforcement learning as well as with human-in-the-loop feedback. We evaluate our approach on the popular ALFRED instruction following benchmark and show that these methods achieve state-of-the-art performance (35.41%) with a substantial (8.92% absolute) gap from prior work.