Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks
Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks
复制标题
DOI：
10.48550/arxiv.2210.04476
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Albert Yu;R. Mooney
中科院分区：
文献类型：
--
作者：
Albert Yu;R. Mooney
演示和自然语言指示是指定和教机器人新任务的两种常见方法。但是，对于许多复杂的任务，仅演示或语言指令就包含歧义，从而防止任务清楚地指定。在这种情况下，与单独的模式相比，演示和指导的结合更简洁有效地传达了任务。为了实例化此问题设置，我们培训了一项多项多任务策略，以进行数百个具有挑战性的机器人选择任务，并提出Del-Taco（联合演示语言任务条件），这是将机器人策略调节任务嵌入的方法由两个组成部分组成：视觉演示和语言指令。通过允许这两种方式在新颖的任务规范中相互歧义和澄清，Del-Taco（1）大大减少了指定新任务所需的教师努力，并且（2）（2）在新的对象和说明上取得更好的概括性，而不是先前的任务。 - 调解方法。据我们所知，这是第一部表明，同时在演示和语言嵌入中同时调节多任务机器人操纵策略可以提高样本效率和概括性，而不是仅对任何一种方式进行调节。请参阅https://deltaco-robot.github.io/的其他材料
Demonstrations and natural language instructions are two common ways to specify and teach robots novel tasks. However, for many complex tasks, a demonstration or language instruction alone contains ambiguities, preventing tasks from being specified clearly. In such cases, a combination of both a demonstration and an instruction more concisely and effectively conveys the task to the robot than either modality alone. To instantiate this problem setting, we train a single multi-task policy on a few hundred challenging robotic pick-and-place tasks and propose DeL-TaCo (Joint Demo-Language Task Conditioning), a method for conditioning a robotic policy on task embeddings comprised of two components: a visual demonstration and a language instruction. By allowing these two modalities to mutually disambiguate and clarify each other during novel task specification, DeL-TaCo (1) substantially decreases the teacher effort needed to specify a new task and (2) achieves better generalization performance on novel objects and instructions over previous task-conditioning methods. To our knowledge, this is the first work to show that simultaneously conditioning a multi-task robotic manipulation policy on both demonstration and language embeddings improves sample efficiency and generalization over conditioning on either modality alone. See additional materials at https://deltaco-robot.github.io/