NRI: Rich Task Perception for Programming by Demonstration

NRI：演示编程的丰富任务感知

• 批准号: 1525251
• 负责人: Dieter Fox
• 金额: $ 120万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-09-01 至 2019-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1525251&HistoricalAwards=false
• 关键词: NRI; Rich; Task; Perception; Programming

Robots that can work alongside humans and take on repetitive, time-consuming tasks could greatly improve productivity and reliability in task-oriented environments such as laboratories, manufacturing facilities, or commercial kitchens. One of the key challenges in realizing this vision is that every combination of environment, user, and task presents unique requirements for the robot's behavior and it is impractical to employ traditional approaches for programming these robots. Instead, the PIs envision robots that are programmable by their end-users in their particular operation environment and for the particular tasks they are needed for. To overcome limitations of existing approaches, the PIs propose to develop a framework for rich task perception, which is able to extract detailed task descriptions from intuitive human demonstrations. Building on recent advances in depth camera sensing, GPU-optimized visual processing, and language understanding, the proposed framework will track all objects and people in a scene, recognize their goals and task context, and parse speech to extract higher-level task structure from a demonstration. The PIs will also introduce new programming by demonstration techniques that take full advantage of such rich task information and enable users to program robots by demonstrating their desired behavior. The proposed research has the potential to advance national health, prosperity and welfare by developing research and commercial robotic systems for use in factories, laboratories, and households. It will be an enabling technology for a new generation of highly flexible robots that can be programmed on-the-job to increase the productivity of task environments, such as laboratories or manufacturing facilities. The proposed work will also promote the progress of science by enabling reliable documentation and replication of experiments performed in scientific research wet-labs. Through a new undergraduate capstone course, this project will educate students to develop and program this next generation of robots. To motivate participation in STEM careers, the PIs will demonstrate their work at yearly public outreach events at the University of Washington, and will organize a summer camp for K-16 students through the UW DawgBytes program.Co-robots that can take on repetitive, time-consuming tasks could greatly improve productivity and reliability in task-oriented environments currently occupied by human workers; such as laboratories, manufacturing facilities, or commercial kitchens. One of the key challenges in realizing this vision is that every combination of environment, user, and task presents unique requirements for the co-robot's behavior and it is impractical to employ traditional approaches for programming these robots. Instead, the PIs envision co-robots that are programmable by their end-users in their particular operation environment and for the particular tasks they are needed for. A popular end-user programming approach in robotics is Programming by Demonstration (PbD), which enables users to program robots by demonstrating their desired behavior. While state-of-the-art PbD techniques have generated impressive robotic behaviors, current approaches have limitations that prevent them from becoming practical and widely adopted. Many of these limitations are specifically related to perception, preventing robots from understanding the detailed context of human demonstrations. To overcome these limitations, The PIs propose to develop a framework for rich task perception, which is able to extract detailed task descriptions from intuitive human demonstrations. Building on recent advances in RGB-D camera sensing, GPU-optimized visual processing, and language grounding, the proposed framework will track all objects and people in a scene at a very fi ne granularity, and parse speech to extract higher-level task structure from a demonstration. The PIs will also introduce new PbD techniques that better take advantage of such rich task information both in the programming and execution of tasks.

机器人可以与人类一起工作，并承担重复，耗时的任务，可以大大提高实验室，制造设施或商业厨房等任务导向环境中的生产力和可靠性。实现这一愿景的关键挑战之一是，环境，用户和任务的每一个组合都对机器人的行为提出了独特的要求，采用传统的方法对这些机器人进行编程是不切实际的。相反，PI设想的机器人可由其最终用户在特定的操作环境中编程，并执行所需的特定任务。为了克服现有方法的局限性，PI建议开发一个框架，丰富的任务感知，这是能够提取详细的任务描述直观的人类演示。基于深度相机传感、GPU优化的视觉处理和语言理解方面的最新进展，该框架将跟踪场景中的所有对象和人员，识别他们的目标和任务上下文，并解析语音以从演示中提取更高级别的任务结构。PI还将通过演示技术引入新的编程，充分利用这些丰富的任务信息，并使用户能够通过演示其所需的行为来对机器人进行编程。拟议的研究有可能通过开发用于工厂，实验室和家庭的研究和商业机器人系统来促进国民健康，繁荣和福利。它将成为新一代高度灵活的机器人的一项使能技术，这些机器人可以在工作中编程，以提高实验室或制造设施等任务环境的生产力。拟议的工作还将促进科学的进步，使可靠的文件和复制在科学研究湿实验室进行的实验。通过一个新的本科顶点课程，该项目将教育学生开发和编程这种下一代机器人。为了激励人们参与STEM职业，PI将在华盛顿大学的年度公共宣传活动中展示他们的工作，并将通过UW Dawgarts计划为K-16学生组织一个夏令营。可以承担重复性，耗时任务的合作机器人可以大大提高目前由人类工人占据的任务导向环境中的生产力和可靠性;例如实验室、制造设施或商用厨房。实现这一愿景的关键挑战之一是，环境，用户和任务的每一个组合提出了独特的要求，合作机器人的行为，它是不切实际的，采用传统的方法来编程这些机器人。相反，PI设想的合作机器人可由其最终用户在其特定的操作环境中编程，并执行所需的特定任务。在机器人技术中，一种流行的最终用户编程方法是演示编程（PbD），它使用户能够通过演示他们想要的行为来对机器人进行编程。虽然最先进的PbD技术已经产生了令人印象深刻的机器人行为，但目前的方法存在局限性，使其无法实用和广泛采用。其中许多限制与感知有关，阻止机器人理解人类演示的详细背景。为了克服这些限制，PI建议开发一个丰富的任务感知框架，该框架能够从直观的人类演示中提取详细的任务描述。基于RGB-D相机传感、GPU优化的视觉处理和语言基础的最新进展，该框架将以非常细的粒度跟踪场景中的所有对象和人，并解析语音以从演示中提取更高级别的任务结构。PI还将引入新的PbD技术，以便在任务的编程和执行中更好地利用这些丰富的任务信息。

项目成果

Single Modality Performance on Visual Navigation & QA

视觉导航的单一模态性能

• 发表时间: 2019
• 期刊: Proc. of Conference of the North American Chapter of the Association for Computational Linguistics (NAACL
• 作者: Thomason, J.;Gordon, D.;Bisk, Y.
• 通讯作者: Bisk, Y.

What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning

我现在应该怎么做？

• 发表时间: 2020
• 期刊: Association for the Advancement of Artificial Intelligence (AAAI
• 作者: Gordon, D.;Fox, D.;Farhadi, A.
• 通讯作者: Farhadi, A.