NRI: Rich Task Perception for Programming by Demonstration

机器人可以与人类一起工作，承担重复、耗时的任务，可以极大地提高任务导向环境（如实验室、制造设施或商业厨房）的生产率和可靠性。实现这一愿景的关键挑战之一是，环境、用户和任务的每一个组合都对机器人的行为提出了独特的要求，采用传统的方法对这些机器人进行编程是不切实际的。相反，pi设想的机器人是由最终用户在特定的操作环境中为他们需要的特定任务进行编程的。为了克服现有方法的局限性，pi建议开发一个富任务感知框架，该框架能够从直观的人类演示中提取详细的任务描述。基于深度相机传感、gpu优化视觉处理和语言理解方面的最新进展，该框架将跟踪场景中的所有物体和人，识别其目标和任务上下文，并解析语音以从演示中提取更高级别的任务结构。pi还将通过演示技术引入新的编程，充分利用这些丰富的任务信息，使用户能够通过演示其期望的行为来编程机器人。拟议的研究有潜力通过开发用于工厂、实验室和家庭的研究和商业机器人系统来促进国家的健康、繁荣和福利。它将成为新一代高度灵活的机器人的一项使能技术，这些机器人可以在工作中编程，以提高任务环境（如实验室或制造设施）的生产率。拟议的工作还将促进科学的进步，使在科学研究湿实验室中进行的实验能够得到可靠的记录和复制。通过一门新的本科顶点课程，该项目将教育学生开发和编程下一代机器人。为了激励STEM职业的参与，pi将在华盛顿大学的年度公共宣传活动中展示他们的工作，并将通过华盛顿大学DawgBytes计划为K-16学生组织夏令营。协作机器人可以承担重复的、耗时的任务，可以大大提高目前由人类工人占据的以任务为导向的环境中的生产力和可靠性；如实验室、生产设施或商业厨房。实现这一愿景的关键挑战之一是，环境、用户和任务的每一个组合都对协作机器人的行为提出了独特的要求，采用传统的方法对这些机器人进行编程是不切实际的。相反，pi设想的协作机器人是由最终用户在特定的操作环境中为他们需要的特定任务进行编程的。在机器人技术中，一种流行的终端用户编程方法是通过演示编程（PbD），它使用户能够通过演示他们想要的行为来编程机器人。虽然最先进的PbD技术已经产生了令人印象深刻的机器人行为，但目前的方法有局限性，阻碍了它们变得实用和广泛采用。这些限制中有许多与感知有关，使机器人无法理解人类演示的详细背景。为了克服这些限制，pi建议开发一个富任务感知框架，该框架能够从直观的人类演示中提取详细的任务描述。基于RGB-D相机传感、gpu优化的视觉处理和语言基础的最新进展，该框架将以非常精细的粒度跟踪场景中的所有对象和人，并解析语音以从演示中提取更高级别的任务结构。pi还将引入新的PbD技术，以便在任务的编程和执行中更好地利用这些丰富的任务信息。

Robots that can work alongside humans and take on repetitive, time-consuming tasks could greatly improve productivity and reliability in task-oriented environments such as laboratories, manufacturing facilities, or commercial kitchens. One of the key challenges in realizing this vision is that every combination of environment, user, and task presents unique requirements for the robot's behavior and it is impractical to employ traditional approaches for programming these robots. Instead, the PIs envision robots that are programmable by their end-users in their particular operation environment and for the particular tasks they are needed for. To overcome limitations of existing approaches, the PIs propose to develop a framework for rich task perception, which is able to extract detailed task descriptions from intuitive human demonstrations. Building on recent advances in depth camera sensing, GPU-optimized visual processing, and language understanding, the proposed framework will track all objects and people in a scene, recognize their goals and task context, and parse speech to extract higher-level task structure from a demonstration. The PIs will also introduce new programming by demonstration techniques that take full advantage of such rich task information and enable users to program robots by demonstrating their desired behavior. The proposed research has the potential to advance national health, prosperity and welfare by developing research and commercial robotic systems for use in factories, laboratories, and households. It will be an enabling technology for a new generation of highly flexible robots that can be programmed on-the-job to increase the productivity of task environments, such as laboratories or manufacturing facilities. The proposed work will also promote the progress of science by enabling reliable documentation and replication of experiments performed in scientific research wet-labs. Through a new undergraduate capstone course, this project will educate students to develop and program this next generation of robots. To motivate participation in STEM careers, the PIs will demonstrate their work at yearly public outreach events at the University of Washington, and will organize a summer camp for K-16 students through the UW DawgBytes program.Co-robots that can take on repetitive, time-consuming tasks could greatly improve productivity and reliability in task-oriented environments currently occupied by human workers; such as laboratories, manufacturing facilities, or commercial kitchens. One of the key challenges in realizing this vision is that every combination of environment, user, and task presents unique requirements for the co-robot's behavior and it is impractical to employ traditional approaches for programming these robots. Instead, the PIs envision co-robots that are programmable by their end-users in their particular operation environment and for the particular tasks they are needed for. A popular end-user programming approach in robotics is Programming by Demonstration (PbD), which enables users to program robots by demonstrating their desired behavior. While state-of-the-art PbD techniques have generated impressive robotic behaviors, current approaches have limitations that prevent them from becoming practical and widely adopted. Many of these limitations are specifically related to perception, preventing robots from understanding the detailed context of human demonstrations. To overcome these limitations, The PIs propose to develop a framework for rich task perception, which is able to extract detailed task descriptions from intuitive human demonstrations. Building on recent advances in RGB-D camera sensing, GPU-optimized visual processing, and language grounding, the proposed framework will track all objects and people in a scene at a very fi ne granularity, and parse speech to extract higher-level task structure from a demonstration. The PIs will also introduce new PbD techniques that better take advantage of such rich task information both in the programming and execution of tasks.