NRI: Rich Task Perception for Programming by Demonstration

点击翻译按钮获取中文摘要

Robots that can work alongside humans and take on repetitive, time-consuming tasks could greatly improve productivity and reliability in task-oriented environments such as laboratories, manufacturing facilities, or commercial kitchens. One of the key challenges in realizing this vision is that every combination of environment, user, and task presents unique requirements for the robot's behavior and it is impractical to employ traditional approaches for programming these robots. Instead, the PIs envision robots that are programmable by their end-users in their particular operation environment and for the particular tasks they are needed for. To overcome limitations of existing approaches, the PIs propose to develop a framework for rich task perception, which is able to extract detailed task descriptions from intuitive human demonstrations. Building on recent advances in depth camera sensing, GPU-optimized visual processing, and language understanding, the proposed framework will track all objects and people in a scene, recognize their goals and task context, and parse speech to extract higher-level task structure from a demonstration. The PIs will also introduce new programming by demonstration techniques that take full advantage of such rich task information and enable users to program robots by demonstrating their desired behavior. The proposed research has the potential to advance national health, prosperity and welfare by developing research and commercial robotic systems for use in factories, laboratories, and households. It will be an enabling technology for a new generation of highly flexible robots that can be programmed on-the-job to increase the productivity of task environments, such as laboratories or manufacturing facilities. The proposed work will also promote the progress of science by enabling reliable documentation and replication of experiments performed in scientific research wet-labs. Through a new undergraduate capstone course, this project will educate students to develop and program this next generation of robots. To motivate participation in STEM careers, the PIs will demonstrate their work at yearly public outreach events at the University of Washington, and will organize a summer camp for K-16 students through the UW DawgBytes program.Co-robots that can take on repetitive, time-consuming tasks could greatly improve productivity and reliability in task-oriented environments currently occupied by human workers; such as laboratories, manufacturing facilities, or commercial kitchens. One of the key challenges in realizing this vision is that every combination of environment, user, and task presents unique requirements for the co-robot's behavior and it is impractical to employ traditional approaches for programming these robots. Instead, the PIs envision co-robots that are programmable by their end-users in their particular operation environment and for the particular tasks they are needed for. A popular end-user programming approach in robotics is Programming by Demonstration (PbD), which enables users to program robots by demonstrating their desired behavior. While state-of-the-art PbD techniques have generated impressive robotic behaviors, current approaches have limitations that prevent them from becoming practical and widely adopted. Many of these limitations are specifically related to perception, preventing robots from understanding the detailed context of human demonstrations. To overcome these limitations, The PIs propose to develop a framework for rich task perception, which is able to extract detailed task descriptions from intuitive human demonstrations. Building on recent advances in RGB-D camera sensing, GPU-optimized visual processing, and language grounding, the proposed framework will track all objects and people in a scene at a very fi ne granularity, and parse speech to extract higher-level task structure from a demonstration. The PIs will also introduce new PbD techniques that better take advantage of such rich task information both in the programming and execution of tasks.