CAREER: Active Scene Understanding By and For Robot Manipulation

职业：机器人操作的活动场景理解

• 批准号: 2348698
• 负责人: Shuran Song
• 金额: $ 60万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2027-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2348698&HistoricalAwards=false
• 关键词: CAREER; Active; Scene; Understanding; Robot

Despite significant progress, most robot perception systems today remain limited to "seeing what they are asked to see" – detecting pre-defined categories of objects by watching static images or videos. In contrast, humans constantly decide "what to see" and "how to see it" using active exploration. This ability is central to problem-solving and adaptability to novel scenarios but remains missing from robots today. To bridge this gap, this Faculty Early Career Development (CAREER) project aims to study a self-improving robot perception system using manipulation skills – referred to as active scene understanding. The framework suggested in this project improves a robot's fundamental capabilities in perception and planning and therefore impacts many application domains such as service robots or field exploration, where robots need to rapidly analyze their environments in order to swiftly react to evolving situations. The research and education plans are integrated through a Cloud-Enabled Robot Learning Platform, which allows students to participate in robotics education and research without the limits of robot and compute hardware accessibility.This project tackles a number of challenges in active scene understanding to achieve a unified and practical framework. The key idea of the approach is to leverage the synergies between a robot's perception and interaction algorithms to create self-supervisory signals. On the one hand, the robot can use its own actions and the corresponding action effects (i.e., visual observation of subsequent states) as ground truth labels for training its visual predictive model. On the other hand, the robot can also use the statistics provided by the perception model (e.g., uncertainty, novelty, and predictability) as a reward signal to improve its manipulation policy. Ultimately, the robot could combine the learned visual predictive model and manipulation policy to facilitate efficient action planning for downstream tasks.This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管取得了重大进展，但今天大多数机器人感知系统仍然局限于“看到他们被要求看到的东西”-通过观看静态图像或视频来检测预定义的对象类别。相比之下，人类通过积极的探索，不断地决定“看什么”和“如何看”。这种能力是解决问题和适应新场景的核心，但今天的机器人仍然缺乏这种能力。为了弥合这一差距，这个教师早期职业发展（CAREER）项目旨在研究一个自我改进的机器人感知系统，使用操纵技能-被称为主动场景理解。该项目中建议的框架提高了机器人在感知和规划方面的基本能力，因此影响了许多应用领域，如服务机器人或现场勘探，机器人需要快速分析其环境，以便迅速对不断变化的情况做出反应。研究和教育计划通过云支持的机器人学习平台进行整合，该平台允许学生参与机器人教育和研究，而不受机器人和计算机硬件的限制。该项目解决了主动场景理解方面的许多挑战，以实现统一和实用的框架。该方法的关键思想是利用机器人的感知和交互算法之间的协同作用来创建自我监督信号。一方面，机器人可以使用自己的动作和相应的动作效果（即，后续状态的视觉观察）作为用于训练其视觉预测模型的基础事实标签。另一方面，机器人还可以使用由感知模型提供的统计数据（例如，不确定性，新奇和可预测性）作为改善其操纵政策的奖励信号。最终，机器人可以联合收割机结合学习的视觉预测模型和操作策略，以促进下游任务的有效行动规划。由工程局（ENG）和计算机与信息科学与工程局（CISE）共同管理和资助该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects

• DOI: 10.48550/arxiv.2302.11553
• 发表时间: 2023-02
• 期刊: ArXiv
• 作者: Zhenjia Xu;Zhou Xian;Xingyu Lin;Cheng Chi;Zhiao Huang;Chuang Gan;Shuran Song

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition

• DOI: 10.48550/arxiv.2307.14535
• 发表时间: 2023-07
• 期刊: ArXiv
• 作者: Huy Ha;Peter R. Florence;Shuran Song

Structure from Action: Learning Interactions for 3D Articulated Object Structure Discovery

• DOI: 10.1109/iros55552.2023.10342135
• 发表时间: 2022-07
• 期刊: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
• 作者: Neil Nie;S. Gadre;Kiana Ehsani;Shuran Song