CAREER: Active Scene Understanding By and For Robot Manipulation

尽管取得了重大进展，但目前大多数机器人感知系统仍然局限于“看到他们被要求看到的东西”——通过观看静态图像或视频来检测预先定义的物体类别。相比之下，人类不断地通过主动探索来决定“看什么”和“怎么看”。这种能力是解决问题和适应新情况的核心能力，但在今天的机器人中仍然缺失。为了弥补这一差距，该学院早期职业发展（Career）项目旨在研究一种利用操纵技能（即主动场景理解）自我改进的机器人感知系统。该项目提出的框架提高了机器人在感知和规划方面的基本能力，因此影响了许多应用领域，如服务机器人或野外勘探，机器人需要快速分析其环境，以便迅速对不断变化的情况做出反应。研究和教育计划通过云支持的机器人学习平台进行整合，该平台允许学生参与机器人教育和研究，而不受机器人和计算硬件可访问性的限制。该项目解决了主动场景理解中的一些挑战，以实现统一和实用的框架。该方法的关键思想是利用机器人感知和交互算法之间的协同作用来创建自我监督信号。一方面，机器人可以使用自己的动作和相应的动作效果（即后续状态的视觉观察）作为训练其视觉预测模型的ground truth标签。另一方面，机器人也可以使用感知模型提供的统计数据（如不确定性、新颖性和可预测性）作为奖励信号来改进其操作策略。最终，机器人可以将学习到的视觉预测模型和操作策略相结合，以促进下游任务的有效行动规划。该项目由跨部门机器人基础研究项目支持，由工程（ENG）和计算机与信息科学与工程（CISE）联合管理和资助。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Despite significant progress, most robot perception systems today remain limited to "seeing what they are asked to see" – detecting pre-defined categories of objects by watching static images or videos. In contrast, humans constantly decide "what to see" and "how to see it" using active exploration. This ability is central to problem-solving and adaptability to novel scenarios but remains missing from robots today. To bridge this gap, this Faculty Early Career Development (CAREER) project aims to study a self-improving robot perception system using manipulation skills – referred to as active scene understanding. The framework suggested in this project improves a robot's fundamental capabilities in perception and planning and therefore impacts many application domains such as service robots or field exploration, where robots need to rapidly analyze their environments in order to swiftly react to evolving situations. The research and education plans are integrated through a Cloud-Enabled Robot Learning Platform, which allows students to participate in robotics education and research without the limits of robot and compute hardware accessibility.This project tackles a number of challenges in active scene understanding to achieve a unified and practical framework. The key idea of the approach is to leverage the synergies between a robot's perception and interaction algorithms to create self-supervisory signals. On the one hand, the robot can use its own actions and the corresponding action effects (i.e., visual observation of subsequent states) as ground truth labels for training its visual predictive model. On the other hand, the robot can also use the statistics provided by the perception model (e.g., uncertainty, novelty, and predictability) as a reward signal to improve its manipulation policy. Ultimately, the robot could combine the learned visual predictive model and manipulation policy to facilitate efficient action planning for downstream tasks.This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.