CAREER: Active Scene Understanding By and For Robot Manipulation

尽管取得了重大进展，但如今大多数机器人感知系统仍仅限于“看到它们被要求看到的东西”--通过观看静态图像或视频来检测预先定义的对象类别。相比之下，人类不断地通过积极的探索来决定“看什么”和“怎么看”。这种能力是解决问题和适应新场景的核心，但今天的机器人仍然缺乏这种能力。为了弥补这一差距，这个学院早期职业发展(CALEAR)项目旨在研究一种使用操作技能的自我改进的机器人感知系统-称为主动场景理解。该项目中提出的框架提高了机器人在感知和规划方面的基本能力，因此影响了许多应用领域，如服务机器人或野外探索，在这些领域，机器人需要快速分析其环境，以便对不断变化的情况做出快速反应。研究和教育计划通过支持云的机器人学习平台进行整合，该平台允许学生参与机器人教育和研究，而不受机器人和计算机硬件可访问性的限制。该项目解决了主动场景理解方面的一些挑战，以实现统一和实用的框架。该方法的关键思想是利用机器人的感知和交互算法之间的协同作用来创建自我监控信号。一方面，机器人可以使用自己的动作和相应的动作效果(即对后续状态的视觉观察)作为训练其视觉预测模型的地面真实标签。另一方面，机器人还可以使用感知模型提供的统计数据(如不确定性、新颖性和可预测性)作为奖励信号来改进其操作策略。最终，机器人可以将学习到的视觉预测模型和操作策略结合起来，促进下游任务的有效行动计划。该项目由跨部门的机器人基础研究计划支持，该计划由工程总监(ENG)和计算机和信息科学与工程(CEISE)联合管理和资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Despite significant progress, most robot perception systems today remain limited to "seeing what they are asked to see" – detecting pre-defined categories of objects by watching static images or videos. In contrast, humans constantly decide "what to see" and "how to see it" using active exploration. This ability is central to problem-solving and adaptability to novel scenarios but remains missing from robots today. To bridge this gap, this Faculty Early Career Development (CAREER) project aims to study a self-improving robot perception system using manipulation skills – referred to as active scene understanding. The framework suggested in this project improves a robot's fundamental capabilities in perception and planning and therefore impacts many application domains such as service robots or field exploration, where robots need to rapidly analyze their environments in order to swiftly react to evolving situations. The research and education plans are integrated through a Cloud-Enabled Robot Learning Platform, which allows students to participate in robotics education and research without the limits of robot and compute hardware accessibility.This project tackles a number of challenges in active scene understanding to achieve a unified and practical framework. The key idea of the approach is to leverage the synergies between a robot's perception and interaction algorithms to create self-supervisory signals. On the one hand, the robot can use its own actions and the corresponding action effects (i.e., visual observation of subsequent states) as ground truth labels for training its visual predictive model. On the other hand, the robot can also use the statistics provided by the perception model (e.g., uncertainty, novelty, and predictability) as a reward signal to improve its manipulation policy. Ultimately, the robot could combine the learned visual predictive model and manipulation policy to facilitate efficient action planning for downstream tasks.This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.