NRI: Hierarchical Representation Learning for Robot Assistants

北美有1800多万人由于行动不便而身体残疾，限制了他们的独立性、生活方式和日常活动能力。五分之一的老年人行动不便，数百万行动不便的人是退伍军人，相当数量的人因疾病和意外而行动不便。由于人工智能的最新进展，机器人有望为残疾人提供及时的帮助，并推动他们生活质量、独立性和生产力的提高。这个项目引入了一个自动化的机器人助手，它能够识别一个人的目标，并在正确的时间为他们提供正确的对象，从而帮助人们执行复杂的活动，如做饭、修复对象和做家务。从视觉和对话两方面来看，研究产品将能够预测一个人在不久的将来需要什么物品，并在正确的时间交付。此外，该项目将通过一系列系统设计的辅助机器人课程和年度顶峰项目，在机器人学、计算机视觉和自然语言处理的交叉领域创造新的教育机会。由于多学科的紧密结合和巨大的实际影响，这些教育项目将成为培训下一代机器人专家和增加该领域多样性的极好平台。本研究项目提出了一种新的辅助机器人层次表示学习框架，该框架作为一个公共接口来推动机器人、计算机视觉和自然语言理解之间的集成。该项目包括三个推进器。首先，团队将开发分层任务表示法。第二，将开展人类意图预测和验证。最后的重点将涉及有意图的规划。与机器人学中已有的状态表示不同，新的表示利用了非欧几里德几何，如双曲流形。由于双曲空间是树的连续模拟，它为从大规模的未标记的教学视频中学习任务层次提供了新的机会。这种任务表征能够预测人们的活动，引导对话以减少不确定性，并为长期规划提供密集的回报。这一代表是从大规模的无标签教学视频中学习的，使这种方法具有灵活性和适应性，适用于为行动受限的人提供即时对象交付的许多现实世界应用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

More than eighteen million people in North America have a physical disability due to limited mobility, restricting their independence, lifestyle, and ability to perform daily activities. One in five older adults struggle with mobility, millions of people with limited mobility are veterans, and a significant number of people have limited mobility because of diseases and accidents. Due to recent advances in artificial intelligence, robots hold great promise to provide timely assistance to people with disabilities, and drive improvements to their quality of life, independence, and productivity. This project introduces an automated robot assistant that is able to recognize a person’s goal, and provide them the right object at the right time, thereby helping people perform complex activities, such as cooking, object repair, and housekeeping. From both sight and dialogue, the research products will be able to anticipate what objects a person will need in the near future, and deliver it at exactly the right moment. Furthermore, the project will generate new educational opportunities at the intersection of robotics, computer vision and natural language processing through a series of systematically designed curriculum and annual capstone projects for assistive robotics. Due to the tight integration of multiple disciplines and the large practical impact, these educational programs will serve as an excellent platform for training the next generation of roboticists and increasing the diversity in the field. This research project introduces a novel hierarchical representation learning framework for assistive robots, which serves as a common interface to drive integration between robotics, computer vision, and natural language understanding. The project includes three thrusts. First, the team will develop hierarchical task representations. Second, human intention prediction and verification will developed. The final thrust will address intention-aware planning. Unlike established state representations in robotics, the new representation leverages non-Euclidean geometry, such as hyperbolic manifolds. Since hyperbolic space is a continuous analog of a tree, it provides new opportunities for learning task hierarchies from large-scale unlabeled instructional videos. This task representation is able to anticipate the activities of people, steer dialogue to reduce uncertainty, and provide dense rewards for long-horizon planning. This representation is learned from large-scale unlabeled instructional videos, making this approach flexible and adaptable to the many real-world applications of just-in-time object delivery for people with limited mobility.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.