NRI: Hierarchical Representation Learning for Robot Assistants

NRI：机器人助手的分层表示学习

• 批准号: 2132519
• 负责人: Shuran Song
• 金额: $ 150万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2132519&HistoricalAwards=false
• 关键词: NRI; Hierarchical; Representation; Learning; Robot

More than eighteen million people in North America have a physical disability due to limited mobility, restricting their independence, lifestyle, and ability to perform daily activities. One in five older adults struggle with mobility, millions of people with limited mobility are veterans, and a significant number of people have limited mobility because of diseases and accidents. Due to recent advances in artificial intelligence, robots hold great promise to provide timely assistance to people with disabilities, and drive improvements to their quality of life, independence, and productivity. This project introduces an automated robot assistant that is able to recognize a person’s goal, and provide them the right object at the right time, thereby helping people perform complex activities, such as cooking, object repair, and housekeeping. From both sight and dialogue, the research products will be able to anticipate what objects a person will need in the near future, and deliver it at exactly the right moment. Furthermore, the project will generate new educational opportunities at the intersection of robotics, computer vision and natural language processing through a series of systematically designed curriculum and annual capstone projects for assistive robotics. Due to the tight integration of multiple disciplines and the large practical impact, these educational programs will serve as an excellent platform for training the next generation of roboticists and increasing the diversity in the field. This research project introduces a novel hierarchical representation learning framework for assistive robots, which serves as a common interface to drive integration between robotics, computer vision, and natural language understanding. The project includes three thrusts. First, the team will develop hierarchical task representations. Second, human intention prediction and verification will developed. The final thrust will address intention-aware planning. Unlike established state representations in robotics, the new representation leverages non-Euclidean geometry, such as hyperbolic manifolds. Since hyperbolic space is a continuous analog of a tree, it provides new opportunities for learning task hierarchies from large-scale unlabeled instructional videos. This task representation is able to anticipate the activities of people, steer dialogue to reduce uncertainty, and provide dense rewards for long-horizon planning. This representation is learned from large-scale unlabeled instructional videos, making this approach flexible and adaptable to the many real-world applications of just-in-time object delivery for people with limited mobility.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在北美，超过1800万人由于行动不便而有身体残疾，这限制了他们的独立性、生活方式和日常活动能力。五分之一的老年人行动不便，数百万行动不便的人是退伍军人，还有相当多的人由于疾病和事故而行动不便。由于人工智能的最新进展，机器人有望为残疾人提供及时的帮助，并提高他们的生活质量，独立性和生产力。该项目介绍了一种自动化机器人助手，能够识别人的目标，并在正确的时间为他们提供正确的对象，从而帮助人们执行复杂的活动，如烹饪，对象修复和家务。通过视觉和对话，研究产品将能够预测一个人在不久的将来会需要什么样的物品，并在正确的时间交付。此外，该项目将通过一系列系统设计的课程和年度辅助机器人顶点项目，在机器人、计算机视觉和自然语言处理的交叉点上创造新的教育机会。 由于多学科的紧密结合和巨大的实际影响，这些教育计划将成为培养下一代机器人专家和增加该领域多样性的绝佳平台。该研究项目为辅助机器人引入了一种新的分层表示学习框架，该框架作为一个通用接口，用于驱动机器人，计算机视觉和自然语言理解之间的集成。该项目包括三个重点。首先，团队将开发分层任务表示。第二，人类意图预测和验证将得到发展。最后一个重点是意图感知规划。与机器人中已建立的状态表示不同，新的表示利用了非欧几里得几何，如双曲流形。由于双曲空间是树的连续模拟，它为从大规模未标记的教学视频中学习任务层次结构提供了新的机会。 这种任务表示能够预测人们的活动，引导对话以减少不确定性，并为长期规划提供密集的奖励。 这种表示是从大规模的无标签教学视频中学习到的，使这种方法灵活且适用于许多为行动不便的人提供即时对象交付的实际应用。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models

• DOI: 10.48550/arxiv.2207.11514
• 发表时间: 2022-07
• 作者: Huy Ha;Shuran Song

Representing Spatial Trajectories as Distributions

• DOI: 10.48550/arxiv.2210.01322
• 发表时间: 2022-10
• 期刊: ArXiv
• 作者: D'idac Sur'is;Carl Vondrick