NRI: Hierarchical Representation Learning for Robot Assistants

NRI：机器人助手的分层表示学习

• 批准号: 2405103
• 负责人: Shuran Song
• 金额: $ 150万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2405103&HistoricalAwards=false
• 关键词: NRI; Hierarchical; Representation; Learning; Robot

More than eighteen million people in North America have a physical disability due to limited mobility, restricting their independence, lifestyle, and ability to perform daily activities. One in five older adults struggle with mobility, millions of people with limited mobility are veterans, and a significant number of people have limited mobility because of diseases and accidents. Due to recent advances in artificial intelligence, robots hold great promise to provide timely assistance to people with disabilities, and drive improvements to their quality of life, independence, and productivity. This project introduces an automated robot assistant that is able to recognize a person’s goal, and provide them the right object at the right time, thereby helping people perform complex activities, such as cooking, object repair, and housekeeping. From both sight and dialogue, the research products will be able to anticipate what objects a person will need in the near future, and deliver it at exactly the right moment. Furthermore, the project will generate new educational opportunities at the intersection of robotics, computer vision and natural language processing through a series of systematically designed curriculum and annual capstone projects for assistive robotics. Due to the tight integration of multiple disciplines and the large practical impact, these educational programs will serve as an excellent platform for training the next generation of roboticists and increasing the diversity in the field. This research project introduces a novel hierarchical representation learning framework for assistive robots, which serves as a common interface to drive integration between robotics, computer vision, and natural language understanding. The project includes three thrusts. First, the team will develop hierarchical task representations. Second, human intention prediction and verification will developed. The final thrust will address intention-aware planning. Unlike established state representations in robotics, the new representation leverages non-Euclidean geometry, such as hyperbolic manifolds. Since hyperbolic space is a continuous analog of a tree, it provides new opportunities for learning task hierarchies from large-scale unlabeled instructional videos. This task representation is able to anticipate the activities of people, steer dialogue to reduce uncertainty, and provide dense rewards for long-horizon planning. This representation is learned from large-scale unlabeled instructional videos, making this approach flexible and adaptable to the many real-world applications of just-in-time object delivery for people with limited mobility.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在北美，有超过1800万人由于行动不便而身体残疾，限制了他们的独立性、生活方式和日常活动的能力。五分之一的老年人行动不便，数百万行动不便的人是退伍军人，还有相当多的人由于疾病和事故而行动不便。由于人工智能的最新进展，机器人有望为残疾人提供及时的帮助，并提高他们的生活质量、独立性和生产力。这个项目引入了一个自动化的机器人助手，它能够识别一个人的目标，并在正确的时间为他们提供正确的对象，从而帮助人们执行复杂的活动，如烹饪、物体修理和家务管理。通过视觉和对话，研究产品将能够预测一个人在不久的将来需要什么东西，并在正确的时刻提供它。此外，该项目将通过一系列系统设计的辅助机器人课程和年度顶点项目，在机器人、计算机视觉和自然语言处理的交叉领域产生新的教育机会。由于多学科的紧密结合和巨大的实际影响，这些教育项目将成为培养下一代机器人专家和增加该领域多样性的绝佳平台。本研究项目为辅助机器人引入了一种新的分层表示学习框架，作为驱动机器人、计算机视觉和自然语言理解之间集成的通用接口。该项目包括三个重点。首先，团队将开发分层任务表示。第二，人类意图预测和验证将得到发展。最后的重点将解决意图意识规划。与机器人中已建立的状态表示不同，新的表示利用了非欧几里得几何，如双曲流形。由于双曲空间是树的连续模拟，它为从大规模未标记的教学视频中学习任务层次结构提供了新的机会。这种任务表示能够预测人们的活动，引导对话以减少不确定性，并为长期规划提供密集的奖励。这种表示是从大规模的无标签教学视频中学习到的，使这种方法灵活且适应于为行动不便的人提供即时对象的许多实际应用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

TidyBot: Personalized Robot Assistance with Large Language Models

• DOI: 10.1007/s10514-023-10139-z
• 发表时间: 2023-05
• 期刊: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
• 作者: Jimmy Wu;Rika Antonova;Adam Kan;Marion Lepert;Andy Zeng;Shuran Song;J. Bohg;S. Rusinkiewicz;T. Funkhouser

FLEX: Full-Body Grasping Without Full-Body Grasps

• DOI: 10.1109/cvpr52729.2023.02029
• 发表时间: 2022-11
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Purva Tendulkar;D'idac Sur'is;Carl Vondrick

REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction

• DOI: 10.48550/arxiv.2306.15724
• 发表时间: 2023-06
• 期刊: ArXiv
• 作者: Zeyi Liu;Arpit Bahety;Shuran Song

Pre-Finetuning for Few-Shot Emotional Speech Recognition

• DOI: 10.48550/arxiv.2302.12921
• 发表时间: 2023-02
• 作者: Maximillian Chen;Zhou Yu