NRI: FND: Semi-Supervised Deep Learning for Domain Adaptation in Robotic Language Acquisition

NRI：FND：用于机器人语言习得领域适应的半监督深度学习

• 批准号: 2024878
• 负责人: Cynthia Matuszek
• 金额: $ 74.87万
• 依托单位: University of Maryland Baltimore County
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2024878&HistoricalAwards=false
• 关键词: NRI; FND; Semi; Supervised; Deep

This project will enable robots to learn to perform tasks with human teammates from language and other human modalities, and then transfer the learned knowledge across heterogeneous platforms and tasks. This will ultimately allow human-robot teaming in domains where people use varied language and instructions to complete complex tasks. As robots become more capable and ubiquitous, they are increasingly moving into complex, human-centric environments such as workplaces and homes. Being able to deploy useful robots in settings where human specialists are stretched thin, such as assistive technology, elder care, and education, has the potential to have far-reaching impacts on human quality of life. Achieving this will require the development of robots that learn, from natural interaction, about an end user's goals and environment. This work is intended to make robots more accessible and usable for non-specialists. In order to verify success and involve the broader community, tasks will be drawn from and tested in conjunction with community Makerspaces, which are strongly linked with both education and community involvement. The award includes an education and outreach plan designed to increase participation by and retention of women and underrepresented minorities (URM) in robotics and computing, engaging with UMBC's large URM population and world-class programs in this space.This award addresses how collaborative learning and successful performance during human-robot interactions can be accomplished by learning from and acting on grounded language. To accomplish this, this project will revolve around learning structured representations of abstract knowledge with goal-directed task completion, grounded in a physical context. There are three high-level research thrusts. In the first, new perceptual models to learn an alignment among a robot's multiple, heterogeneous sensor and data streams will be developed. In the second, synchronous grounded language models will be developed to better capture both general linguistic and implicit contextual expectations that are needed for completing tasks. In the third, a deep reinforcement learning framework will be developed that can leverage the advances achieved by the first two thrusts, allowing the development of techniques for learning conceptual knowledge. Taken together, these advances will allow an agent to achieve domain adaptation, improve its behaviors in new environments, and transfer conceptual knowledge among robotic agents.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将使机器人能够从语言和其他人类模式中学习与人类队友一起执行任务，然后在异构平台和任务之间传输所学到的知识。这最终将允许人类与机器人在人们使用不同语言和指令来完成复杂任务的领域进行合作。随着机器人变得越来越有能力和无处不在，它们越来越多地进入以人为中心的复杂环境，如工作场所和家庭。能够在人类专家捉襟见肘的环境中部署有用的机器人，例如辅助技术，老年人护理和教育，有可能对人类的生活质量产生深远的影响。实现这一目标将需要开发能够从自然交互中学习最终用户目标和环境的机器人。这项工作的目的是使机器人更容易为非专业人士使用。为了验证成功并让更广泛的社区参与，将从社区创客空间中提取任务并与社区创客空间一起进行测试，社区创客空间与教育和社区参与密切相关。该奖项包括一项教育和推广计划，旨在提高妇女和代表性不足的少数民族（URM）在机器人和计算领域的参与度和保留率，与UMBC庞大的URM人口和该领域的世界级项目合作。该奖项旨在探讨如何通过学习和运用基础语言来实现人机交互过程中的协作学习和成功表现。为了实现这一目标，该项目将围绕学习抽象知识的结构化表示与目标导向的任务完成，在物理环境中接地。有三个高层次的研究重点。首先，将开发新的感知模型来学习机器人的多个异构传感器和数据流之间的对齐。在第二，同步接地语言模型将被开发，以更好地捕捉完成任务所需的一般语言和隐式上下文的期望。在第三个方面，将开发一个深度强化学习框架，该框架可以利用前两个方面取得的进展，从而开发学习概念知识的技术。这些进步将使智能体能够实现领域适应，改善其在新环境中的行为，并在机器人智能体之间传递概念知识。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Measuring Equality in Machine Learning Security Defenses: A Case Study in Speech Recognition

• DOI: 10.1145/3605764.3623911
• 发表时间: 2023-02
• 期刊: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security
• 作者: Luke E. Richards;Edward Raff;Cynthia Matuszek

POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events

• DOI: 10.48550/arxiv.2212.02629
• 发表时间: 2022-12
• 作者: Sai Vallurupalli;Sayontan Ghosh;K. Erk;Niranjan Balasubramanian;Francis Ferraro

Jointly Identifying and Fixing Inconsistent Readings from Information Extraction Systems

联合识别和修复信息提取系统的不一致读数

• 发表时间: 2022
• 期刊: Third Deep Learning Inside Out (DeeLIO
• 作者: Padia, Ankur;Ferraro, Francis;Finin, Tim
• 通讯作者: Finin, Tim

Lessons From A Small-Scale Robot Joining Experiment in VR

小型机器人参与 VR 实验的经验教训

• 发表时间: 2023
• 期刊: and Mixed-Reality for Human-Robot Interactions (VAM-HRI
• 作者: Higgins, Padraig;Barron, Ryan;Engel, Don;Matuszek, Cynthia
• 通讯作者: Matuszek, Cynthia

Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech

• DOI: 10.1609/aaai.v36i10.21335
• 发表时间: 2021-12
• 作者: Gaoussou Youssouf Kebe;Luke E. Richards;Edward Raff;Francis Ferraro;Cynthia Matuszek