EAGER: Learning Language in Simulation for Real Robot Interaction

EAGER：在模拟中学习语言以实现真实的机器人交互

• 批准号: 1940931
• 负责人: Cynthia Matuszek
• 金额: $ 21.95万
• 依托单位: University of Maryland Baltimore County
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-12-01 至 2021-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1940931&HistoricalAwards=false
• 关键词: EAGER; Learning; Language; Simulation; Real

While robots are rapidly becoming more capable and ubiquitous, theirutility is still severely limited by the inability of regular users tocustomize their behaviors. This EArly Grant for Exploratory Research (EAGER) will explore how examples of language, gaze, and other communications can be collected from avirtual interaction with a robot in order to learn how robots caninteract better with end users. Current robots' difficulty of use andinflexibility are major factors preventing them from being morebroadly available to populations that might benefit, such asaging-in-place seniors. One promising solution is to let users controland teach robots with natural language, an intuitive and comfortablemechanism. This has led to active research in the area of groundedlanguage acquisition: learning language that refers to and is informedby the physical world. Given the complexity of robotic systems, thereis growing interest in approaches that take advantage of the latest invirtual reality technology, which can lower the barrier of entry tothis research.This EAGER project develops infrastructure that will lay the necessarygroundwork for applying simulation-to-reality approaches to naturallanguage interactions with robots. This project aims to bootstraprobots' learning to understand language, using a combination of datacollected in a high-fidelity virtual reality environment withsimulated robots and real-world testing on physical robots. A personwill interact with simulated robots in virtual reality, and his or heractions and language will be recorded. By integrating with existingrobotics technology, this project will model the connection betweenthe language people use and the robot's perceptions and actions.Natural language descriptions of what is happening in simulation willbe obtained and used to train a joint model of language and simulatedpercepts as a way to learn grounded language. The effectiveness of theframework and algorithms will be measured on automaticprediction/generation tasks and transferability of learned models to areal, physical robot. This work will serve as a proof of concept forthe value of combining robotics simulation with human interaction, aswell as providing interested researchers with resources to bootstraptheir own work.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

虽然机器人正在迅速变得越来越有能力和无处不在，但它们的实用性仍然受到普通用户无法定制其行为的严重限制。EARLY探索性研究资助（EAGER）将探索如何从与机器人的虚拟交互中收集语言，凝视和其他通信的例子，以了解机器人如何更好地与最终用户交互。目前机器人的使用困难和不灵活性是阻止它们更广泛地应用于可能受益的人群的主要因素，例如老年人。一个有希望的解决方案是让用户用自然语言控制和教授机器人，这是一种直观和舒适的机制。这导致了对跨语言习得领域的积极研究：学习涉及物理世界并由物理世界提供信息的语言。鉴于机器人系统的复杂性，人们对利用最新虚拟现实技术的方法越来越感兴趣，这可以降低进入这项研究的门槛。EAGER项目开发的基础设施将为将模拟到现实的方法应用于与机器人的自然语言交互奠定必要的基础。该项目旨在引导probots学习理解语言，使用高保真虚拟现实环境中收集的模拟机器人和物理机器人的真实世界测试相结合。一个人将在虚拟现实中与模拟机器人互动，他或她的动作和语言将被记录下来。通过与现有的机器人技术相结合，该项目将对人类使用的语言与机器人的感知和动作之间的联系进行建模。将获得对模拟中发生的事情的自然语言描述，并用于训练语言和模拟感知的联合模型，作为学习基础语言的一种方式。该框架和算法的有效性将在自动预测/生成任务和学习模型到真实物理机器人的可转移性上进行测量。这项工作将作为结合机器人模拟与人类互动的价值的概念证明，以及为感兴趣的研究人员提供资源来引导他们自己的工作。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

Planning with Abstract Learned Models While Learning Transferable Subtasks

在学习可转移子任务的同时使用抽象学习模型进行规划

• DOI: 10.1609/aaai.v34i06.6555
• 发表时间: 2020
• 期刊: Proceedings of the AAAI Conference on Artificial Intelligence
• 作者: Winder, John;Milani, Stephanie;Landen, Matthew;Oh, Erebus;Parr, Shane;Squire, Shawn;desJardins, Marie;Matuszek, Cynthia
• 通讯作者: Matuszek, Cynthia

Jointly Identifying and Fixing Inconsistent Readings from Information Extraction Systems

联合识别和修复信息提取系统的不一致读数

• 发表时间: 2022
• 期刊: Third Deep Learning Inside Out (DeeLIO
• 作者: Padia, Ankur;Ferraro, Francis;Finin, Tim
• 通讯作者: Finin, Tim

Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech

• DOI: 10.1609/aaai.v36i10.21335
• 发表时间: 2021-12
• 作者: Gaoussou Youssouf Kebe;Luke E. Richards;Edward Raff;Francis Ferraro;Cynthia Matuszek

Head Pose for Object Deixis in VR-Based Human-Robot Interaction

基于 VR 的人机交互中物体指示的头部姿势

• 发表时间: 2022
• 期刊: International Conference on Robot & Human Interactive Communication (Ro-Man
• 作者: Higgins, Padraig;Barron, Ryan;Matuszek, Cynthia
• 通讯作者: Matuszek, Cynthia

Neural Variational Learning for Grounded Language Acquisition

• DOI: 10.1109/ro-man50785.2021.9515374
• 发表时间: 2021-07
• 期刊: 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)
• 作者: Nisha Pillai;Cynthia Matuszek;Francis Ferraro