EAGER: Learning Language in Simulation for Real Robot Interaction

虽然机器人正在迅速变得更有能力和无处不在，但它们的实用性仍然受到常规用户无法定制其行为的严重限制。这项早期的探索性研究拨款(AGIRE)将探索如何从与机器人的虚拟互动中收集语言、凝视和其他交流的例子，以便了解机器人如何更好地与最终用户互动。目前机器人的使用困难和缺乏灵活性是阻碍它们更广泛地应用于可能受益的人群的主要因素，例如就地老龄化的老年人。一个有希望的解决方案是让用户控制用自然语言教授机器人，这是一种直观而舒适的机制。这导致了在扎根语言习得领域的活跃研究：学习与物理世界有关并由物理世界提供信息的语言。考虑到机器人系统的复杂性，人们对利用最新虚拟现实技术的方法越来越感兴趣，这可以降低这项研究的进入门槛。这个渴望的项目开发的基础设施将为将模拟到现实的方法应用于与机器人的自然语言交互奠定必要的基础。该项目旨在利用在高保真虚拟现实环境中收集的数据和模拟机器人以及在真实机器人上的真实世界测试相结合的方式，引导机器人学习理解语言。一个人将在虚拟现实中与模拟机器人互动，他或她的动作和语言将被记录下来。通过与现有的机器人技术相结合，该项目将对人们使用的语言与机器人的感知和动作之间的联系进行建模。将获得对模拟过程中发生的事情的自然语言描述，并将其用于训练语言和模拟感知的联合模型，作为学习扎根语言的一种方式。框架和算法的有效性将在自动预测/生成任务和学习模型到区域物理机器人的可转移性上进行衡量。这项工作将作为将机器人模拟与人类交互相结合的价值的概念证明，并为感兴趣的研究人员提供资源来启动他们自己的工作。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

While robots are rapidly becoming more capable and ubiquitous, theirutility is still severely limited by the inability of regular users tocustomize their behaviors. This EArly Grant for Exploratory Research (EAGER) will explore how examples of language, gaze, and other communications can be collected from avirtual interaction with a robot in order to learn how robots caninteract better with end users. Current robots' difficulty of use andinflexibility are major factors preventing them from being morebroadly available to populations that might benefit, such asaging-in-place seniors. One promising solution is to let users controland teach robots with natural language, an intuitive and comfortablemechanism. This has led to active research in the area of groundedlanguage acquisition: learning language that refers to and is informedby the physical world. Given the complexity of robotic systems, thereis growing interest in approaches that take advantage of the latest invirtual reality technology, which can lower the barrier of entry tothis research.This EAGER project develops infrastructure that will lay the necessarygroundwork for applying simulation-to-reality approaches to naturallanguage interactions with robots. This project aims to bootstraprobots' learning to understand language, using a combination of datacollected in a high-fidelity virtual reality environment withsimulated robots and real-world testing on physical robots. A personwill interact with simulated robots in virtual reality, and his or heractions and language will be recorded. By integrating with existingrobotics technology, this project will model the connection betweenthe language people use and the robot's perceptions and actions.Natural language descriptions of what is happening in simulation willbe obtained and used to train a joint model of language and simulatedpercepts as a way to learn grounded language. The effectiveness of theframework and algorithms will be measured on automaticprediction/generation tasks and transferability of learned models to areal, physical robot. This work will serve as a proof of concept forthe value of combining robotics simulation with human interaction, aswell as providing interested researchers with resources to bootstraptheir own work.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.