CAREER: Scalable Learning and Models for Mapping Instructions to Actions

强大的语言理解有可能显著提高在复杂环境中运行的自主系统的质量和可访问性。如今，此类系统已经变得越来越普遍，包括自动驾驶汽车、无人机和调查灾区的机器人。自然语言接口为非专业用户控制复杂系统和增加当前系统的可访问性提供了新的机会。然而，现有的方法在表达能力上是有限的，而且往往会让用户失望。这项教师早期职业发展补助金将从根本上改变解决这一问题的方式，并为构建具有强大的自然语言理解能力和通过与用户互动来改进和学习能力的系统提供新的途径。该项目的五年目标是将基础语言理解与机器人代理和自动驾驶汽车直接联系起来，并将实现新的跨学科应用和研究方向。该研究项目的目标是创建一个新的框架，将自然语言指令映射到动作。这项工作没有采用模块化方法，而是采用单模型视图，其中输入文本和原始视觉观察直接映射到操作。虽然该方法包括可以单独训练和重用的组件，但它不需要任何中间符号表示，并且不需要训练传统模块化系统所需的不同类型的训练数据。该项目的五年目标是在现实环境中遵循自然语言指令不断学习反射自主代理。该研究将解决从稀疏的自然信号中学习，对复杂指令序列进行推理，从用户那里持续学习，以及开发可解释的模型。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Robust language understanding has the potential to dramatically improve the quality and accessibility of autonomous systems operating in complex environments. Already today such systems are becoming increasingly common, including self-driving cars, drones, and robots surveying disaster areas. Natural language interfaces open new opportunities for non-expert users to control complex systems and increase the accessibility of current systems. However, existing methods are limited in expressivity and, more often than not, disappoint users. This Faculty Early Career Development Grant will fundamentally transform how this problem is addressed, and provide new avenues to build systems with robust natural language understanding and ability to improve and learn through interaction with users. The project's five-year goal of grounded language understanding directly connects to robotic agents and autonomous cars, and will enable new interdisciplinary applications and research directions.The goal of the research program is to create a new framework for mapping natural language instructions to actions. Instead of taking a modular approach, this work adopts a single-model view, where input text and raw visual observations are directly mapped to actions. While the approach includes components that can be trained and re-used separately, it does not require any intermediate symbolic representation, and does away with the need for different types of training data, as required to train conventional modular systems. The five-year goal of this project is a continuously learning reflective autonomous agent following natural language instructions in realistic environments. The research will address learning from sparse natural signals, reasoning about complex sequences of instructions, learning continuously from users, and developing interpretable models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.