CAREER: Scalable Learning and Models for Mapping Instructions to Actions

稳健的语言理解有可能极大地提高在复杂环境中运行的自主系统的质量和可访问性。如今，这样的系统已经变得越来越常见，包括自动驾驶汽车、无人机和探测灾区的机器人。自然语言界面为非专家用户控制复杂系统和增加当前系统的可访问性提供了新的机会。然而，现有的方法在表现力方面是有限的，而且往往会让用户失望。该学院早期职业发展助学金将从根本上改变解决这一问题的方式，并提供新的途径，以建立具有强大的自然语言理解和能力的系统，并通过与用户的互动来改进和学习。该项目的五年目标是扎根语言理解，直接连接到机器人代理和自动驾驶汽车，并将使新的跨学科应用和研究方向。该研究计划的目标是创建一个新的框架，将自然语言指令映射到行动。这项工作不是采用模块化方法，而是采用单一模型视图，其中输入文本和原始视觉观察直接映射到操作。虽然该方法包括可以单独训练和重复使用的组件，但它不需要任何中间符号表示，并且不需要训练传统模块系统所需的不同类型的训练数据。这个项目的五年目标是一个在现实环境中遵循自然语言指令的持续学习的反思性自主代理。这项研究将致力于从稀疏的自然信号中学习，对复杂的指令序列进行推理，从用户那里持续学习，并开发可解释的模型。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Robust language understanding has the potential to dramatically improve the quality and accessibility of autonomous systems operating in complex environments. Already today such systems are becoming increasingly common, including self-driving cars, drones, and robots surveying disaster areas. Natural language interfaces open new opportunities for non-expert users to control complex systems and increase the accessibility of current systems. However, existing methods are limited in expressivity and, more often than not, disappoint users. This Faculty Early Career Development Grant will fundamentally transform how this problem is addressed, and provide new avenues to build systems with robust natural language understanding and ability to improve and learn through interaction with users. The project's five-year goal of grounded language understanding directly connects to robotic agents and autonomous cars, and will enable new interdisciplinary applications and research directions.The goal of the research program is to create a new framework for mapping natural language instructions to actions. Instead of taking a modular approach, this work adopts a single-model view, where input text and raw visual observations are directly mapped to actions. While the approach includes components that can be trained and re-used separately, it does not require any intermediate symbolic representation, and does away with the need for different types of training data, as required to train conventional modular systems. The five-year goal of this project is a continuously learning reflective autonomous agent following natural language instructions in realistic environments. The research will address learning from sparse natural signals, reasoning about complex sequences of instructions, learning continuously from users, and developing interpretable models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.