CAREER: Scalable Learning and Models for Mapping Instructions to Actions

职业：可扩展的学习和将指令映射到行动的模型

• 批准号: 1750499
• 负责人: Yoav Artzi
• 金额: $ 55.13万
• 依托单位: Cornell University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-06-01 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1750499&HistoricalAwards=false
• 关键词: CAREER; Scalable; Learning; Models; Mapping

Robust language understanding has the potential to dramatically improve the quality and accessibility of autonomous systems operating in complex environments. Already today such systems are becoming increasingly common, including self-driving cars, drones, and robots surveying disaster areas. Natural language interfaces open new opportunities for non-expert users to control complex systems and increase the accessibility of current systems. However, existing methods are limited in expressivity and, more often than not, disappoint users. This Faculty Early Career Development Grant will fundamentally transform how this problem is addressed, and provide new avenues to build systems with robust natural language understanding and ability to improve and learn through interaction with users. The project's five-year goal of grounded language understanding directly connects to robotic agents and autonomous cars, and will enable new interdisciplinary applications and research directions.The goal of the research program is to create a new framework for mapping natural language instructions to actions. Instead of taking a modular approach, this work adopts a single-model view, where input text and raw visual observations are directly mapped to actions. While the approach includes components that can be trained and re-used separately, it does not require any intermediate symbolic representation, and does away with the need for different types of training data, as required to train conventional modular systems. The five-year goal of this project is a continuously learning reflective autonomous agent following natural language instructions in realistic environments. The research will address learning from sparse natural signals, reasoning about complex sequences of instructions, learning continuously from users, and developing interpretable models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

强大的语言理解有可能大大提高在复杂环境中运行的自治系统的质量和可访问性。如今，这种系统已经变得越来越普遍，包括自动驾驶汽车、无人机和机器人在灾区进行调查。自然语言界面为非专家用户控制复杂系统和增加当前系统的可访问性提供了新的机会。然而，现有的方法是有限的，在表现力，往往不是，令用户失望。这一教师早期职业发展补助金将从根本上改变如何解决这个问题，并提供新的途径来构建具有强大的自然语言理解能力的系统，并通过与用户的互动来改进和学习。该项目的五年目标是接地语言理解直接连接到机器人代理和自动汽车，并将实现新的跨学科应用和研究方向。该研究计划的目标是创建一个新的框架，将自然语言指令映射到动作。这项工作没有采用模块化的方法，而是采用了单模型视图，其中输入文本和原始视觉观察直接映射到动作。虽然该方法包括可以单独训练和重用的组件，但它不需要任何中间符号表示，并且不需要训练传统模块化系统所需的不同类型的训练数据。这个项目的五年目标是一个在现实环境中遵循自然语言指令的不断学习的反射自主代理。该研究将解决从稀疏的自然信号中学习，推理复杂的指令序列，从用户那里不断学习，以及开发可解释的模型。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

项目成果

Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight

• 发表时间: 2019-10
• 期刊: ArXiv
• 作者: Valts Blukis;Yannick Terme;Eyvind Niklasson;Ross A. Knepper;Yoav Artzi

Who’s Waldo? Linking People Across Text and Images

• DOI: 10.1109/iccv48922.2021.00141
• 发表时间: 2021-08
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Claire Yuqing Cui;Apoorv Khandelwal;Yoav Artzi;Noah Snavely;Hadar Averbuch-Elor

Analysis of Language Change in Collaborative Instruction Following

• DOI: 10.18653/v1/2021.findings-emnlp.239
• 发表时间: 2021-09
• 期刊: ArXiv
• 作者: Anna Effenberger;Eva Yan;Rhia Singh;Alane Suhr;Yoav Artzi

Executing Instructions in Situated Collaborative Interactions

在情境协作交互中执行指令

• DOI: 10.18653/v1/d19-1218
• 发表时间: 2019
• 期刊: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP
• 作者: Suhr, Alane;Yan, Claudia;Schluger, Jack;Yu, Stanley;Khader, Hadi;Mouallem, Marwa;Zhang, Iris;Artzi, Yoav
• 通讯作者: Artzi, Yoav

TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments

• DOI: 10.1109/cvpr.2019.01282
• 发表时间: 2018-11
• 期刊: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Howard Chen;Alane Suhr;Dipendra Kumar Misra;Noah Snavely;Yoav Artzi