RI: Medium: Learning to Map and Navigate with Vision and Language

RI：媒介：学习用视觉和语言绘制地图和导航

• 批准号: 2212433
• 负责人: Kostas Daniilidis
• 金额: $ 120万
• 依托单位: University of Pennsylvania
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212433&HistoricalAwards=false
• 关键词: RI; Medium; Learning; Map; Navigate

This project aims to advance the state of the art in robotic mapping and navigation by enabling spatial understanding using semantic maps and spatial reasoning for following language instructions given only visual inputs. Current performance in those tasks is low because of the inability to ground semantic entities and instructions spatially. Instead of grounding semantics to images, spatial understanding and navigation can be achieved if a system uses maps as an intermediate representation, as also indicated by behavioral and neural findings in spatial cognition. Building a map of an unseen space without exhaustive exploration can be learned, and this process can be facilitated by cross-modal language-vision attentional mechanisms. The project will integrate research with education and outreach underrepresented groups in Philadelphia neighborhoods as a target broadening the participation.This research is centered around understanding how vision and language interact to create better spatial representations like maps and facilitate navigation. The project will approach the vision-language from three angles. (i) How robots can learn to predict a map when entering an unseen environment using active learning. (ii) How navigation instructions can be encoded into spatial configuration schemata and navigational concepts that can be better aligned to maps and paths than raw language embeddings, and (iii) how navigational language representations can facilitate the creation of maps in unseen environments, and how one can follow instructions by using maps and language to create paths to follow.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在通过使用语义地图和空间推理来实现空间理解，从而在仅给定视觉输入的语言指令中推进机器人地图和导航的最新技术。目前在这些任务中的表现很低，因为无法在空间上建立语义实体和指令。如果系统使用地图作为中间表示，就可以实现空间理解和导航，而不是基于图像的语义，正如空间认知中的行为和神经发现所表明的那样。在没有详尽探索的情况下构建一个未知空间的地图是可以学习的，这一过程可以通过跨模态语言-视觉注意机制来促进。该项目将把研究与教育结合起来，并将费城社区中未被充分代表的群体作为扩大参与的目标。这项研究的核心是理解视觉和语言如何相互作用，以创造更好的空间表征，如地图和方便导航。该项目将从三个角度探讨视觉语言。(i)机器人如何通过主动学习在进入未知环境时学习预测地图。（ii）如何将导航指令编码为空间配置模式和导航概念，从而比原始语言嵌入更能与地图和路径对齐；（iii）导航语言表示如何促进在未知环境中创建地图，以及如何通过使用地图和语言创建路径来遵循指令。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Cross-modal Map Learning for Vision and Language Navigation

• DOI: 10.1109/cvpr52688.2022.01502
• 发表时间: 2022-03
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: G. Georgakis;Karl Schmeckpeper;Karan Wanchoo;Soham Dan;E. Miltsakaki;D. Roth;Kostas Daniilidis