RI: Medium: Learning to Map and Navigate with Vision and Language

RI：媒介：学习用视觉和语言绘制地图和导航

• 批准号: 2212433
• 负责人: Kostas Daniilidis
• 金额: $ 120万
• 依托单位: University of Pennsylvania
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212433&HistoricalAwards=false
• 关键词: RI; Medium; Learning; Map; Navigate

This project aims to advance the state of the art in robotic mapping and navigation by enabling spatial understanding using semantic maps and spatial reasoning for following language instructions given only visual inputs. Current performance in those tasks is low because of the inability to ground semantic entities and instructions spatially. Instead of grounding semantics to images, spatial understanding and navigation can be achieved if a system uses maps as an intermediate representation, as also indicated by behavioral and neural findings in spatial cognition. Building a map of an unseen space without exhaustive exploration can be learned, and this process can be facilitated by cross-modal language-vision attentional mechanisms. The project will integrate research with education and outreach underrepresented groups in Philadelphia neighborhoods as a target broadening the participation.This research is centered around understanding how vision and language interact to create better spatial representations like maps and facilitate navigation. The project will approach the vision-language from three angles. (i) How robots can learn to predict a map when entering an unseen environment using active learning. (ii) How navigation instructions can be encoded into spatial configuration schemata and navigational concepts that can be better aligned to maps and paths than raw language embeddings, and (iii) how navigational language representations can facilitate the creation of maps in unseen environments, and how one can follow instructions by using maps and language to create paths to follow.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在通过使用语义地图和空间推理来实现空间理解，以遵循仅给予视觉输入的语言指令，从而推进机器人测绘和导航的最新技术水平。这些任务的当前性能很低，因为无法在空间上将语义实体和指令接地。如果一个系统使用地图作为中间表示，而不是将语义与图像联系起来，就可以实现空间理解和导航，空间认知中的行为和神经发现也表明了这一点。构建一个看不见的空间的地图，而不需要穷尽的探索是可以学习的，这个过程可以通过跨模态的语言-视觉注意机制来促进。该项目将把研究与教育和外展费城社区的代表性不足的群体作为扩大参与的目标。这项研究的重点是了解视觉和语言如何相互作用，以创建更好的空间表示，如地图和方便导航。该项目将从三个角度探讨视觉语言。(i)机器人如何使用主动学习在进入看不见的环境时学习预测地图。(ii)导航指令如何被编码到空间配置模式和导航概念中，这些概念可以比原始语言嵌入更好地与地图和路径对齐，以及（iii）导航语言表示如何促进在看不见的环境中创建地图，以及如何通过使用地图和语言来创建路径来遵循指示。该奖项反映了NSF的法定使命，并被认为是值得的。通过使用基金会的知识价值和更广泛的影响审查标准进行评估来提供支持。

项目成果

Cross-modal Map Learning for Vision and Language Navigation

• DOI: 10.1109/cvpr52688.2022.01502
• 发表时间: 2022-03
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: G. Georgakis;Karl Schmeckpeper;Karan Wanchoo;Soham Dan;E. Miltsakaki;D. Roth;Kostas Daniilidis