RI: Medium: Learning to Map and Navigate with Vision and Language

点击翻译按钮获取中文摘要

This project aims to advance the state of the art in robotic mapping and navigation by enabling spatial understanding using semantic maps and spatial reasoning for following language instructions given only visual inputs. Current performance in those tasks is low because of the inability to ground semantic entities and instructions spatially. Instead of grounding semantics to images, spatial understanding and navigation can be achieved if a system uses maps as an intermediate representation, as also indicated by behavioral and neural findings in spatial cognition. Building a map of an unseen space without exhaustive exploration can be learned, and this process can be facilitated by cross-modal language-vision attentional mechanisms. The project will integrate research with education and outreach underrepresented groups in Philadelphia neighborhoods as a target broadening the participation.This research is centered around understanding how vision and language interact to create better spatial representations like maps and facilitate navigation. The project will approach the vision-language from three angles. (i) How robots can learn to predict a map when entering an unseen environment using active learning. (ii) How navigation instructions can be encoded into spatial configuration schemata and navigational concepts that can be better aligned to maps and paths than raw language embeddings, and (iii) how navigational language representations can facilitate the creation of maps in unseen environments, and how one can follow instructions by using maps and language to create paths to follow.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.