Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models-猫眼课题宝

DOI：

10.48550/arxiv.2207.11514

发表时间：

2022-07

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Huy Ha;Shuran Song

我们研究开放世界的3D场景理解，一个家庭的任务，需要代理的原因与开放的词汇和域外的视觉输入的3D环境-机器人在非结构化的3D世界中操作的关键技能。为此，我们提出了语义抽象（SemAbs），这是一个框架，它为2D视觉语言模型（VLM）提供了新的3D空间功能，同时保持了它们的零射击鲁棒性。我们使用从CLIP中提取的关联图来实现这种抽象，并以语义不可知的方式在这些抽象之上学习3D空间和几何推理技能。我们证明了SemAbs在两个开放世界3D场景理解任务中的有用性：1）完成部分观察到的对象和2）从语言描述中定位隐藏对象。实验表明，SemAbs可以推广到新的词汇，材料/照明，类和域（即，真实世界的扫描）从有限的3D合成数据上训练。代码和数据可在https://semantic-abstraction.cs.columbia.edu/上获得

We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extracted from CLIP, and learn 3D spatial and geometric reasoning skills on top of those abstractions in a semantic-agnostic manner. We demonstrate the usefulness of SemAbs on two open-world 3D scene understanding tasks: 1) completing partially observed objects and 2) localizing hidden objects from language descriptions. Experiments show that SemAbs can generalize to novel vocabulary, materials/lighting, classes, and domains (i.e., real-world scans) from training on limited 3D synthetic data. Code and data is available at https://semantic-abstraction.cs.columbia.edu/

Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models

文献摘要

被引文献

相似文献