3D Highlighter: Localizing Regions on 3D Shapes via Text Descriptions
3D Highlighter: Localizing Regions on 3D Shapes via Text Descriptions
复制标题
DOI：
10.1109/cvpr52729.2023.02005
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Dale Decatur;Itai Lang;Rana Hanocka
中科院分区：
文献类型：
--
作者：
Dale Decatur;Itai Lang;Rana Hanocka
我们提出了3D荧光笔，一种使用文本作为输入在网格上定位语义区域的技术。我们的系统的一个关键功能是能够解释“域外”本地化。我们的系统展示了推理的能力，在输入的3D形状上放置非明显相关的概念，例如将衣服添加到裸露的3D动物模型。我们的方法使用神经域将文本描述上下文化，并使用概率加权混合来为形状的相应区域着色。我们的神经优化由预训练的CLIP编码器指导，它绕过了对任何3D数据集或3D注释的需要。因此，3D Highlighter是高度灵活的，通用的，并且能够在无数的输入形状上产生本地化。我们的代码可在https://github.com/threedle/3DHighlighter上公开获取。
We present 3D Highlighter, a technique for localizing semantic regions on a mesh using text as input. A key feature of our system is the ability to interpret “out-of-domain” localizations. Our system demonstrates the ability to reason about where to place non-obviously related concepts on an input 3D shape, such as adding clothing to a bare 3D animal model. Our method contextualizes the text description using a neural field and colors the corresponding region of the shape using a probability-weighted blend. Our neural optimization is guided by a pre-trained CLIP encoder, which bypasses the need for any 3D datasets or 3D annotations. Thus, 3D Highlighter is highly flexible, general, and capable of producing localizations on a myriad of input shapes. Our code is publicly available at https://github.com/threedle/3DHighlighter.