ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model
ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model
复制标题
DOI：
10.48550/arxiv.2207.09446
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Rao Fu;Xiaoyu Zhan;Yiwen Chen;Daniel Ritchie;Srinath Sridhar
中科院分区：
文献类型：
--
作者：
Rao Fu;Xiaoyu Zhan;Yiwen Chen;Daniel Ritchie;Srinath Sridhar
我们提出了 ShapeCrafter，一种用于递归文本条件 3D 形状生成的神经网络。生成文本条件 3D 形状的现有方法需要消耗整个文本提示才能在一个步骤中生成 3D 形状。然而，人类倾向于递归地描述形状——我们可能从初始描述开始，并根据中间结果逐步添加细节。为了捕捉这个递归过程，我们引入了一种生成 3D 形状分布的方法，以初始短语为条件，随着更多短语的添加而逐渐演变。由于现有数据集不足以训练这种方法，因此我们提出了 Text2Shape++，这是一个包含 369K 形状文本对的大型数据集，支持递归形状生成。为了捕获通常用于细化形状描述的局部细节，我们在矢量量化的深度隐式函数的基础上构建，这些函数生成高质量形状的分布。结果表明，我们的方法可以生成与文本描述一致的形状，并且随着更多短语的添加，形状逐渐演变。我们的方法支持形状编辑、外推，并且可以在人机协作中实现创意设计的新应用。
We present ShapeCrafter, a neural network for recursive text-conditioned 3D shape generation. Existing methods to generate text-conditioned 3D shapes consume an entire text prompt to generate a 3D shape in a single step. However, humans tend to describe shapes recursively-we may start with an initial description and progressively add details based on intermediate results. To capture this recursive process, we introduce a method to generate a 3D shape distribution, conditioned on an initial phrase, that gradually evolves as more phrases are added. Since existing datasets are insufficient for training this approach, we present Text2Shape++, a large dataset of 369K shape-text pairs that supports recursive shape generation. To capture local details that are often used to refine shape descriptions, we build on top of vector-quantized deep implicit functions that generate a distribution of high-quality shapes. Results show that our method can generate shapes consistent with text descriptions, and shapes evolve gradually as more phrases are added. Our method supports shape editing, extrapolation, and can enable new applications in human-machine collaboration for creative design.