HCC: Small: 3DStylus: User-Guided Shape Manipulation using Neural Priors

数字3D模型用于各种各样的应用，包括制造，工程，医学和娱乐。然而，对3D模型需求的快速增长已经超过了我们构建它们的能力。采用当前的建模工具来执行最基本的3D操作，例如选择形状上的区域，需要多年的广泛培训和经验。该项目将开发新的和可访问的工具和技术，用于创建和操作3D对象，消除现有的进入技术壁垒，从而使3D内容创建民主化。项目成果将产生广泛的影响，使3D建模更容易获得，使专家和非专业用户能够为各种行业和专业的应用程序创建和转换对象。该项目以开发3DStylus为中心，这是一套基础工具，用户可以使用简单的文本作为输入，编辑和修改现有的3D形状，并从头开始创建新的3D形状。3D Stylus将包含三个核心组件：(i) 3D编辑器，它将通过合并文本指定的纹理、材料和保留底层形状的本地化修改来增强现有的3D模型；（ii） 3D Morpher，将现有3D模型转换为新的文本指定几何形状，同时保留原始纹理细节；以及（iii） 3D Creator，这将允许用户直观地创建新颖的3D几何形状，并进行所需的编辑和操作，仅使用文本描述。该研究将利用深度学习技术，这些技术已经在2D领域得到了成功的应用（DALL-E的革命性影响证明了这一点），但由于可用的高质量3D数据相对较少，在3D领域通常是遥不可及的。设想的新方法将使用大量可用的2D数据集作为编辑3D对象的信号。因此，可以利用深度学习的转型潜力来实现先进的3D建模功能，而不需要大型3D数据集。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Digital 3D models are used in a wide variety of applications, including manufacturing, engineering, medicine, and entertainment. The rapid growth in demand for 3D models, however, has outpaced our ability to construct them. Employing current modeling tools to perform even the most basic of 3D operations, such as selecting a region on a shape, requires years of extensive training and experience. This project will develop new and accessible tools and techniques for creating and manipulating 3D objects, eliminating existing technical barriers of entry and thereby democratizing 3D content creation. Project outcomes will have broad impact by making 3D modeling more accessible, empowering both expert and lay users to create and transform objects for applications across a wide range of industries and professions. The project is centered around the development of 3DStylus, a suite of foundational tools that will enable users to edit and modify existing 3D shapes and create new ones from scratch, using simple text as input. 3D Stylus will encompass three central components: (i) 3D Editor, which will enhance existing 3D models by incorporating text-specified textures, materials, and localized modifications that preserve the underlying shape; (ii) 3D Morpher, which will transform existing 3D models into new text-specified geometry, while preserving the original texture details; and (iii) 3D Creator, which will allow users to intuitively create novel 3D geometries, and make desired edits and manipulations, using only a text description. The research will leverage deep learning techniques which have been used so successfully in 2D (as evidenced by the revolutionary impact of DALL-E), but which have generally been out-of-reach in 3D given the relatively small amount of available high-quality 3D data. The novel approaches envisioned will instead use the abundantly available 2D datasets as a signal for editing 3D objects. Thus, the transformational potential of deep learning can be harnessed to achieve advanced 3D modeling capabilities, without the need for large 3D datasets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.