CAREER: Learning Neurosymbolic 3D Models

在众多行业的推动下，高质量的3D模型的需求日益增长，并且需要合成训练数据来扩展自主视觉系统。但是创建这样的模型是一个费力且耗时的过程，需要多年的培训，因此目前的实践将不足以满足未来的数据需求。一种方法是通过3D物体的生成模型，即让机器学习合成高质量的物体，这是一个尚未实现的美好愿景。现有的3D生成模型分为两大类，每一类都有局限性。符号生成模型（如形状语法）可以使非专家生成高质量的几何形状，但其表达能力严重有限，而神经生成模型是灵活的，理论上可以学习表达任何形状，但它们是不可理解的，并产生有缺陷的几何形状。这个项目将探索一种新的生成形状模型，它结合了两个世界的优点：神经符号3D模型。主要的见解是使用一个符号程序来模拟一个3D对象的逻辑部分结构（例如，椅子的腿连接到它的座位），然后使用神经网络将这个结构提炼成高质量的几何形状。这种表示支持新对象的合成，从现实世界的传感器输入重建对象，以及对象结构和几何的高级编辑。它还支持高阶对象属性的建模，包括运动学和物理学。为了能够大规模生成计算机视觉和机器人的合成3D训练数据，将实现和发布广泛使用的ShapeNet数据集的神经符号版本。为了促进3D内容创作的大众化，该项目将与Unity Technologies合作，将神经符号3D模型集成到他们流行的3D图形引擎中。项目成果还将包括一个开源的教学深度学习框架，以教育新一代的研究人员，使他们具备神经符号建模所需的多学科技能，并与旨在提高学生对神经网络基础知识的掌握的活动（例如，通过暑期学校试点新的综合视觉计算课程，并接待来自历史上代表性不足的群体的访问学生研究人员）相一致。视觉成分识别理论认为，人们首先通过理解物体的基本部分来识别物体，然后使用二级过程来处理仅凭这些部分无法区分的物体。神经符号三维模型通过两个算法阶段将这一理论用于对象合成。第一阶段是一种新的过程表示，称为分层零件图程序，这是一种人类可读的计算机程序，在执行时，它在多个细节层次上构建连接对象零件的图形，其中底层细节由参数基元（如长方体和圆柱体）组成。虽然这些图形暗示了形状，但并没有捕捉到现实世界中物体的全部几何形状。因此，该模型的第二阶段是一种新的神经自适应细分过程，将低保真零件转换为高保真的表面几何形状。这种分解很自然地适用于人造物体的常见情况，但它也可以扩展到有机物体。假设这种3D对象生成方法将能够以统一的、易于编辑的表示形式有效地合成和重建各种高质量的对象。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

High-quality 3D models are increasingly in demand, driven by numerous industries and by the need for synthetic training data to scale up autonomous vision systems. But creating such models is a laborious and time-consuming process requiring years of training, so current practice will be insufficient to satisfy future data demands. One way forward is through generative models of 3D objects, that is to have machines learn to synthesize high-quality objects, a nice vision which has yet to be realized. Existing 3D generative models fall into one of two broad categories, each with limitations. Symbolic generative models such as shape grammars can enable non-experts to generate high-quality geometry but have severely limited expressiveness, while neural generative models are flexible and can in theory learn to express any shape but they are inscrutable and produce flawed geometry. This project will explore a new class of generative shape model that combines the best of both worlds: neuro-symbolic 3D models. The main insight is to use a symbolic program to model the logical part structure of a 3D object (e.g., the legs of a chair are connected to its seat), and then to use neural networks to refine this structure into high-quality geometry. Such a representation supports synthesis of new objects, reconstruction of objects from real-world sensor input, and high-level editing of object structure and geometry. It also supports modeling of higher-order object properties, including kinematics and physics. To enable massive-scale generation of synthetic 3D training data for computer vision and robotics, a neuro-symbolic version of the widely used ShapeNet dataset will be implemented and released. To help democratize 3D content creation, the project will collaborate with Unity Technologies to integrate neuro-symbolic 3D models into their popular 3D graphics engine. Project outcomes will also include an open-source, pedagogical deep learning framework to educate a new generation of researchers with the multidisciplinary skillset needed for neuro-symbolic modeling, in concert with activities (e.g., piloting new integrated visual computing curricula via summer schools and hosting visiting student researchers from historically under-represented groups) designed to improve student mastery of neural network fundamentals.The recognition-by-components theory of vision posits that people recognize objects by first understanding their fundamental parts and then using a secondary process to handle objects that are not distinguishable by these parts alone. Neuro-symbolic 3D models operationalize this theory for object synthesis via two algorithmic phases. The first phase is a new procedural representation called a hierarchical part graph program that is a human-readable computer program which, when executed, constructs a graph of connected object parts at multiple levels of detail wherein the bottom level of detail consists of parametric primitives such as cuboids and cylinders. While suggestive of shape, these graphs do not capture the full variety of geometry found in real-world objects. Thus, the second phase of the model is a new neural adaptive subdivision procedure which converts the low-fidelity parts into high-fidelity surface geometry. This decomposition is a natural fit for the common case of human-made objects, but it can also be extended to organic objects. The hypothesis is that this approach to 3D object generation will be able to efficiently synthesize and reconstruct a variety of high-quality objects in a unified, easily-editable representation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.