CAREER: Learning Neurosymbolic 3D Models

高质量的3D模型的需求越来越大，这是由众多行业以及对合成训练数据的需求推动的，以扩大自主视觉系统。但创建这样的模型是一个费力耗时的过程，需要多年的培训，因此目前的做法将不足以满足未来的数据需求。一种前进的方式是通过3D对象的生成模型，即让机器学习合成高质量的对象，这是一个尚未实现的美好愿景。现有的3D生成模型分为两大类，每一类都有局限性。符号生成模型（如形状语法）可以使非专家生成高质量的几何图形，但表达能力非常有限，而神经生成模型是灵活的，理论上可以学习表达任何形状，但它们是不可理解的，并产生有缺陷的几何图形。 这个项目将探索一类新的生成形状模型，结合了两个世界的最佳：神经符号3D模型。主要的见解是使用符号程序来建模3D对象的逻辑部分结构（例如，椅子的腿连接到它的座位上），然后使用神经网络将这个结构细化为高质量的几何形状。这样的表示支持新对象的合成，从真实世界的传感器输入的对象的重建，以及对象结构和几何形状的高级编辑。它还支持高阶对象属性的建模，包括运动学和物理学。 为了能够为计算机视觉和机器人技术大规模生成合成3D训练数据，将实施并发布广泛使用的ShapeNet数据集的神经符号版本。为了帮助3D内容创作大众化，该项目将与Unity Technologies合作，将神经符号3D模型集成到他们流行的3D图形引擎中。项目成果还将包括一个开源的教学深度学习框架，以教育新一代研究人员具备神经符号建模所需的多学科技能，并与活动（例如，通过暑期学校试行新的综合视觉计算课程，并接待来自历史上代表性不足的群体的访问学生研究人员），旨在提高学生对神经网络基础知识的掌握。视觉的成分识别理论认为，人们首先通过理解物体的基本部分来识别物体，然后使用次要过程来处理仅凭这些部分无法区分的物体。神经符号3D模型通过两个算法阶段将该理论用于对象合成。第一阶段是一个新的程序表示称为分层零件图程序，这是一个人类可读的计算机程序，当执行时，构建一个图形的连接对象部分在多个层次的细节，其中的底层细节由参数图元，如长方体和圆柱体。虽然这些图形暗示了形状，但它们并没有捕捉到现实世界中物体的各种几何形状。因此，模型的第二阶段是一个新的神经自适应细分过程，将低保真度的零件转换为高保真度的表面几何形状。这种分解很自然地适用于人造物体的常见情况，但它也可以扩展到有机物体。假设这种3D物体生成方法将能够以统一的、易于编辑的表示形式有效地合成和重建各种高质量的物体。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

High-quality 3D models are increasingly in demand, driven by numerous industries and by the need for synthetic training data to scale up autonomous vision systems. But creating such models is a laborious and time-consuming process requiring years of training, so current practice will be insufficient to satisfy future data demands. One way forward is through generative models of 3D objects, that is to have machines learn to synthesize high-quality objects, a nice vision which has yet to be realized. Existing 3D generative models fall into one of two broad categories, each with limitations. Symbolic generative models such as shape grammars can enable non-experts to generate high-quality geometry but have severely limited expressiveness, while neural generative models are flexible and can in theory learn to express any shape but they are inscrutable and produce flawed geometry. This project will explore a new class of generative shape model that combines the best of both worlds: neuro-symbolic 3D models. The main insight is to use a symbolic program to model the logical part structure of a 3D object (e.g., the legs of a chair are connected to its seat), and then to use neural networks to refine this structure into high-quality geometry. Such a representation supports synthesis of new objects, reconstruction of objects from real-world sensor input, and high-level editing of object structure and geometry. It also supports modeling of higher-order object properties, including kinematics and physics. To enable massive-scale generation of synthetic 3D training data for computer vision and robotics, a neuro-symbolic version of the widely used ShapeNet dataset will be implemented and released. To help democratize 3D content creation, the project will collaborate with Unity Technologies to integrate neuro-symbolic 3D models into their popular 3D graphics engine. Project outcomes will also include an open-source, pedagogical deep learning framework to educate a new generation of researchers with the multidisciplinary skillset needed for neuro-symbolic modeling, in concert with activities (e.g., piloting new integrated visual computing curricula via summer schools and hosting visiting student researchers from historically under-represented groups) designed to improve student mastery of neural network fundamentals.The recognition-by-components theory of vision posits that people recognize objects by first understanding their fundamental parts and then using a secondary process to handle objects that are not distinguishable by these parts alone. Neuro-symbolic 3D models operationalize this theory for object synthesis via two algorithmic phases. The first phase is a new procedural representation called a hierarchical part graph program that is a human-readable computer program which, when executed, constructs a graph of connected object parts at multiple levels of detail wherein the bottom level of detail consists of parametric primitives such as cuboids and cylinders. While suggestive of shape, these graphs do not capture the full variety of geometry found in real-world objects. Thus, the second phase of the model is a new neural adaptive subdivision procedure which converts the low-fidelity parts into high-fidelity surface geometry. This decomposition is a natural fit for the common case of human-made objects, but it can also be extended to organic objects. The hypothesis is that this approach to 3D object generation will be able to efficiently synthesize and reconstruct a variety of high-quality objects in a unified, easily-editable representation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.