CAREER: Learning Neurosymbolic 3D Models

点击翻译按钮获取中文摘要

High-quality 3D models are increasingly in demand, driven by numerous industries and by the need for synthetic training data to scale up autonomous vision systems. But creating such models is a laborious and time-consuming process requiring years of training, so current practice will be insufficient to satisfy future data demands. One way forward is through generative models of 3D objects, that is to have machines learn to synthesize high-quality objects, a nice vision which has yet to be realized. Existing 3D generative models fall into one of two broad categories, each with limitations. Symbolic generative models such as shape grammars can enable non-experts to generate high-quality geometry but have severely limited expressiveness, while neural generative models are flexible and can in theory learn to express any shape but they are inscrutable and produce flawed geometry. This project will explore a new class of generative shape model that combines the best of both worlds: neuro-symbolic 3D models. The main insight is to use a symbolic program to model the logical part structure of a 3D object (e.g., the legs of a chair are connected to its seat), and then to use neural networks to refine this structure into high-quality geometry. Such a representation supports synthesis of new objects, reconstruction of objects from real-world sensor input, and high-level editing of object structure and geometry. It also supports modeling of higher-order object properties, including kinematics and physics. To enable massive-scale generation of synthetic 3D training data for computer vision and robotics, a neuro-symbolic version of the widely used ShapeNet dataset will be implemented and released. To help democratize 3D content creation, the project will collaborate with Unity Technologies to integrate neuro-symbolic 3D models into their popular 3D graphics engine. Project outcomes will also include an open-source, pedagogical deep learning framework to educate a new generation of researchers with the multidisciplinary skillset needed for neuro-symbolic modeling, in concert with activities (e.g., piloting new integrated visual computing curricula via summer schools and hosting visiting student researchers from historically under-represented groups) designed to improve student mastery of neural network fundamentals.The recognition-by-components theory of vision posits that people recognize objects by first understanding their fundamental parts and then using a secondary process to handle objects that are not distinguishable by these parts alone. Neuro-symbolic 3D models operationalize this theory for object synthesis via two algorithmic phases. The first phase is a new procedural representation called a hierarchical part graph program that is a human-readable computer program which, when executed, constructs a graph of connected object parts at multiple levels of detail wherein the bottom level of detail consists of parametric primitives such as cuboids and cylinders. While suggestive of shape, these graphs do not capture the full variety of geometry found in real-world objects. Thus, the second phase of the model is a new neural adaptive subdivision procedure which converts the low-fidelity parts into high-fidelity surface geometry. This decomposition is a natural fit for the common case of human-made objects, but it can also be extended to organic objects. The hypothesis is that this approach to 3D object generation will be able to efficiently synthesize and reconstruct a variety of high-quality objects in a unified, easily-editable representation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.