Generating Usable 3D Objects via Deep Learning

点击翻译按钮获取中文摘要

Deep learning has recently been successful in 3D geometric modelling and computer vision. Various applications of deep learning exist that generate or work with 3D shapes (e.g., shape generation, 3D reconstruction, etc.). Until now, 3D generated shapes are usually simple and fragile, lack geometric features, possess noise or irregularities, and are not accompanied by high-quality textures. To utilize the result of generative models in animation or manufacturing industries, we need high-quality, durable, and textured shapes. This proposal aims to design deep generative models that can produce "usable" 3D shapes meaning that they are fabricable, realistic (i.e., detailed and textured), diverse, and functional. While choosing appropriate deep generative models might be problem specific (e.g., convolutional vs implicit), deep implicit models have been successful to represent a shape as they provide a smooth representation and can handle topological varieties. Therefore, the main focus of this proposal is to develop methodologies to advance current deep implicit models to produce "usable" shapes. My ultimate target would be to design generative models capable of producing shapes with qualities that one cannot distinguish from those professionally made by an artist or a modeler. To achieve these goals, I start by defining four projects to explore different aspects of "shape usability" and its applications. First, I intend to enhance deep generative models to produce "fabricable" shapes that can be fabricated using available devices such as CNC machines or 3D prints. Depending on the fabrication type, a shape should hold certain properties such as "carvability" or "balance" for an efficient fabrication. Second project is to generate detailed shapes with appropriate textures by learning a mapping between texture domain and geometry. This is important as the utilized shapes in industry (e.g., video games) are often accompanied by complex textures. Third project is to add diversity to the generated shapes by diversifying style or functionality. To do so, shapes should be generated with respect to the whole data and mode collapse should be avoided. Also, style or functionality should be disentangled from the shape's content to transfer them across models. Lastly, I would like to collect and synthesize useful 3D datasets and explore the applications of generative models for data augmentation. For instance, capable generative models can be used to augment driving scenes by synthesizing rare scenes to train autonomous cars that behave safely in complex driving scenarios. Previous works on images have shown that generative models are capable of producing super-realistic images (e.g., StyleGAN). Also, recent advancements on 3D generative models especially deep implicit networks have shown a promising future. This proposal is an attempt to add new knowledge to the area of 3D generative models and advance them to produce more realistic and usable 3D shapes.