Generating Usable 3D Objects via Deep Learning

深度学习最近在3D几何建模和计算机视觉方面取得了成功。存在生成3D形状或与3D形状一起工作的深度学习的各种应用（例如，形状生成、3D重建等）。到目前为止，3D生成的形状通常是简单和脆弱的，缺乏几何特征，具有噪音或不规则性，并且没有伴随着高质量的纹理。为了在动画或制造业中利用生成模型的结果，我们需要高质量，耐用和纹理化的形状。该提案旨在设计可以产生"可用" 3D形状的深度生成模型，这意味着它们是可制造的、逼真的（即，细节和纹理）、多样性和功能性。虽然选择适当的深度生成模型可能是特定于问题的（例如，卷积与隐式），深度隐式模型已经成功地表示形状，因为它们提供了平滑的表示并且可以处理拓扑变化。因此，本提案的主要重点是开发方法，以推进当前的深层隐式模型，以产生“可用”的形状。我的最终目标将是设计生成模型，能够生成具有无法与艺术家或建模师专业制作的形状区分开来的质量的形状。为了实现这些目标，我首先定义了四个项目来探索“形状可用性”及其应用的不同方面。首先，我打算增强深度生成模型，以产生“可制造”的形状，这些形状可以使用现有的设备（如CNC机器或3D打印）制造。根据制造类型，形状应该保持某些属性，如“雕刻性”或“平衡”，以实现有效的制造。第二个项目是通过学习纹理域和几何之间的映射来生成具有适当纹理的详细形状。这是重要的，因为工业中所使用的形状（例如，视频游戏）通常伴随着复杂的纹理。第三个项目是通过多样化的风格或功能来增加多样性。要做到这一点，形状应相对于整个数据生成，并应避免模式崩溃。此外，风格或功能应该从形状的内容中分离出来，以便在模型之间进行传输。最后，我想收集和合成有用的3D数据集，并探索生成模型在数据增强中的应用。例如，有能力的生成模型可用于通过合成罕见场景来增强驾驶场景，以训练在复杂驾驶场景中安全运行的自动驾驶汽车。先前关于图像的工作已经表明，生成模型能够产生超逼真的图像（例如，StyleGAN）。此外，最近在3D生成模型，特别是深度隐式网络方面的进展显示出了光明的未来。这个建议是一个尝试，以增加新的知识领域的3D生成模型，并推进他们产生更真实和可用的3D形状。

Deep learning has recently been successful in 3D geometric modelling and computer vision. Various applications of deep learning exist that generate or work with 3D shapes (e.g., shape generation, 3D reconstruction, etc.). Until now, 3D generated shapes are usually simple and fragile, lack geometric features, possess noise or irregularities, and are not accompanied by high-quality textures. To utilize the result of generative models in animation or manufacturing industries, we need high-quality, durable, and textured shapes. This proposal aims to design deep generative models that can produce "usable" 3D shapes meaning that they are fabricable, realistic (i.e., detailed and textured), diverse, and functional. While choosing appropriate deep generative models might be problem specific (e.g., convolutional vs implicit), deep implicit models have been successful to represent a shape as they provide a smooth representation and can handle topological varieties. Therefore, the main focus of this proposal is to develop methodologies to advance current deep implicit models to produce "usable" shapes. My ultimate target would be to design generative models capable of producing shapes with qualities that one cannot distinguish from those professionally made by an artist or a modeler. To achieve these goals, I start by defining four projects to explore different aspects of "shape usability" and its applications. First, I intend to enhance deep generative models to produce "fabricable" shapes that can be fabricated using available devices such as CNC machines or 3D prints. Depending on the fabrication type, a shape should hold certain properties such as "carvability" or "balance" for an efficient fabrication. Second project is to generate detailed shapes with appropriate textures by learning a mapping between texture domain and geometry. This is important as the utilized shapes in industry (e.g., video games) are often accompanied by complex textures. Third project is to add diversity to the generated shapes by diversifying style or functionality. To do so, shapes should be generated with respect to the whole data and mode collapse should be avoided. Also, style or functionality should be disentangled from the shape's content to transfer them across models. Lastly, I would like to collect and synthesize useful 3D datasets and explore the applications of generative models for data augmentation. For instance, capable generative models can be used to augment driving scenes by synthesizing rare scenes to train autonomous cars that behave safely in complex driving scenarios. Previous works on images have shown that generative models are capable of producing super-realistic images (e.g., StyleGAN). Also, recent advancements on 3D generative models especially deep implicit networks have shown a promising future. This proposal is an attempt to add new knowledge to the area of 3D generative models and advance them to produce more realistic and usable 3D shapes.