Diffusion models for Single-image 3D reconstruction of deformable objects.

研究背景的简要描述，包括潜在的影响：可变形物体的3D重建在各个领域具有丰富的潜在应用，包括AR/VR，游戏和动物行为研究。然而，这些3D模型的创建和动画需要3D艺术家的大量努力和专业知识。这为创建多样化和丰富的3D环境以及为新对象类别生成3D资产带来了重大障碍。这项研究的目的是创建一个深度学习模型，从单个输入图像自动生成这些资产。这是一个非常实际但具有挑战性的任务，因为模型必须对对象的可能形状和外观有先验的理解。收集3D地面实况数据以了解这种先验需要付出巨大的努力。最近，人们越来越有兴趣从广泛可用的数据中学习这些先验知识：互联网图像。从这些数据中学习提出了许多挑战，如缺乏多视图约束，噪声数据，遮挡，缺乏不同的观点。考虑到这些挑战，现有的最先进的方法无法实现准确、高保真的结果，并且仅限于特定的对象类别。研究方法的新奇：为了解决上述局限性，我计划将现有方法与强大的预训练2D文本到图像扩散模型联合收割机相结合。这些模型有可能提供额外的先验，这将导致更忠实的3D重建。此外，它们还可以帮助模型对看不见的类别进行泛化，而无需收集额外的训练图像。这项工作可能会导致一种新的最先进的方法。任何参与的公司或合作者：我是ELLIS博士项目的一部分，这是一个泛欧洲的博士研究项目，所以ELLIS将参与其中。我的研究将在这些领域-计算机视觉，深度学习，AI鲁棒性和AI伦理。

Brief description of the context of the research including the potential impact: 3D reconstruction of deformable objects has a wealth of potential applications across various fields, including AR/VR, gaming, and animal behaviour research. However, the creation and animation of these 3D models requires significant effort and expert knowledge of a 3D artist. This presents a significant barrier for creating diverse and abundant 3D environments, as well as generating 3D assets for novel object categories. The aim of this research would be to create a deep learning model that automatically generates such assets from a single input image. This is a very practical, but challenging task as the model must have an a priori understanding of the possible shapes and appearances of the object. Collecting 3D ground-truth data to learn this prior requires significant effort. Recently, there has been a growing interest to instead learn these priors from data that are widely available: Internet images. Learning from such data poses many challenges such as the lack of multiview constraints, noisy data, occlusions, lack of diverse viewpoints. Given these challenges, the existing state-of-the-art methods are not capable of achieving accurate, high-fidelity results and are limited to specific object categories. Novelty of the research methodology:To address the above limitations, I plan to combine the existing approaches with powerful pretrained 2D text-to-image diffusion models. These models have the potential to provide additional priors which would lead to more faithful 3D reconstructions. In addition, they can help the model to generalize on unseen categories without the need to collect additional training images. This work potentially would result in a novel state-of-the-art method. Any companies or collaborators involved: I am part of the ELLIS PhD program which is a pan European PhD Research programme, so ELLIS will be involved. My research will be in these area - Computer Vision, Deep Learning, AI Robustness and AI Ethics.