Diffusion models for Single-image 3D reconstruction of deformable objects.

点击翻译按钮获取中文摘要

Brief description of the context of the research including the potential impact: 3D reconstruction of deformable objects has a wealth of potential applications across various fields, including AR/VR, gaming, and animal behaviour research. However, the creation and animation of these 3D models requires significant effort and expert knowledge of a 3D artist. This presents a significant barrier for creating diverse and abundant 3D environments, as well as generating 3D assets for novel object categories. The aim of this research would be to create a deep learning model that automatically generates such assets from a single input image. This is a very practical, but challenging task as the model must have an a priori understanding of the possible shapes and appearances of the object. Collecting 3D ground-truth data to learn this prior requires significant effort. Recently, there has been a growing interest to instead learn these priors from data that are widely available: Internet images. Learning from such data poses many challenges such as the lack of multiview constraints, noisy data, occlusions, lack of diverse viewpoints. Given these challenges, the existing state-of-the-art methods are not capable of achieving accurate, high-fidelity results and are limited to specific object categories. Novelty of the research methodology:To address the above limitations, I plan to combine the existing approaches with powerful pretrained 2D text-to-image diffusion models. These models have the potential to provide additional priors which would lead to more faithful 3D reconstructions. In addition, they can help the model to generalize on unseen categories without the need to collect additional training images. This work potentially would result in a novel state-of-the-art method. Any companies or collaborators involved: I am part of the ELLIS PhD program which is a pan European PhD Research programme, so ELLIS will be involved. My research will be in these area - Computer Vision, Deep Learning, AI Robustness and AI Ethics.