RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
复制标题
DOI：
10.1109/cvpr52729.2023.01213
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
Titas Anciukevicius;Zexiang Xu;Matthew Fisher;Paul Henderson;Hakan Bilen;N. Mitra;Paul Guerrero
中科院分区：
文献类型：
--
作者：
Titas Anciukevicius;Zexiang Xu;Matthew Fisher;Paul Henderson;Hakan Bilen;N. Mitra;Paul Guerrero
扩散模型目前实现了有条件和无条件图像生成的最先进的性能。然而，到目前为止，图像扩散模型不支持3D理解所需的任务，如视图一致的3D生成或单视图对象重建。在本文中，我们提出了RenderDiffusion，这是第一个用于3D生成和推理的扩散模型，仅使用单目2D监督进行训练。我们的方法的核心是一种新的图像去噪架构，该架构在每个去噪步骤中生成并呈现场景的中间三维表示。这在扩散过程中实施了强感应结构，提供了3D一致的表示，同时仅需要2D监督。生成的3D表示可以从任何视图渲染。我们在FFHQ、AFHQ、ShapeNet和CLEVR数据集上评估了RenderDiffusion，显示出在生成3D场景和从2D图像推断3D场景方面具有竞争力的性能。此外，我们基于扩散的方法允许我们使用2D修复来编辑3D场景。
Diffusion models currently achieve state-of-the-art performance for both conditional and unconditional image generation. However, so far, image diffusion models do not support tasks required for 3D understanding, such as view-consistent 3D generation or single-view object reconstruction. In this paper, we present RenderDiffusion, the first diffusion model for 3D generation and inference, trained using only monocular 2D supervision. Central to our method is a novel image denoising architecture that generates and renders an intermediate three-dimensional representation of a scene in each denoising step. This enforces a strong inductive structure within the diffusion process, providing a 3D consistent representation while only requiring 2D supervision. The resulting 3D representation can be rendered from any view. We evaluate RenderDiffusion on FFHQ, AFHQ, ShapeNet and CLEVR datasets, showing competitive performance for generation of 3D scenes and inference of 3D scenes from 2D images. Additionally, our diffusion-based approach allows us to use 2D inpainting to edit 3D scenes.