Inferring 3D Shapes from Image Collections Using Adversarial Networks
Inferring 3D Shapes from Image Collections Using Adversarial Networks
复制标题
DOI：
10.1007/s11263-020-01335-w
复制
发表时间：
2020-06-24
影响因子：
19.5
通讯作者：
Wang, Rui
中科院分区：
文献类型：
--
作者：
Gadelha, Matheus;Rai, Aartika;Wang, Rui
我们研究了在三维形状上学习概率分布的问题，给出了从未知视点获取的多个物体的二维视图。我们的方法称为投影生成对抗网络（PrGAN），该方法训练3D形状的深度生成模型，其投影（或渲染）与所提供的2D视图的分布相匹配。可微投影模块的添加使我们能够在学习阶段推断底层3D形状分布，而无需访问任何显式3D或视点注释。我们表明，我们的方法产生的3D形状的质量与直接在3D数据上训练的gan相当。实验还表明，将二维形状分解为几何和视点，可以得到一个良好的二维形状生成模型。我们的模型的关键优势在于它可以估计3D形状，视点，并以完全无监督的方式从输入图像中生成新的视图。我们进一步研究了如何在训练时提供诸如深度、视点或部分分割等额外信息的情况下改进生成模型。为此，我们提出了新的可微投影算子，可以用来学习更好的三维生成模型。我们的实验表明，prgan可以成功地利用额外的视觉线索来创造更多样化和准确的形状。
We investigate the problem of learning a probabilistic distribution over three-dimensional shapes given two-dimensional views of multiple objects taken from unknown viewpoints. Our approach calledprojective generative adversarial network(PrGAN) trains a deep generative model of 3D shapes whose projections (or renderings) matches the distribution of the provided 2D views. The addition of adifferentiable projection moduleallows us to infer the underlying 3D shape distribution without access to any explicit 3D or viewpoint annotation during the learning phase. We show that our approach produces 3D shapes of comparable quality to GANs trained directly on 3D data. Experiments also show that the disentangled representation of 2D shapes into geometry and viewpoint leads to a good generative model of 2D shapes. The key advantage of our model is that it estimates 3D shape, viewpoint, and generates novel views from an input image in a completely unsupervised manner. We further investigate how the generative models can be improved if additional information such as depth, viewpoint or part segmentations is available at training time. To this end, we present new differentiable projection operators that can be used to learn better 3D generative models. Our experiments show thatPrGANcan successfully leverage extra visual cues to create more diverse and accurate shapes.