Synthesizing 3D Shapes From Silhouette Image Collections Using Multi-Projection Generative Adversarial Networks
Synthesizing 3D Shapes From Silhouette Image Collections Using Multi-Projection Generative Adversarial Networks
复制标题
DOI：
10.1109/cvpr.2019.00568
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Xiao Li;Yue Dong;P. Peers;Xin Tong
中科院分区：
文献类型：
--
作者：
Xiao Li;Yue Dong;P. Peers;Xin Tong
我们提出了一种新的基于弱监督学习的方法，用于从未被遮挡的图像集合中生成特定类别的新型3D形状。我们的方法是弱监督的，并且仅需要来自未被遮挡的特定类别物体的轮廓标注。我们的方法不需要获取物体的3D形状、每个物体从不同视角的多次观测、图像内像素对应关系或任何视角标注。我们方法的关键是一种新颖的多投影生成对抗网络（MP - GAN），它训练一个3D形状生成器使其与3D形状的多个2D投影保持一致，并且无需直接获取这些3D形状。这是通过多个判别器实现的，这些判别器对从不同视角看到的3D形状的2D投影分布进行编码。此外，为了确定每个轮廓图像的视角信息，我们还在生成器合成的3D形状的可视化上训练一个视角预测网络。我们在训练生成器和训练视角预测网络之间迭代交替。我们在合成图像数据集和真实图像数据集上验证了我们的多投影GAN。此外，我们还表明，多投影GAN可以帮助从低维训练数据集中学习其他高维分布，例如从图像中学习特定材质类别的空间变化反射特性。
We present a new weakly supervised learning-based method for generating novel category-specific 3D shapes from unoccluded image collections. Our method is weakly supervised and only requires silhouette annotations from unoccluded, category-specific objects. Our method does not require access to the object’s 3D shape, multiple observations per object from different views, intra-image pixel correspondences, or any view annotations. Key to our method is a novel multi-projection generative adversarial network (MP-GAN) that trains a 3D shape generator to be consistent with multiple 2D projections of the 3D shapes, and without direct access to these 3D shapes. This is achieved through multiple discriminators that encode the distribution of 2D projections of the 3D shapes seen from a different views. Additionally, to determine the view information for each silhouette image, we also train a view prediction network on visualizations of 3D shapes synthesized by the generator. We iteratively alternate between training the generator and training the view prediction network. We validate our multi-projection GAN on both synthetic and real image datasets. Furthermore, we also show that multi-projection GANs can aid in learning other high-dimensional distributions from lower dimensional training datasets, such as material-class specific spatially varying reflectance properties from images.