Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching
Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching
复制标题
DOI：
10.1109/iccv48922.2021.01224
复制
发表时间：
2021-08
期刊：
影响因子：
--
通讯作者：
Chao Chen;Zhizhong Han;Yu-Shen Liu;Matthias Zwicker
中科院分区：
文献类型：
--
作者：
Chao Chen;Zhizhong Han;Yu-Shen Liu;Matthias Zwicker
学习在没有3D监控的情况下生成3D点云是一个重要但具有挑战性的问题。目前的解决方案利用各种可区分的渲染器将生成的3D点云投影到2D图像平面上，并使用2D地面真实图像的每像素差分来训练深度神经网络。然而，这些解决方案仍在努力完全恢复3D形状的精细结构，如细管或平面。为了解决这一问题，我们提出了一种无监督的精细结构三维点云生成方法。具体地说，我们将3D点云学习归结为2D投影匹配问题。我们没有使用整个2D轮廓图像作为常规的像素监督，而是引入了结构自适应采样来随机采样轮廓内的2D点作为不规则点监督，从而缓解了从不同视角采样的一致性问题。该方法通过神经网络生成3D点云，该点云的2D投影与不同视角下的不规则点监督相匹配。我们的2D投影匹配方法使神经网络能够比使用每像素差分学习更准确的结构信息，特别是对于精细和薄的3D结构。该方法可以从不同分辨率的二维轮廓图像中恢复出精细的三维结构，并且对不同的采样方法和不规则点监督下的点数具有较强的鲁棒性。在广泛使用的基准测试中，我们的方法优于其他方法。我们的代码、数据和模型可在http-s://github.com/chenchao15/2D_projectionjnatching.上获得
Learning to generate 3D point clouds without 3D supervision is an important but challenging problem. Current solutions leverage various differentiable renderers to project the generated 3D point clouds onto a 2D image plane, and train deep neural networks using the per-pixel difference with 2D ground truth images. However, these solutions are still struggling to fully recover fine structures of 3D shapes, such as thin tubes or planes. To resolve this issue, we propose an unsupervised approach for 3D point cloud generation with fine structures. Specifically, we cast 3D point cloud learning as a 2D projection matching problem. Rather than using entire 2D silhouette images as a regular pixel supervision, we introduce structure adaptive sampling to randomly sample 2D points within the silhouettes as an irregular point supervision, which alleviates the consistency issue of sampling from different view angles. Our method pushes the neural network to generate a 3D point cloud whose 2D projections match the irregular point supervision from different view angles. Our 2D projection matching approach enables the neural network to learn more accurate structure information than using the per-pixel difference, especially for fine and thin 3D structures. Our method can recover fine 3D structures from 2D silhouette images at different resolutions, and is robust to different sampling methods and point number in irregular point supervision. Our method outperforms others under widely used benchmarks. Our code, data and models are available at http-s://github.com/chenchao15/2D_projectionjnatching.