Inferring implicit 3D representations from human figures on pictorial maps
Inferring implicit 3D representations from human figures on pictorial maps
复制标题
从图画地图上的人物推断隐式 3D 表示
DOI：
10.1080/15230406.2023.2224063
复制
发表时间：
2023
影响因子：
2.5
通讯作者：
Schnürer R
中科院分区：
文献类型：
--
作者：
Schnürer R
在这项工作中，我们提出了一种自动化的工作流程，将人物形象--图形地图上最常见的实体之一--带入三维。我们的工作流程基于训练数据和神经网络，用于从照片中重建真实人体的单视图3D。我们首先让一个由完全连通的层组成的网络来估计2D位置点的深度坐标。将获得的3D位姿点与人体部位的2D模板一起输入到深度隐式曲面网络中，以推断3D符号距离场(SDF)。通过对人体各部位的拼接，得到不同视角下人体整体的2D深度图像和身体部位模板，并将其送入一个全卷积网络进行UV图像预测。这些UV图像和给定透视的纹理被插入到生成网络中，以内嵌其他视图的纹理。纹理由卡通化网络增强，面部细节由自动编码器重新合成。最后，在光线行进器中将生成的纹理指定给推断的身体部位。在验证了几种网络配置后，我们使用12个图画人物测试了我们的工作流。创建的3D模型总体上看起来很有希望，特别是考虑到基于轮廓的3D恢复和隐式SDF的实时渲染的挑战。需要进一步的改进，以减少身体部分之间的差距，并为纹理增加图画细节。总体而言，构建的图形可用于数字3D地图中的动画和故事讲述。
In this work, we present an automated workflow to bring human figures, one of the most frequently appearing entities on pictorial maps, to the third dimension. Our workflow is based on training data and neural networks for single-view 3D reconstruction of real humans from photos. We first let a network consisting of fully connected layers estimate the depth coordinate of 2D pose points. The gained 3D pose points are inputted together with 2D masks of body parts into a deep implicit surface network to infer 3D signed distance fields (SDFs). By assembling all body parts, we derive 2D depth images and body part masks of the whole figure for different views, which are fed into a fully convolutional network to predict UV images. These UV images and the texture for the given perspective are inserted into a generative network to inpaint the textures for the other views. The textures are enhanced by a cartoonization network and facial details are resynthesized by an autoencoder. Finally, the generated textures are assigned to the inferred body parts in a ray marcher. We test our workflow with 12 pictorial human figures after having validated several network configurations. The created 3D models look generally promising, especially when considering the challenges of silhouette-based 3D recovery and real-time rendering of the implicit SDFs. Further improvement is needed to reduce gaps between the body parts and to add pictorial details to the textures. Overall, the constructed figures may be used for animation and storytelling in digital 3D maps.