EgoRenderer: Rendering Human Avatars from Egocentric Camera Images
EgoRenderer: Rendering Human Avatars from Egocentric Camera Images
复制标题
DOI：
10.1109/iccv48922.2021.01426
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
T. Hu;Kripasindhu Sarkar;Lingjie Liu;Matthias Zwicker;C. Theobalt
中科院分区：
文献类型：
--
作者：
T. Hu;Kripasindhu Sarkar;Lingjie Liu;Matthias Zwicker;C. Theobalt
我们推出了 EgoRenderer，这是一个用于渲染由安装在帽子或 VR 耳机上的可穿戴、以自我为中心的鱼眼相机捕获的人的全身神经化身的系统。我们的系统从任意虚拟摄像机位置呈现演员及其动作的逼真新颖视图。由于自上而下的视图和较大的扭曲，从这种以自我为中心的图像中渲染全身头像面临着独特的挑战。我们通过将渲染过程分解为几个步骤来应对这些挑战，包括纹理合成、姿势构建和神经图像翻译。对于纹理合成，我们提出了 Ego-DPNet，这是一种神经网络，可以推断输入鱼眼图像和底层参数化身体模型之间的密集对应关系，并从以自我为中心的输入中提取纹理。此外，为了对动态外观进行编码，我们的方法还学习了隐式纹理堆栈，该堆栈可以捕获姿势和视点之间的详细外观变化。为了生成正确的姿势，我们首先使用参数模型从自我中心的角度估计身体姿势。然后，我们通过将参数模型投影到用户指定的目标视点来合成外部自由视点姿势图像。接下来，我们将目标姿态图像和纹理组合成组合特征图像，使用神经图像翻译网络将其转换为输出彩色图像。实验评估表明，EgoRenderer 能够生成佩戴自我中心相机的人的逼真的自由视角化身。与几个基线的比较证明了我们方法的优点。
We present EgoRenderer, a system for rendering full-body neural avatars of a person captured by a wearable, egocentric fisheye camera that is mounted on a cap or a VR headset. Our system renders photorealistic novel views of the actor and her motion from arbitrary virtual camera locations. Rendering full-body avatars from such egocentric images come with unique challenges due to the top-down view and large distortions. We tackle these challenges by decomposing the rendering process into several steps, including texture synthesis, pose construction, and neural image translation. For texture synthesis, we propose Ego-DPNet, a neural network that infers dense correspondences between the input fisheye images and an underlying parametric body model, and to extract textures from egocentric inputs. In addition, to encode dynamic appearances, our approach also learns an implicit texture stack that captures detailed appearance variation across poses and viewpoints. For correct pose generation, we first estimate body pose from the egocentric view using a parametric model. We then synthesize an external free-viewpoint pose image by projecting the parametric model to the user-specified target viewpoint. We next combine the target pose image and the textures into a combined feature image, which is transformed into the output color image using a neural image translation network. Experimental evaluations show that EgoRenderer is capable of generating realistic free-viewpoint avatars of a person wearing an egocentric camera. Comparisons to several baselines demonstrate the advantages of our approach.