Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
复制标题
DOI：
--
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Youngjoon Kwon;Dahun Kim;Duygu Ceylan;H. Fuchs
中科院分区：
文献类型：
--
作者：
Youngjoon Kwon;Dahun Kim;Duygu Ceylan;H. Fuchs
在本文中，我们的目标是合成一个自由视点视频的任意人的表现，使用稀疏的多视角相机。最近，有几项研究通过学习特定于人的神经辐射场（NeRF）来捕捉特定人的外观来解决这个问题。同时，一些工作提出使用像素对齐的特征来将辐射场推广到任意新的场景和对象。然而，由于身体部位的严重闭塞和动态关节，将这种泛化方法应用于人类是非常具有挑战性的。为了解决这个问题，我们提出了神经人类表演者，这是一种新的方法，它基于参数化人体模型学习可泛化的神经辐射场，以实现鲁棒的性能捕获。具体来说，我们首先介绍了一个时间Transformer，聚合跟踪的视觉功能的基础上，随着时间的推移的骨骼身体运动。此外，多视图Transformer被提出来在每个时间步执行时间融合特征和像素对齐特征之间的交叉注意，以整合来自多个视图的动态观察。在ZJU-MoCap和AIST数据集上的实验表明，我们的方法在看不见的身份和姿态上显着优于最近的可推广的NeRF方法。视频结果和代码可在https://youngjoongunc.github.io/nhp上获得。
In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Specifically, we first introduce a temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, a multi-view transformer is proposed to perform cross-attention between the temporally-fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses. The video results and code are available at https://youngjoongunc.github.io/nhp.