EPIC Fields: Marrying 3D Geometry and Video Understanding
EPIC Fields: Marrying 3D Geometry and Video Understanding
复制标题
DOI：
10.48550/arxiv.2306.08731
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Vadim Tschernezki;Ahmad Darkhalil;Zhifan Zhu;D. Fouhey;Iro Laina;Diane Larlus;D. Damen;A. Vedaldi-A.-Vedal
中科院分区：
文献类型：
--
作者：
Vadim Tschernezki;Ahmad Darkhalil;Zhifan Zhu;D. Fouhey;Iro Laina;Diane Larlus;D. Damen;A. Vedaldi-A.-Vedal
神经渲染正在推动学习，3D几何和视频理解的统一，这已经等待了二十多年。然而，由于缺乏适当的数据集和基准，进展仍然受到阻碍。为了解决这一差距，我们引入了EPIC Fields，这是一种带有3D相机信息的EPIC-KITCHENS增强版。与其他用于神经渲染的数据集一样，EPIC Fields消除了使用摄影测量重建相机的复杂而昂贵的步骤，并允许研究人员专注于建模问题。我们说明了摄影测量的挑战，以自我为中心的动态动作的视频，并提出创新，以解决这些问题。与其他神经渲染数据集相比，EPIC Fields更适合视频理解，因为它与标记的动作片段和最近的VISOR片段注释配对。为了进一步激励社区，我们还评估了神经渲染和分割动态对象的两个基准任务，并提供了强大的基线，展示了今天不可能实现的目标。我们还强调了几何的优势，在半监督视频对象分割的VISOR注释。EPIC Fields重建了EPICKITCHENS中96%的视频，在45个厨房记录的99小时内记录了1900万帧。
Neural rendering is fuelling a unification of learning, 3D geometry and video understanding that has been waiting for more than two decades. Progress, however, is still hampered by a lack of suitable datasets and benchmarks. To address this gap, we introduce EPIC Fields, an augmentation of EPIC-KITCHENS with 3D camera information. Like other datasets for neural rendering, EPIC Fields removes the complex and expensive step of reconstructing cameras using photogrammetry, and allows researchers to focus on modelling problems. We illustrate the challenge of photogrammetry in egocentric videos of dynamic actions and propose innovations to address them. Compared to other neural rendering datasets, EPIC Fields is better tailored to video understanding because it is paired with labelled action segments and the recent VISOR segment annotations. To further motivate the community, we also evaluate two benchmark tasks in neural rendering and segmenting dynamic objects, with strong baselines that showcase what is not possible today. We also highlight the advantage of geometry in semi-supervised video object segmentations on the VISOR annotations. EPIC Fields reconstructs 96% of videos in EPICKITCHENS, registering 19M frames in 99 hours recorded in 45 kitchens.