The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines
复制标题
DOI：
10.1109/tpami.2020.2991965
复制
发表时间：
2021-11-01
影响因子：
23.6
通讯作者：
Wray, Michael
中科院分区：
文献类型：
--
作者：
Damen, Dima;Doughty, Hazel;Wray, Michael
自2018年推出以来，EPIC-KITCHENS作为最大的以自我为中心的视频基准备受关注，为人们与物体的互动、注意力甚至意图提供了独特的视角。在本文中，我们详细介绍了32名参与者如何在他们的原生厨房环境中捕获这个大规模数据集，并通过动作和对象交互进行密集注释。我们的视频记录了无脚本的日常活动，因为每次参与者进入他们的厨房都会开始录音。来自10个不同国家的参与者在4个国家进行了记录，这导致了厨房习惯和烹饪风格的高度多样化。我们的数据集包含55小时的视频，由11.5M帧组成，我们密集地标记了总共39.6K的动作片段和454.2K的对象边界框。我们的注释是独特的，因为我们让参与者讲述他们自己的视频（录制后），从而反映了真实的意图，我们在这些基础上众包事实。我们描述了我们的目标，行动和预期挑战，并评估了两个测试分割的几个基线，看到和看不到的厨房。我们引入了新的基线，突出了数据集的多模态性质，以及明确的时间建模对区分细粒度动作（例如，“关闭”和“打开”）的重要性。
Since its introduction in 2018, EPIC-KITCHENS has attracted attention as the largest egocentric video benchmark, offering a unique viewpoint on people's interaction with objects, their attention, and even intention. In this paper, we detail how this large-scale dataset was captured by 32 participants in their native kitchen environments, and densely annotated with actions and object interactions. Our videos depict nonscripted daily activities, as recording is started every time a participant entered their kitchen. Recording took place in four countries by participants belonging to ten different nationalities, resulting in highly diverse kitchen habits and cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.2K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens. We introduce new baselines that highlight the multimodal nature of the dataset and the importance of explicit temporal modelling to discriminate fine-grained actions (e.g., 'closing a tap' from 'opening' it up).