TCGL: Temporal Contrastive Graph for Self-Supervised Video Representation Learning
TCGL: Temporal Contrastive Graph for Self-Supervised Video Representation Learning
复制标题
DOI：
10.1109/tip.2022.3147032
复制
发表时间：
2022-01-01
影响因子：
10.6
通讯作者：
Lin, Liang
中科院分区：
文献类型：
--
作者：
Liu, Yang;Wang, Keze;Lin, Liang
视频自监督学习是一项具有挑战性的任务，它需要模型具有强大的表达能力，以利用丰富的时空知识，并从大量未标记的视频中生成有效的监督信号。然而，现有的方法未能增加时间的多样性的未标记的视频和忽视精心建模的多尺度时间依赖性，在一个明确的方式。为了克服这些局限性，我们利用视频中的多尺度时间依赖性，提出了一种新的视频自监督学习框架，称为时间对比图学习（TCGL），它联合建模片段间和片段内的时间依赖性与混合图对比学习策略的时间表示学习。具体来说，时空知识发现（STKD）模块首先介绍了从视频中提取运动增强的时空表示的离散余弦变换的频域分析的基础上。为了明确地对未标记视频的多尺度时间依赖性进行建模，我们的TCGL将关于帧和片段顺序的先验知识集成到图结构中，即，片段内/片段间时间对比图（TCG）。然后，设计了具体的对比学习模块，以最大限度地提高不同图视图中节点之间的一致性。为了为未标记的视频生成监督信号，我们引入了自适应片段顺序预测（ASOP）模块，该模块利用视频片段之间的关系知识来学习全局上下文表示，并自适应地重新校准通道特征。实验结果表明，我们的TCGL在大规模动作识别和视频检索基准的国家的最先进的方法的优越性。该代码可在https://github.com/YangLiu9208/TCGL上公开获取。
Video self-supervised learning is a challenging task, which requires significant expressive power from the model to leverage rich spatial-temporal knowledge and generate effective supervisory signals from large amounts of unlabeled videos. However, existing methods fail to increase the temporal diversity of unlabeled videos and ignore elaborately modeling multi-scale temporal dependencies in an explicit way. To overcome these limitations, we take advantage of the multi-scale temporal dependencies within videos and propose a novel video self-supervised learning framework named Temporal Contrastive Graph Learning (TCGL), which jointly models the inter-snippet and intra-snippet temporal dependencies for temporal representation learning with a hybrid graph contrastive learning strategy. Specifically, a Spatial-Temporal Knowledge Discovering (STKD) module is first introduced to extract motion-enhanced spatial-temporal representations from videos based on the frequency domain analysis of discrete cosine transform. To explicitly model multi-scale temporal dependencies of unlabeled videos, our TCGL integrates the prior knowledge about the frame and snippet orders into graph structures, i.e., the intra-/inter-snippet Temporal Contrastive Graphs (TCG). Then, specific contrastive learning modules are designed to maximize the agreement between nodes in different graph views. To generate supervisory signals for unlabeled videos, we introduce an Adaptive Snippet Order Prediction (ASOP) module which leverages the relational knowledge among video snippets to learn the global context representation and recalibrate the channel-wise features adaptively. Experimental results demonstrate the superiority of our TCGL over the state-of-the-art methods on large-scale action recognition and video retrieval benchmarks. The code is publicly available at https://github.com/YangLiu9208/TCGL.