Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning
Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning
复制标题
DOI：
10.48550/arxiv.2207.09644
复制
发表时间：
2022-07
期刊：
影响因子：
56.9
通讯作者：
Yuxiao Chen;Long Zhao;Jianbo Yuan;Yu Tian;Zhaoyang Xia;Shijie Geng;Ligong Han;Dimitris N. Metaxas
中科院分区：
文献类型：
--
作者：
Yuxiao Chen;Long Zhao;Jianbo Yuan;Yu Tian;Zhaoyang Xia;Shijie Geng;Ligong Han;Dimitris N. Metaxas
尽管完全监督的人类骨骼序列建模取得了成功，但利用自监督预训练进行骨骼序列表示学习一直是一个活跃的领域，因为大规模获取特定任务的骨骼注释是困难的。目前的研究主要集中在使用对比学习来学习视频级的时间和判别信息，但忽略了人体骨骼的分层时空性质。不同于这种肤浅的监督在视频层面上，我们提出了一个自我监督的分层预训练计划纳入一个分层的基于变换器的骨架序列编码器（Hi-TRS），明确捕捉空间，短期和长期的时间依赖性在帧，剪辑和视频级别，分别。为了评估所提出的具有Hi-TRS的自监督预训练方案，我们进行了广泛的实验，涵盖了三个基于机器人的下游任务，包括动作识别，动作检测和运动预测。在监督和半监督评估协议下，我们的方法达到了最先进的性能。此外，我们证明了我们的模型在预训练阶段学习的先验知识对不同的下游任务具有很强的转移能力。
Despite the success of fully-supervised human skeleton sequence modeling, utilizing self-supervised pre-training for skeleton sequence representation learning has been an active field because acquiring task-specific skeleton annotations at large scales is difficult. Recent studies focus on learning video-level temporal and discriminative information using contrastive learning, but overlook the hierarchical spatial-temporal nature of human skeletons. Different from such superficial supervision at the video level, we propose a self-supervised hierarchical pre-training scheme incorporated into a hierarchical Transformer-based skeleton sequence encoder (Hi-TRS), to explicitly capture spatial, short-term, and long-term temporal dependencies at frame, clip, and video levels, respectively. To evaluate the proposed self-supervised pre-training scheme with Hi-TRS, we conduct extensive experiments covering three skeleton-based downstream tasks including action recognition, action detection, and motion prediction. Under both supervised and semi-supervised evaluation protocols, our method achieves the state-of-the-art performance. Additionally, we demonstrate that the prior knowledge learned by our model in the pre-training stage has strong transfer capability for different downstream tasks.