A Unified Framework for Multi-Modal Isolated Gesture Recognition
A Unified Framework for Multi-Modal Isolated Gesture Recognition
复制标题
多模态孤立手势识别的统一框架
DOI：
10.1145/3131343
复制
发表时间：
2018-04-01
影响因子：
5.1
通讯作者：
Li, Stan Z.
中科院分区：
文献类型：
--
作者：
Duan, Jiali;Wan, Jun;Li, Stan Z.
在本文中，我们专注于孤立的手势识别，并通过涉及RGB流，深度流和显着性流来探索不同的模态。我们的目标是通过提出一个利用多模态融合优势的统一框架，进一步推动这一领域的边界。具体而言，时空网络架构的共识投票的基础上提出了明确的模型的长期结构的视频序列，并减少估计方差时，面对全面的类间变化。此外，还并行聚合了一个三维深度-显著性卷积网络，以捕捉细微的运动特征。进行了大量的实验来分析每个组件的性能，我们提出的方法在两个公共基准测试ChaLearn IsoGD和RGBD-HuDaAct上取得了最好的结果，分别超过了最接近的竞争对手10%和15%。我们的项目和代码将在https://davidsonic.github.io/index/acm_tomm_2017.html上发布。
In this article, we focus on isolated gesture recognition and explore different modalities by involving RGB stream, depth stream, and saliency stream for inspection. Our goal is to push the boundary of this realm even further by proposing a unified framework that exploits the advantages of multi-modality fusion. Specifically, a spatial-temporal network architecture based on consensus-voting has been proposed to explicitly model the long-term structure of the video sequence and to reduce estimation variance when confronted with comprehensive inter-class variations. In addition, a three-dimensional depth-saliency convolutional network is aggregated in parallel to capture subtle motion characteristics. Extensive experiments are done to analyze the performance of each component and our proposed approach achieves the best results on two public benchmarks, ChaLearn IsoGD and RGBD-HuDaAct, outperforming the closest competitor by a margin of over 10% and 15%, respectively. Our project and codes will be released at https://davidsonic.github.io/index/acm_tomm_2017.html.