Graph Distillation for Action Detection with Privileged Modalities
Graph Distillation for Action Detection with Privileged Modalities
复制标题
DOI：
10.1007/978-3-030-01264-9_11
复制
发表时间：
2017-11
期刊：
影响因子：
--
通讯作者：
Zelun Luo;Jun-Ting Hsieh;Lu Jiang;Juan Carlos Niebles;Li Fei-Fei-Li-Fei-Fei-48004138
中科院分区：
文献类型：
--
作者：
Zelun Luo;Jun-Ting Hsieh;Lu Jiang;Juan Carlos Niebles;Li Fei-Fei-Li-Fei-Fei-48004138
我们提出了一种技术，可以在现实且具有挑战性的条件下处理多模态视频中的动作检测，在该条件下，只有有限的训练数据和部分观察到的模态可用。迁移学习中的常见方法没有利用源域中潜在可用的额外模式。另一方面，以前的多模态学习工作仅关注单个领域或任务，没有处理训练和测试之间的模态差异。在这项工作中，我们提出了一种称为图蒸馏的方法，该方法结合了源域中大规模多模态数据集的丰富特权信息，并改进了训练数据和模式稀缺的目标域中的学习。我们评估了多模态视频中的动作分类和检测任务的方法，并表明我们的模型在 NTU RGB+ D 和 PKU-MMD 基准上大幅优于最先进的模型。代码发布于http://alan。愿景/eccv18_graph/。
We propose a technique that tackles action detection in multimodal videos under a realistic and challenging condition in which only limited training data and partially observed modalities are available. Common methods in transfer learning do not take advantage of the extra modalities potentially available in the source domain. On the other hand, previous work on multimodal learning only focuses on a single domain or task and does not handle the modality discrepancy between training and testing. In this work, we propose a method termed graph distillation that incorporates rich privileged information from a large-scale multimodal dataset in the source domain, and improves the learning in the target domain where training data and modalities are scarce. We evaluate our approach on action classification and detection tasks in multimodal videos, and show that our model outperforms the state-of-the-art by a large margin on the NTU RGB+ D and PKU-MMD benchmarks. The code is released at http://alan. vision/eccv18_graph/.