Learning Semantics-Preserving Attention and Contextual Interaction for Group Activity Recognition
Learning Semantics-Preserving Attention and Contextual Interaction for Group Activity Recognition
复制标题
学习语义——为群体活动识别保留注意力和情境交互
DOI：
10.1109/tip.2019.2914577
复制
发表时间：
2019
影响因子：
10.6
通讯作者：
Zhou Jie
中科院分区：
文献类型：
--
作者：
Tang Yansong;Lu Jiwen;Wang Zian;Yang Ming;Zhou Jie
在这篇论文中，我们探讨了群体活动识别的问题，通过学习语义保持注意和不同人之间的上下文交互。传统的方法通常通过池化操作将从个体中提取的特征聚合在一起，这缺乏物理意义，并且不能充分挖掘用于群体活动识别的上下文信息。为了解决这个问题，我们开发了一个语义保持师生（SPTS）网络架构。我们的SPTS网络首先学习语义域中的教师网络，该网络基于个人行为的单词对群体活动的单词进行分类。然后，我们设计了一个学生网络的外观域，根据输入的视频识别群体活动。在学习过程中，我们强制学生网络模仿教师网络。通过这种方式，我们将语义保持的注意力分配给不同的人，从而更有效地寻找关键人并丢弃误导人，而不需要额外的标记数据。此外，一组人本质上存在于一个基于图的结构中，其中人和他们的关系可以分别被视为图的节点和边。在此基础上，我们在教师网络和学生网络上构建了两个图卷积模块，以推理不同人之间的依赖关系。此外，我们还根据动作分割任务的中间特征扩展了我们的动作分割任务方法。在四个群体活动分析数据集上的实验结果清楚地表明，与现有技术相比，我们的方法具有上级性能。
In this paper, we investigate the problem of group activity recognition by learning semantics-preserving attention and contextual interaction among different people. Conventional methods usually aggregate the features extracted from individual persons by pooling operations, which lack physical meaning and cannot fully explore the contextual information for group activity recognition. To address this, we develop a Semantics-Preserving Teacher-Student (SPTS) networks architecture. Our SPTS networks first learn a Teacher Network in the semantic domain that classifies thewordof group activity based on thewordsof individual actions. Then, we design a Student Network in the appearance domain that recognizes the group activity according to the input video. We enforce the Student Network to mimic the Teacher Network in the learning procedure. In this way, we allocate semantics-preserving attention to different people, which is more effective to seek the key people and discard the misleading people, while no extra labeled data are required. Moreover, a group of people inherently lie in a graph-based structure, where the people and their relationship can be regarded as the nodes and edges of a graph, respectively. Based on this, we build two graph convolutional modules on both the Teacher Network and the Student Network to reason the dependency among different people. Furthermore, we extend our approach on action segmentation task based on its intermediate features. The experimental results on four datasets for group activity analysis clearly show the superior performance of our method in comparison with the state-of-the-art.