Unified Spatio-Temporal Attention Networks for Action Recognition in Videos
Unified Spatio-Temporal Attention Networks for Action Recognition in Videos
复制标题
用于视频中动作识别的统一时空注意力网络
DOI：
10.1109/tmm.2018.2862341
复制
发表时间：
2019-02-01
影响因子：
7.3
通讯作者：
Rui, Yong
中科院分区：
文献类型：
--
作者：
Li, Dong;Yao, Ting;Rui, Yong
识别视频中的动作并不是一项简单的任务，因为视频是一种信息密集型媒体，包含多种模式。此外，在每个模态上，动作可以只出现在一些空间区域，或者只有部分时间视频片段可以包含该动作。一个有效的问题是如何定位出席的空间区域和选择视频片段的行动识别。在本文中，我们设计了一种称为AttCell的通用注意力神经细胞，它不仅可以估计每个空间位置的注意力概率，还可以估计时间序列中每个视频片段的注意力概率。利用AttCell，提出了一个多模态下的统一时空注意网络（STAN）。具体来说，STAN提取一个卷积层的特征映射作为每个模态上的局部描述符，并将提取的描述符与AttCell测量的空间注意力池作为每个段的表示。然后，我们将每个模态上的表示连接起来，先验地寻求对时间注意力的共识，将视频片段的组合表示整体地融合到视频表示中以进行识别。我们的模型不同于传统的深度网络，后者专注于注意力机制，因为我们的时间注意力在不同的模式和视频片段之间提供了原则性的和全局的指导。在四个公共数据集上进行了广泛的实验；UCF101、CCV、THUMOS14、Sports-1M；我们的STAN始终如一地在几种最先进的技术中取得卓越的效果。更值得注意的是，当利用不同数量的模式时，我们验证并证明了我们的建议的有效性。
Recognizing actions in videos is not a trivial task because video is an information-intensive media and includes multiple modalities. Moreover, on each modality, an action may only appear at some spatial regions, or only part of the temporal video segments may contain the action. A valid question is how to locate the attended spatial areas and selective video segments for action recognition. In this paper, we devise a general attention neural cell, called AttCell, that estimates the attention probability not only at each spatial location but also for each video segment in a temporal sequence. With AttCell, a unified Spatio-Temporal Attention Networks (STAN) is proposed in the context of multiple modalities. Specifically, STAN extracts the feature map of one convolutional layer as the local descriptors on each modality and pools the extracted descriptors with the spatial attention measured by AttCell as a representation of each segment. Then, we concatenate the representation on each modality to seek a consensus on the temporal attention, a priori, to holistically fuse the combined representation of video segments to the video representation for recognition. Our model differs from conventional deep networks, which focus on the attention mechanism, because our temporal attention provides a principled and global guidance across different modalities and video segments. Extensive experiments are conducted on four public datasets; UCF101, CCV, THUMOS14, and Sports-1M; our STAN consistently achieves superior results over several state-of-the-art techniques. More remarkably, we validate and demonstrate the effectiveness of our proposal when capitalizing on the different number of modalities.