Entropy guided attention network for weakly-supervised action localization
Entropy guided attention network for weakly-supervised action localization
复制标题
DOI：
10.1016/j.patcog.2022.108718
复制
发表时间：
2022-04
期刊：
影响因子：
--
通讯作者：
Yi Cheng;Ying Sun;Hehe Fan;Tao Zhuo;J. Lim;Mohan S. Kankanhalli
中科院分区：
文献类型：
--
作者：
Yi Cheng;Ying Sun;Hehe Fan;Tao Zhuo;J. Lim;Mohan S. Kankanhalli
弱监督时间动作定位（WTAL）的一个主要挑战是处理视频中的不同背景。为了对背景帧进行建模，大多数现有方法将其视为附加的操作类。然而，由于背景帧通常不共享公共语义，因此将所有不同的背景帧压缩到单个类中会阻碍网络优化。此外，网络会感到困惑，并且在对具有不可见背景帧的视频进行测试时往往会失败。为了解决这个问题，我们提出了一个熵引导的注意力网络（EGA-Net）来处理背景帧作为域外样本。具体而言，我们设计了一个两个分支模块，其中域分支通过学习类无关的注意力图来检测帧是否是动作，动作分支通过学习类特定的注意力图来识别帧的动作类别。通过聚合两个注意力图来模拟帧的联合域类分布，我们的EGA网络可以处理不同的背景。为了只使用视频级别的类别标签来训练类别无关的注意力地图，我们提出了熵引导损失（EGL），它使用熵作为监督信号来区分动作和背景。此外，我们提出了一个全球相似性损失（GSL），以提高行动特定的注意力地图通过行动类中心。在THUMOS 14、ActivityNet1.2和ActivityNet1.3数据集上的大量实验证明了EGA-Net的有效性。
One major challenge of Weakly-supervised Temporal Action Localization (WTAL) is to handle diverse backgrounds in videos. To model background frames, most existing methods treat them as an additional action class. However, because background frames usually do not share common semantics, squeezing all the different background frames into a single class hinders network optimization. Moreover, the network would be confused and tends to fail when tested on videos with unseen background frames. To address this problem, we propose an Entropy Guided Attention Network (EGA-Net) to treat background frames as out-of-domain samples. Specifically, we design a two-branch module, where a domain branch detects whether a frame is an action by learning a class-agnostic attention map, and an action branch recognizes the action category of the frame by learning a class-specific attention map. By aggregating the two attention maps to model the joint domain-class distribution of frames, our EGA-Net can handle varying backgrounds. To train the class-agnostic attention map with only the video-level class labels, we propose an Entropy Guided Loss (EGL), which employs entropy as the supervision signal to distinguish action and background. Moreover, we propose a Global Similarity Loss (GSL) to enhance the action-specific attention map via action class center. Extensive experiments on THUMOS14, ActivityNet1.2 and ActivityNet1.3 datasets demonstrate the effectiveness of our EGA-Net.