A Multi-Modal Transformer Network for Action Detection
A Multi-Modal Transformer Network for Action Detection
复制标题
DOI：
10.1016/j.patcog.2023.109713
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Matthew Korban;S. Acton;Peter Youngs
中科院分区：
文献类型：
--
作者：
Matthew Korban;S. Acton;Peter Youngs
本文提出了一种新的多模态变压器网络，用于检测未修剪视频中的动作。为了丰富动作特征，我们的变压器网络利用了一种新的多模态注意机制来计算不同空间和运动模态组合之间的相关性。以前没有人尝试探索行动之间的这种相关性。为了更有效地利用运动和空间模态，我们提出了一种校正由摄像机运动引起的运动失真的算法。这种运动失真在未经修剪的视频中很常见，严重降低了光流场等运动特征的表现力。我们提出的算法在两个公共基准上优于最先进的方法，THUMOS14和ActivityNet。我们还对新的教学活动数据集进行了对比实验，其中包括从小学捕获的大量具有挑战性的课堂视频。
This paper proposes a novel multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our transformer network utilizes a new multi-modal attention mechanism that computes the correlations between different spatial and motion modalities combinations. Exploring such correlations for actions has not been attempted previously. To use the motion and spatial modality more effectively, we suggest an algorithm that corrects the motion distortion caused by camera movement. Such motion distortion, common in untrimmed videos, severely reduces the expressive power of motion features such as optical flow fields. Our proposed algorithm outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and ActivityNet. We also conducted comparative experiments on our new instructional activity dataset, including a large set of challenging classroom videos captured from elementary schools.