Action Localization Through Continual Predictive Learning
Action Localization Through Continual Predictive Learning
复制标题
通过持续预测学习进行动作本地化
DOI：
10.1007/978-3-030-58568-6_18
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Sarkar, Sudeep
中科院分区：
文献类型：
--
作者：
Aakur, Sathyanarayanan;Sarkar, Sudeep
动作定位问题涉及在图像中随时间和空间定位视频中的动作。当前的主流方法使用监督学习来解决这个问题。它们需要大量带注释的训练数据，以感兴趣区域周围的帧级边界框注释的形式。在本文中，我们提出了一种基于持续学习的新方法，该方法使用特征级预测进行自我监督。它不需要任何帧级边界框方面的训练注释。该方法受到视觉事件感知认知模型的启发，该模型提出了一种基于预测的事件理解方法。我们使用一堆 LSTM 与 CNN 编码器相结合，以及新颖的注意力机制，对视频中的事件进行建模，并使用该模型来预测未来帧的高级特征。预测误差用于连续学习模型的参数。这种自我监督框架并不像其他方法那样复杂，但在学习标记和本地化的鲁棒视觉表示方面非常有效。应该指出的是，该方法以流式传输方式输出，只需要一次通过视频，使其适合实时处理。我们在三个数据集——UCF Sports、JHMDB 和 THUMOS’13 上证明了这一点，并表明所提出的方法优于弱监督和无监督基线，并且与完全监督基线相比获得了有竞争力的性能。最后，我们表明所提出的框架可以推广到以自我为中心的视频，并在无监督注视预测任务上取得最先进的结果。代码可在项目页面上找到（https://saakur.github.io/Projects/ActionLocalization/）。
The problem of action localization involves locating the action in the video, both over time and spatially in the image. The current dominant approaches use supervised learning to solve this problem. They require large amounts of annotated training data, in the form of frame-level bounding box annotations around the region of interest. In this paper, we present a new approach based on continual learning that uses feature-level predictions for self-supervision. It does not require any training annotations in terms of frame-level bounding boxes. The approach is inspired by cognitive models of visual event perception that propose a prediction-based approach to event understanding. We use a stack of LSTMs coupled with a CNN encoder, along with novel attention mechanisms, to model the events in the video and use this model to predict high-level features for the future frames. The prediction errors are used to learn the parameters of the models continuously. This self-supervised framework is not complicated as other approaches but is very effective in learning robust visual representations for both labeling and localization. It should be noted that the approach outputs in a streaming fashion, requiring only a single pass through the video, making it amenable for real-time processing. We demonstrate this on three datasets - UCF Sports, JHMDB, and THUMOS’13 and show that the proposed approach outperforms weakly-supervised and unsupervised baselines and obtains competitive performance compared to fully supervised baselines. Finally, we show that the proposed framework can generalize to egocentric videos and achieve state-of-the-art results on theunsupervisedgaze prediction task. Code is available on the project page( https://saakur.github.io/Projects/ActionLocalization/ .).