Feature Re-Learning with Data Augmentation for Video Relevance Prediction
Feature Re-Learning with Data Augmentation for Video Relevance Prediction
复制标题
通过数据增强进行特征重新学习以进行视频相关性预测
DOI：
10.1109/tkde.2019.2947442
复制
发表时间：
2019
影响因子：
8.9
通讯作者：
Xirong Li
中科院分区：
文献类型：
--
作者：
Jianfeng Dong;Xun Wang;Leimin Zhang;Chaoxi Xu;Gang Yang;Xirong Li
预测两个给定视频之间的视觉内容的相关性是基于内容的视频推荐和检索的关键组成部分。由于预训练图像和视频卷积神经网络模型的可用性越来越高，深度视觉特征被广泛用于视频内容表示。然而，由于两个视频的相关性取决于任务，因此这种现成的功能并不总是对所有任务都是最佳的。此外，由于包括版权、隐私和安全在内的各种问题，人们可能只能访问预先计算的视频特征，而不是原始视频。在本文中，我们提出了特征再学习，以提高视频相关性预测，而不需要重新访问原始视频内容。特别地，通过仿射变换将给定的深度特征投影到新的空间中来实现再学习。我们通过一种新的负增强三重排序损失来优化再学习过程。为了生成更多的训练数据，我们提出了一种新的数据增强策略，它直接作用于帧级和视频级特征。在Hulu 2018年基于内容的视频相关性预测挑战赛的背景下进行的大量实验证明了所提出的方法的有效性及其基于内容的视频相关性预测的最新性能。
Predicting the relevance between two given videos with respect to their visual content is a key component for content-based video recommendation and retrieval. Thanks to the increasing availability of pre-trained image and video convolutional neural network models, deep visual features are widely used for video content representation. However, as how two videos are relevant is task-dependent, such off-the-shelf features are not always optimal for all tasks. Moreover, due to varied concerns including copyright, privacy and security, one might have access to only pre-computed video features rather than original videos. We propose in this paper feature re-learning for improving video relevance prediction, with no need of revisiting the original video content. In particular, re-learning is realized by projecting a given deep feature into a new space by an affine transformation. We optimize the re-learning process by a novel negative-enhanced triplet ranking loss. In order to generate more training data, we propose a new data augmentation strategy which works directly on frame-level and video-level features. Extensive experiments in the context of the Hulu Content-based Video Relevance Prediction Challenge 2018 justify the effectiveness of the proposed method and its state-of-the-art performance for content-based video relevance prediction.