Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
复制标题
DOI：
10.1109/iccv48922.2021.00175
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Max Bain;Arsha Nagrani;Gül Varol;Andrew Zisserman
中科院分区：
文献类型：
--
作者：
Max Bain;Arsha Nagrani;Gül Varol;Andrew Zisserman
我们在这项工作中的目标是视频-文本检索-特别是能够实现高效的文本到视频检索的联合嵌入。该领域的挑战包括视觉体系结构的设计和训练数据的性质，因为现有的大规模视频-文本训练数据集，如HowTo100M，噪声较大，因此只有通过大量的计算才能在规模上获得具有竞争力的性能。我们提出了一种端到端的可训练模型，该模型旨在利用大规模图像和视频字幕数据集。我们的模型是对最近的VIT和TimeForm体系结构的适应和扩展，包含了对空间和时间的关注。该模型是灵活的，可以在图像和视频文本数据集上进行训练，既可以独立进行，也可以联合进行。它按照课程学习计划进行训练，一开始将图像视为视频的“冻结”快照，然后逐渐学习在视频数据集上训练时关注不断增加的时间背景。我们还提供了一个新的视频-文本预训练数据集WebVid-2M，包括从互联网上抓取的200多万个带有弱字幕的视频。尽管我们在小一个数量级的数据集上进行了训练，但我们表明，这种方法在标准下行视频检索基准上产生了最先进的结果，包括MSR-VTT、MSVD、DiDeMo和LSMDC。
Our objective in this work is video-text retrieval – in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute.We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as ‘frozen’ snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.