Unified Coarse-to-Fine Alignment for Video-Text Retrieval
Unified Coarse-to-Fine Alignment for Video-Text Retrieval
复制标题
DOI：
10.1109/iccv51070.2023.00264
复制
发表时间：
2023-09
期刊：
影响因子：
--
通讯作者：
Ziyang Wang;Yi-Lin Sung;Feng Cheng;Gedas Bertasius;Mohit Bansal
中科院分区：
文献类型：
--
作者：
Ziyang Wang;Yi-Lin Sung;Feng Cheng;Gedas Bertasius;Mohit Bansal
视频文本检索的规范方法利用视觉和文本信息之间的粗粒度或细粒度对齐。然而，根据文本查询检索正确的视频通常具有挑战性，因为它需要能够推理高级（场景）和低级（对象）视觉线索以及它们与文本查询的关系。为此，我们提出了一种统一的粗到精对齐模型，称为 UCoFiA。具体来说，我们的模型捕获不同粒度级别的跨模式相似性信息。为了减轻不相关视觉线索的影响，我们还应用交互式相似性聚合模块（ISA）来考虑不同视觉特征的重要性，同时聚合跨模式相似性以获得每个粒度的相似性得分。最后，我们应用 Sinkhorn-Knopp 算法在求和之前对每个级别的相似性进行归一化，从而缓解不同级别的过度代表性和代表性不足问题。通过共同考虑不同粒度的跨模态相似性，UCoFiA 允许多粒度对齐的有效统一。根据经验，UCoFiA 在多个视频文本检索基准上优于之前最先进的基于 CLIP 的方法，在 MSR-VTT、Activity-Net 和 DiDeMo 上的文本到视频检索 R@1 分别实现了 2.4%、1.4% 和 1.3% 的改进。我们的代码可在 https://github.com/Ziyang412/UCoFiA 上公开获取。
The canonical approach to video-text retrieval leverages a coarse-grained or fine-grained alignment between visual and textual information. However, retrieving the correct video according to the text query is often challenging as it requires the ability to reason about both high-level (scene) and low-level (object) visual clues and how they relate to the text query. To this end, we propose a Unified Coarse-to-fine Alignment model, dubbed UCoFiA. Specifically, our model captures the cross-modal similarity information at different granularity levels. To alleviate the effect of irrelevant visual clues, we also apply an Interactive Similarity Aggregation module (ISA) to consider the importance of different visual features while aggregating the cross-modal similarity to obtain a similarity score for each granularity. Finally, we apply the Sinkhorn-Knopp algorithm to normalize the similarities of each level before summing them, alleviating over- and under-representation issues at different levels. By jointly considering the cross-modal similarity of different granularity, UCoFiA allows the effective unification of multi-grained alignments. Empirically, UCoFiA outperforms previous state-of-the-art CLIP-based methods on multiple video-text retrieval benchmarks, achieving 2.4%, 1.4% and 1.3% improvements in text-to-video retrieval R@1 on MSR-VTT, Activity-Net, and DiDeMo, respectively. Our code is publicly available at https://github.com/Ziyang412/UCoFiA.