Computer Vision - ACCV 2022 - 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV
Computer Vision - ACCV 2022 - 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part IV
复制标题
计算机视觉 - ACCV 2022 - 第十六届亚洲计算机视觉会议，中国澳门，2022 年 12 月 4-8 日，会议记录，第四部分
DOI：
10.1007/978-3-031-26316-3_27
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Fragomeni A
中科院分区：
文献类型：
--
作者：
Fragomeni A
在本文中，我们重新审视跨模态剪辑句子检索的任务，剪辑是一个较长的未修剪的视频的一部分。当剪辑是短的或视觉上模糊的，它的本地时间上下文（即周围的视频片段）的知识可以用来提高检索性能。提出了上下文转换器Transformer;一种编码器架构，其对视频剪辑与其局部时间上下文之间的交互进行建模，以便增强其嵌入式表示。重要的是，我们监督上下文Transformer使用对比损失的跨模态嵌入空间。我们探索视频和文本形式的上下文转换器。结果一致表明，在三个数据集上的性能有所提高：YouCook 2，EPIC-KITCHENS和ActivityNet Captions的剪辑句子版本。详尽的消融研究和上下文分析表明所提出的方法的有效性。
In this paper, we re-examine the task of cross-modal clip-sentence retrieval, where the clip is part of a longer untrimmed video. When the clip is short or visually ambiguous, knowledge of its local temporal context (ie surrounding video segments) can be used to improve the retrieval performance. We propose Context Transformer; an encoder architecture that models the interaction between a video clip and its local temporal context in order to enhance its embedded representations. Importantly, we supervise the context transformer using contrastive losses in the cross-modal embedding space. We explore context transformers for video and text modalities. Results consistently demonstrate improved performance on three datasets: YouCook2, EPIC-KITCHENS and a clip-sentence version of ActivityNet Captions. Exhaustive ablation studies and context analysis show the efficacy of the proposed method.