Semantic Collaborative Learning for Cross-Modal Moment Localization
Semantic Collaborative Learning for Cross-Modal Moment Localization
复制标题
DOI：
10.1145/3620669
复制
发表时间：
2023-09
影响因子：
5.6
通讯作者：
Yupeng Hu;Kun Wang;Meng Liu;Haoyu Tang;Liqiang Nie
中科院分区：
文献类型：
--
作者：
Yupeng Hu;Kun Wang;Meng Liu;Haoyu Tang;Liqiang Nie
通过给定的自然语言查询在未修剪的视频中定位所需的时刻，即跨模态时刻定位，最近引起了广泛的研究关注。然而，这是一项具有挑战性的任务，因为它不仅需要准确地理解模态内的语义信息，还需要显式地捕获模态间的语义相关性（一致性和互补性）。现有的研究主要集中在模态内的语义理解和模态间的语义对齐，而忽略了必要的语义补充。因此，我们提出了一个跨模态语义感知网络，以实现更有效的模态内语义理解和模态间语义协作。具体而言，我们设计了一个双路径表示网络用于模态内语义建模。同时，我们开发了一个语义协同网络，实现了多粒度语义对齐和分层语义补充。因此，在充分的语义协同学习的基础上，可以实现有效的矩定位。大量的对比实验表明，与现有的最先进的竞争对手相比，我们的模型具有良好的性能。
Localizing a desired moment within an untrimmed video via a given natural language query, i.e., cross-modal moment localization, has attracted widespread research attention recently. However, it is a challenging task because it requires not only accurately understanding intra-modal semantic information, but also explicitly capturing inter-modal semantic correlations (consistency and complementarity). Existing efforts mainly focus on intra-modal semantic understanding and inter-modal semantic alignment, while ignoring necessary semantic supplement. Consequently, we present a cross-modal semantic perception network for more effective intra-modal semantic understanding and inter-modal semantic collaboration. Concretely, we design a dual-path representation network for intra-modal semantic modeling. Meanwhile, we develop a semantic collaborative network to achieve multi-granularity semantic alignment and hierarchical semantic supplement. Thereby, effective moment localization can be achieved based on sufficient semantic collaborative learning. Extensive comparison experiments demonstrate the promising performance of our model compared with existing state-of-the-art competitors.