Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
复制标题
DOI：
10.1109/cvpr.2019.00208
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Yale Song;M. Soleymani
中科院分区：
文献类型：
--
作者：
Yale Song;M. Soleymani
视觉语义嵌入的目的是找到一个共享的潜在空间，相关的视觉和文本实例彼此接近。目前大多数方法都学习将实例映射到共享空间中的单个点的内射嵌入函数。不幸的是，内射嵌入不能有效地处理具有多种可能含义的多义实例;它充其量只能找到不同含义的平均表示。这阻碍了它在真实世界场景中的使用，其中单个实例及其跨模式关联通常是模糊的。在这项工作中，我们介绍了多义实例嵌入网络（PIE-Nets），它通过多头自注意和残差学习将全局上下文与本地引导特征相结合来计算实例的多种多样的表示。为了学习视觉语义嵌入，我们将两个PIE-Net捆绑在一起，并在多实例学习框架中对其进行联合优化。现有的跨模态检索研究主要集中在图像-文本数据对上。在这里，我们还解决了一个更具挑战性的视频文本检索案例。为了促进视频文本检索的进一步研究，我们发布了一个新的数据集，其中包含从社交媒体收集的50 K视频句子对，称为MRW（我的反应）。我们使用MS-COCO，TGIF和我们新的MRW数据集在图像-文本和视频-文本检索场景上展示了我们的方法。
Visual-semantic embedding aims to find a shared latent space where related visual and textual instances are close to each other. Most current methods learn injective embedding functions that map an instance to a single point in the shared space. Unfortunately, injective embedding cannot effectively handle polysemous instances with multiple possible meanings; at best, it would find an average representation of different meanings. This hinders its use in real-world scenarios where individual instances and their cross-modal associations are often ambiguous. In this work, we introduce Polysemous Instance Embedding Networks (PIE-Nets) that compute multiple and diverse representations of an instance by combining global context with locally-guided features via multi-head self-attention and residual learning. To learn visual-semantic embedding, we tie-up two PIE-Nets and optimize them jointly in the multiple instance learning framework. Most existing work on cross-modal retrieval focus on image-text pairs of data. Here, we also tackle a more challenging case of video-text retrieval. To facilitate further research in video-text retrieval, we release a new dataset of 50K video-sentence pairs collected from social media, dubbed MRW (my reaction when). We demonstrate our approach on both image-text and video-text retrieval scenarios using MS-COCO, TGIF, and our new MRW dataset.