Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-Modal Retrieval
Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-Modal Retrieval
复制标题
DOI：
10.1145/3387164
复制
发表时间：
2019-08
期刊：
影响因子：
--
通讯作者：
Donghuo Zeng;Yi Yu;K. Oyama
中科院分区：
文献类型：
--
作者：
Donghuo Zeng;Yi Yu;K. Oyama
跨模态检索是指通过一个模态的查询来检索另一个模态的数据，是多媒体、信息检索、计算机视觉和数据库等领域的一个研究热点。现有的研究主要集中在文本-图像、文本-视频和歌词-音频之间的跨模态检索。由于音视频配对数据集和语义信息有限，很少有研究涉及音视频之间的跨模态检索。视听跨模态检索任务的主要挑战集中在从共享子空间学习联合嵌入以计算不同模态之间的相似性，其中生成新的表示是最大化视听模态空间之间的相关性。在这项工作中，我们提出了TNN-C-CCA，这是一种具有聚类典型相关分析的新型深度三元组神经网络，它是一种具有音频分支和视频分支的端到端监督学习架构。我们不仅考虑了公共空间中的匹配对，而且在最大化相关性时还计算了失配对。特别是，作出了两项重大贡献。首先，可以通过构建具有用于最佳投影的三元组损失的深度三元组神经网络来生成更好的表示，以最大化共享子空间中的相关性。其次，在学习阶段使用正例和反例，以提高音频和视频之间的嵌入学习能力。我们的实验是运行在五倍交叉验证，其中平均性能被应用到演示性能的音频-视频跨模态检索。在两个不同的视听数据集上的实验结果表明，该方法的性能优于现有的六种基于典型相关分析的方法和四种基于最新技术的跨模态检索方法。
Cross-modal retrieval aims to retrieve data in one modality by a query in another modality, which has been a very interesting research issue in the field of multimedia, information retrieval, and computer vision, and database. Most existing works focus on cross-modal retrieval between text-image, text-video, and lyrics-audio. Little research addresses cross-modal retrieval between audio and video due to limited audio-video paired datasets and semantic information. The main challenge of the audio-visual cross-modal retrieval task focuses on learning joint embeddings from a shared subspace for computing the similarity across different modalities, where generating new representations is to maximize the correlation between audio and visual modalities space. In this work, we propose TNN-C-CCA, a novel deep triplet neural network with cluster canonical correlation analysis, which is an end-to-end supervised learning architecture with an audio branch and a video branch. We not only consider the matching pairs in the common space but also compute the mismatching pairs when maximizing the correlation. In particular, two significant contributions are made. First, a better representation by constructing a deep triplet neural network with triplet loss for optimal projections can be generated to maximize correlation in the shared subspace. Second, positive examples and negative examples are used in the learning stage to improve the capability of embedding learning between audio and video. Our experiment is run over fivefold cross validation, where average performance is applied to demonstrate the performance of audio-video cross-modal retrieval. The experimental results achieved on two different audio-visual datasets show that the proposed learning architecture with two branches outperforms existing six canonical correlation analysis–based methods and four state-of-the-art-based cross-modal retrieval methods.