Content-Based Music-Image Retrieval Using Self- and Cross-Modal Feature Embedding Memory
Content-Based Music-Image Retrieval Using Self- and Cross-Modal Feature Embedding Memory
复制标题
DOI：
10.1109/wacv56688.2023.00221
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Takayuki Nakatsuka;Masahiro Hamasaki;Masataka Goto
中科院分区：
文献类型：
--
作者：
Takayuki Nakatsuka;Masahiro Hamasaki;Masataka Goto
描述了一种基于深度度量学习的音乐及其代表性图像(音乐音频信号及其封面艺术图像)的基于内容的跨模式检索方法。我们训练音乐和图像编码者，使正的音乐图像对的嵌入彼此靠近，而随机对的音乐图像对的嵌入在共享的嵌入空间中彼此远离。此外，我们还提出了一种称为自模式和跨模式特征嵌入记忆的机制，该机制将先前任何迭代的音乐和图像嵌入存储在记忆中，并使编码者能够挖掘信息对进行训练。为了进行这样的训练，我们构建了一个包含78,325对音乐图像的数据集。我们在此数据集上展示了所提出的机制的有效性：具体而言，我们的机制在平均倒数排名上比基线方法高出×1.93∼3.38，在Recall@50上高出×2.19∼3.56，在中位排名上高出528∼891。
This paper describes a method based on deep metric learning for content-based cross-modal retrieval of a piece of music and its representative image (i.e., a music audio signal and its cover art image). We train music and image encoders so that the embeddings of a positive music-image pair lie close to each other, while those of a random pair lie far from each other, in a shared embedding space. Furthermore, we propose a mechanism called self- and cross-modal feature embedding memory, which stores both the music and image embeddings of any previous iterations in memory and enables the encoders to mine informative pairs for training. To perform such training, we constructed a dataset containing 78,325 music-image pairs. We demonstrate the effectiveness of the proposed mechanism on this dataset: specifically, our mechanism outperforms baseline methods by ×1.93 ∼ 3.38 for the mean reciprocal rank, ×2.19 ∼ 3.56 for recall@50, and 528 ∼ 891 ranks for the median rank.