Wav2CLIP: Learning Robust Audio Representations from Clip
Wav2CLIP: Learning Robust Audio Representations from Clip
复制标题
DOI：
10.1109/icassp43922.2022.9747669
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Ho-Hsiang Wu;Prem Seetharaman;Kundan Kumar;J. Bello
中科院分区：
文献类型：
--
作者：
Ho-Hsiang Wu;Prem Seetharaman;Kundan Kumar;J. Bello
我们提出了Wav2CLIP，这是一种从对比图像预训练（CLIP）中提取的鲁棒音频表示学习方法。我们系统地评估了Wav2CLIP在各种音频任务上的表现，包括分类、检索和生成，并表明Wav2CLIP可以胜过几种公开的预训练音频表示算法。Wav2CLIP将音频投影到具有图像和文本的共享嵌入空间中，这使得多模态应用能够实现，例如零拍摄分类和跨模态检索。此外，与完全监督模型相比，Wav2CLIP只需要10%的数据就可以在下游任务上实现有竞争力的性能，并且比竞争方法更有效地进行预训练，因为它不需要学习视觉模型与听觉模型。最后，我们演示了从Wav2CLIP生成图像作为共享嵌入空间的定性评估。我们的代码和模型权重是开源的，可用于进一步的应用。
We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a variety of audio tasks including classification, retrieval, and generation, and show that Wav2CLIP can outperform several publicly available pre-trained audio representation algorithms. Wav2CLIP projects audio into a shared embedding space with images and text, which enables multimodal applications such as zero-shot classification, and cross-modal retrieval. Furthermore, Wav2CLIP needs just ∼10% of the data to achieve competitive performance on downstream tasks compared with fully supervised models, and is more efficient to pre-train than competing methods as it does not require learning a visual model in concert with an auditory model. Finally, we demonstrate image generation from Wav2CLIP as qualitative assessment of the shared embedding space. Our code and model weights are open sourced and made available for further applications.