Differentiable Alignment Techniques for Music Information Retrieval

音乐信息检索（MIR）的研究领域旨在开发计算工具，使用户能够查找、组织、分析不同形式和方面的音乐，并与之互动。从多媒体的角度来看，音乐是一个具有挑战性的领域，因为许多时间依赖的音乐概念，如旋律、和声、音高和乐器活动、响度、节奏和歌词。给定数据驱动的深度学习方法来捕获这些概念，通常需要细粒度的目标注释来反映底层音乐记录的本地属性。然而，这种强对齐（帧级）注释通常很难获得或生成。近年来，通过开发可用于深度学习管道损失函数的可微对齐技术，一般时间序列分析取得了重大进展。由于对齐过程可以成为可微模型的一部分，因此这种技术使得训练基于弱对齐目标注释的神经网络成为可能，其中只需要知道全局对应关系。在这个项目中，我们的主要目标是在具有挑战性的音乐分析和检索应用的背景下适应、探索和开发可微分对齐技术。基于最近提出的动态时间翘曲的可微版本，我们将从理论和实践的角度系统地研究效率和近似性质。此外，我们将研究时间约束的作用，以更好地处理混杂因素，提高模型和学习表征的可解释性。从MIR的角度来看，我们希望通过利用弱注释的训练数据在分析音乐信号方面取得实质性进展。为此，我们将考虑具体的MIR任务，其中包括许多尚未解决的问题，包括多音高估计，跨版本音乐检索以及音乐模式（如主题和主题）的乐谱-音频匹配。总之，在各种MIR任务取得实质性进展的同时，我们希望更好地理解和推进使用音乐作为具有挑战性的多媒体领域的现代校准技术的研究。

The research field known as Music Information Retrieval (MIR) aims to develop computational tools that allow users to find, organize, analyze, and interact with music in all its different forms and facets. From a multimedia perspective, music is a challenging domain due to the many time-dependent musical concepts such as melody, harmony, pitch and instrumentation activity, loudness, rhythm, and lyrics. Given data-driven deep learning approaches to capture these concepts, one typically requires fine-grained target annotations that reflect the local properties of the underlying music recordings. However, such strongly aligned (frame-level) annotations are generally difficult to obtain or generate. Recent years have seen major advances in general time series analysis by developing differentiable alignment techniques that can be used in loss functions for deep learning pipelines. Since the alignment process can then be part of the differentiable model, such techniques make it possible to train a neural network based on weakly aligned target annotations where only global correspondences need to be known. In this project, our primary goal is to adapt, explore, and develop differentiable alignment techniques in the context of challenging music analysis and retrieval applications. Building upon recently proposed differentiable versions of dynamic time warping, we will systematically study efficiency and approximation properties from a theoretical and practical perspective. Furthermore, we will investigate the role of temporal constraints to better handle confounding factors and improve the explainability of models and learned representations. From an MIR perspective, we want to achieve substantial advances in analyzing music signals by exploiting weakly annotated training data. To this end, we will consider concrete MIR tasks with many yet unsolved problems, including multi-pitch estimation, cross-version music retrieval, and score-audio matching of musical patterns such as themes and leitmotifs. In summary, while making substantial progress for various MIR tasks, we want to gain a better understanding and advance research of modern alignment techniques using music as a challenging multimedia domain.