Differentiable Alignment Techniques for Music Information Retrieval

被称为音乐信息检索(MIR)的研究领域旨在开发计算工具，允许用户查找、组织、分析音乐的所有不同形式和方面并与其交互。从多媒体的角度来看，音乐是一个具有挑战性的领域，因为许多与时间相关的音乐概念，如旋律、和声、音调和乐器活动、响度、节奏和歌词。由于采用数据驱动的深度学习方法来捕获这些概念，因此通常需要细粒度的目标注释来反映底层音乐录音的本地属性。然而，这种强对齐(框架级)注释通常很难获得或生成。近年来，通过开发可用于深度学习管道的损失函数的可微对齐技术，在一般时间序列分析方面取得了重大进展。由于对齐过程然后可以是可微模型的一部分，这样的技术使得能够基于弱对齐的目标注释来训练神经网络，其中仅需要知道全局对应关系。在这个项目中，我们的主要目标是在具有挑战性的音乐分析和检索应用程序的背景下适应、探索和开发可区分的比对技术。在最近提出的动态时间规整的可微版本的基础上，我们将从理论和实践的角度系统地研究效率和逼近性质。此外，我们将研究时间约束的作用，以更好地处理混杂因素，并提高模型和学习表示的可解释性。从MIR的角度来看，我们希望通过利用弱注释的训练数据在分析音乐信号方面取得实质性的进展。为此，我们将考虑具有许多尚未解决的问题的具体MIR任务，包括多音调估计、跨版本音乐检索以及音乐模式(如主题和主旋律)的配乐-音频匹配。总而言之，在为各种MIR任务取得实质性进展的同时，我们希望更好地理解和推进使用音乐作为一个具有挑战性的多媒体领域的现代比对技术的研究。

The research field known as Music Information Retrieval (MIR) aims to develop computational tools that allow users to find, organize, analyze, and interact with music in all its different forms and facets. From a multimedia perspective, music is a challenging domain due to the many time-dependent musical concepts such as melody, harmony, pitch and instrumentation activity, loudness, rhythm, and lyrics. Given data-driven deep learning approaches to capture these concepts, one typically requires fine-grained target annotations that reflect the local properties of the underlying music recordings. However, such strongly aligned (frame-level) annotations are generally difficult to obtain or generate. Recent years have seen major advances in general time series analysis by developing differentiable alignment techniques that can be used in loss functions for deep learning pipelines. Since the alignment process can then be part of the differentiable model, such techniques make it possible to train a neural network based on weakly aligned target annotations where only global correspondences need to be known. In this project, our primary goal is to adapt, explore, and develop differentiable alignment techniques in the context of challenging music analysis and retrieval applications. Building upon recently proposed differentiable versions of dynamic time warping, we will systematically study efficiency and approximation properties from a theoretical and practical perspective. Furthermore, we will investigate the role of temporal constraints to better handle confounding factors and improve the explainability of models and learned representations. From an MIR perspective, we want to achieve substantial advances in analyzing music signals by exploiting weakly annotated training data. To this end, we will consider concrete MIR tasks with many yet unsolved problems, including multi-pitch estimation, cross-version music retrieval, and score-audio matching of musical patterns such as themes and leitmotifs. In summary, while making substantial progress for various MIR tasks, we want to gain a better understanding and advance research of modern alignment techniques using music as a challenging multimedia domain.