Computational Analysis of Music Audio Recordings: A Cross-Version Approach

音乐录音的计算分析是一个高度跨学科的研究领域，涉及音乐学和音乐理论的领域知识以及信号处理和机器学习的方法。从计算机科学的角度来看，音乐音频数据的多样性和复杂性带来了巨大的挑战，这是特定于该领域的。首先，音乐是多方面的，具有不同的语义维度，如时间，音高，音色或风格，需要分解以获得可解释的表征。第二，音乐分析包括分层相关的任务，如估计音高，和弦，局部键，和全局键或检测的起始，节拍，强拍，和结构边界，建议使用多任务的方法。第三，音乐数据是复杂的，由高度相关的源组成，其组成部分在时间和频率上重叠。此外，音乐概念往往是模糊和主观的，因此需要解释的方法和多个注释。第四，音乐场景通常缺乏数据，缺乏大量注释数据的可用性。因此，分析方法经常过度拟合训练数据集中的隐式偏差，对小扰动变得敏感，并且不能很好地推广到看不见的数据。数据稀缺对深度学习方法提出了特殊的挑战，这些方法目前在该领域占据主导地位。这些方法在许多音乐分析任务中取得了实质性的改进，但往往会遇到一种“玻璃天花板”，在此之上很难实现和衡量进一步的进展。为了克服这个问题，这个项目采用了一种跨版本的方法，利用古典音乐的数据集，其中包含几种形式（乐谱和音频），几种表演（解释和安排），以及每个音乐作品的几个注释（多个专家）。这样的数据集允许在版本之间传输注释，并通过测试沿着不同维度的泛化来系统地评估深度学习方法的鲁棒性。例如，在一个实施例中，作品的其他版本、作曲家的其他作品或历史时期的其他作曲家。作为一个主要的概念性贡献，我们应用并进一步发展这种跨版本策略，利用它们来更好地理解分析方法，并使用合适的训练和融合策略来改进这些方法。基于这种跨版本的方法，我们解决了音乐数据的具体挑战，旨在分析方法，特别是用于计算音乐学，并朝着新的方法策略在更广泛的数字人文领域的进展。

The computational analysis of music audio recordings constitutes a highly interdisciplinary research area, involving domain knowledge from musicology and music theory as well as methods from signal processing and machine learning. From a computer science perspective, the variety and complexity of music audio data poses enormous challenges, which are specific to this domain. First, music is multi-faceted, being characterized by different semantic dimensions such as time, pitch, timbre, or style, which need to be disengtangled to obtain interpretable representations. Second, music analysis comprises hierarchically related tasks such as the estimation of pitches, chords, local keys, and global keys or the detection of onsets, beats, downbeats, and structural boundaries, suggesting the use of multi-task approaches. Third, music data is complex, consisting of highly correlated sources whose components overlap in time and frequency. Furthermore, musical notions are often ambiguous and subjective, thus demanding for interpretable methods and multiple annotators. Fourth, music scenarios are often data-scarce, lacking the availability of large amounts of annotated data. As a consequence, analysis methods frequently overfit to implicit biases in the training datasets, become sensitive to small perturbations, and do not generalize well to unseen data. The data scarcity poses particular challenges for deep-learning approaches, which are nowadays dominating the field. These approaches achieved substantial improvements for many music analysis tasks but often hit a kind of "glass ceiling" above which further progress is hard to achieve and to measure. To overcome this problem, this project adopts a cross-version approach by exploiting datasets of classical music, which contain several modalities (score and audio), several performances (interpretations and arrangements), and several annotations (multiple experts) for each musical work. Such datasets allow for transferring annotations between versions and for systematically evaluating the robustness of deep-learning methods by testing generalization along different dimensions, e. g., to other versions of a work, other works by a composer, or other composers from a historical period. As a main conceptual contribution, we apply and further develop such cross-version strategies, exploiting them to better understand the analysis methods and to improve these methods using suitable training and fusion strategies. Based on this cross-version approach, we address the specific challenges of music data, aiming for analysis methods that are of particular use for computational musicology, and progressing towards novel methodological strategies in the wider field of the digital humanities.