Source Separation and Restoration of Sound Components in Music Recordings

该项目旨在开发分离和恢复复杂音乐录音中发生的声音事件的技术。在第一阶段（最初的建议），我们专注于打击声源，在那里我们将鼓录音分解为单个鼓声事件。使用非负矩阵因子反卷积（NMFD）作为我们的中心方法，我们研究了如何生成和整合基于音频和分数的侧信息来指导分解。我们在具体的应用场景中测试了我们的方法，包括音频混音（重鼓）和爵士音乐的摇摆比分析。在项目的第二阶段，我们的目标将大大扩展。首先，我们希望通过考虑其他具有挑战性的音乐场景来超越鼓场景，包括钢琴曲（例如贝多芬奏鸣曲，肖邦玛祖卡），钢琴歌曲（例如舒伯特的克拉维列德）和弦乐（例如贝多芬弦乐四重奏）。在这些场景中，我们的目标是将音乐记录分解为与音符相关的单个声音事件。作为我们的核心方法，我们计划开发一个统一的音频分解框架，将经典的信号处理和机器学习与最近的深度学习（DL）方法相结合。此外，我们希望采用生成式深度学习技术来提高恢复声音事件的感知质量。作为总体目标，我们将研究如何将先验知识（如分数信息）集成到基于dl的学习中，以提高训练模型的可解释性。

This project aims at the development of techniques for separating and restoring sound events as occurring in complex music recordings. In the first phase (initial proposal), we focused on percussive sound sources, where we decomposed a drum recording into individual drum sound events. Using Non-Negative Matrix Factor Deconvolution (NMFD) as our central methodology, we studied how to generate and integrate audio- and score-based side information to guide the decomposition. We tested our approaches within concrete application scenarios, including audio remixing (redrumming) and swing ratio analysis of jazz music. In the second phase of the project, our goals will be significantly extended. First, we want to go beyond the drum scenario by considering other challenging music scenarios, including piano music (e.g., Beethoven Sonatas, Chopin Mazurkas), piano songs (e.g., Klavierlieder by Schubert), and string music (e.g., Beethoven String Quartets). In these scenarios, our goal is to decompose a music recording into individual note-related sound events. As our central methodology, we plan to develop a unifying audio decomposition framework that combines classical signal processing and machine learning with recent deep learning (DL) approaches. Furthermore, we want to adopt generative DL techniques for improving the perceptual quality of restored sound events. As a general goal, we will investigate how prior knowledge, such as score information can be integrated into DL-based learning to improve the interpretability of the trained models.