New machine learning methods for music: learning at multiple timescales

在音乐机器学习算法的设计上已经取得了巨大的进步。然而，大多数方法忽略了全局时间结构，将音频文件切割成5秒或更短的片段，并单独处理每个片段。由于忽略了长时间尺度结构，这些算法无法学习重要的高层次的音乐意义指标，如旋律和和弦的进行。这一差距定义了本研究计划所要解决的核心挑战：我们如何在多个时间尺度上处理音乐，以便同时从音频中学习重要的局部和全局结构？该研究项目的目标是开发新的机器学习算法，用于音乐推荐、类人音乐表演和视频游戏音乐的自适应上下文感知生成。我们的目标是设计分层的多时间尺度算法，在许多层面上纳入证据。我们认为，对于试图学习音乐如何组织的机器学习模型来说，节拍是一个特别有用的先验。一个能够以类似人类的方式跟踪节拍的模型知道音乐信号中什么时候会发生重要事件，从而避免了在音乐中所有可能的时间滞后之间进行指数级长时间的相关性搜索。音乐相关研究解决了与音乐信息检索和音乐表演相关的几个困难和相关的任务。具体任务包括预测两个音频序列之间的相似性，预测音频的艺术家、类型或风格，以人类现实主义的方式表演音乐，生成风格合适且适合特定背景（如恐怖或平静）的音乐（如电子游戏）。除了为行业相关应用做出贡献外，这里进行的基础研究将推动时间序列数据的机器学习算法的发展，并将进一步了解音乐家如何制作音乐以及听众如何感知音乐。

Great advances have been made in the design of machine learning algorithms for music. However, most approaches ignore global temporal structure, chopping an audio file into segments of five seconds or less and treating each segment individually. By ignoring long-timescale structure, these algorithms are unable to learn important high-level indicators of musical meaning such as melody and chord progressions. This gap defines the core challenge addressed in this research proposal: how can we approach music at multiple timescales so as to in parallel learn important local and global structure from audio? The objective of this research program is to develop new machine learning algorithms for music recommendation, human-like music performance and adaptive context-aware generation of music for video games. Our goal is to design hierarchical multi-timescale algorithms that incorporate evidence at many levels. We believe that musical meter is a particularly informative prior for a machine learning model trying to learn how music is organized in time. A model that can track meter in a human-like manner knows when important events will occur in the music signal, thus avoiding an exponentially-long search for correlation among all possible time lags in the music. The music related research addresses several difficult and relevant tasks related to music information retrieval and to music performance. Specific tasks include predicting the similarity between two audio sequences, predicting the artist, genre or style of audio, performing music in a human-realistic fashion, generating music (e.g. for video games) that is stylistically appropriate and that fits a particular context (e.g. scary or calm). In addition to contributing industry-relevant applications, the fundamental research carried out here will advance the state-of-the-art in machine learning algorithms for time series data and will further our understanding of how music is produced by musicians and perceived by listeners.