New machine learning methods for music: learning at multiple timescales

在音乐机器学习算法的设计方面已经取得了巨大的进步。 然而，大多数方法忽略了全局时间结构，将音频文件切成5秒或更短的片段，并单独处理每个片段。 由于忽略了长时间尺度的结构，这些算法无法学习重要的音乐意义的高级指标，如旋律和和弦进行。 这一差距定义了本研究提案中解决的核心挑战：我们如何在多个时间尺度上处理音乐，以便并行地从音频中学习重要的局部和全局结构？ 该研究计划的目标是开发新的机器学习算法，用于音乐推荐，人类音乐表演和视频游戏音乐的自适应上下文感知生成。我们的目标是设计分层多时标算法，将证据在许多层面。我们认为，对于试图学习音乐如何在时间上组织的机器学习模型来说，音乐节拍是一个特别有用的先验知识。可以以类似人类的方式跟踪节拍的模型知道音乐信号中何时会发生重要事件，从而避免对音乐中所有可能的时间滞后之间的相关性进行指数长的搜索。 音乐相关的研究解决了几个困难和相关的任务，音乐信息检索和音乐表演。具体的任务包括预测两个音频序列之间的相似性，预测音频的艺术家，流派或风格，以人类现实的方式表演音乐，生成风格上合适并且适合特定上下文（例如，可怕或平静）的音乐（例如，用于视频游戏）。除了有助于行业相关的应用，这里进行的基础研究将推进时间序列数据的机器学习算法的最新发展，并将进一步了解音乐家如何制作音乐并被听众感知。

Great advances have been made in the design of machine learning algorithms for music. However, most approaches ignore global temporal structure, chopping an audio file into segments of five seconds or less and treating each segment individually. By ignoring long-timescale structure, these algorithms are unable to learn important high-level indicators of musical meaning such as melody and chord progressions. This gap defines the core challenge addressed in this research proposal: how can we approach music at multiple timescales so as to in parallel learn important local and global structure from audio? The objective of this research program is to develop new machine learning algorithms for music recommendation, human-like music performance and adaptive context-aware generation of music for video games. Our goal is to design hierarchical multi-timescale algorithms that incorporate evidence at many levels. We believe that musical meter is a particularly informative prior for a machine learning model trying to learn how music is organized in time. A model that can track meter in a human-like manner knows when important events will occur in the music signal, thus avoiding an exponentially-long search for correlation among all possible time lags in the music. The music related research addresses several difficult and relevant tasks related to music information retrieval and to music performance. Specific tasks include predicting the similarity between two audio sequences, predicting the artist, genre or style of audio, performing music in a human-realistic fashion, generating music (e.g. for video games) that is stylistically appropriate and that fits a particular context (e.g. scary or calm). In addition to contributing industry-relevant applications, the fundamental research carried out here will advance the state-of-the-art in machine learning algorithms for time series data and will further our understanding of how music is produced by musicians and perceived by listeners.