ANR Multimodal analysis and knowledge inference for musical orchestration (MAKIMOno)

该项目科学地解决了音乐最复杂的方面之一：通过各种配器模式使用音色来塑造音乐。在与加拿大公司OrchPlayMusic Inc.的密切互动中，这是一个首创的项目，它将导致人类与数字媒体互动的信息技术的创造，这将从根本上改变配器教学法，为计算机辅助的音乐内容互动创作提供更好的工具，并导致更好地理解配器实践的感知原则。音色是一组复杂的音色，可以区分不同乐器或它们的混合组合发出的声音。伴奏是一种结合不同乐器以达到不同声音目标的音乐创作艺术。演奏教学法只关注于描述作曲家如何为乐器配乐，而不是理解他们为什么做出这样的选择。我们将用计算机科学、数字信号处理和实验心理学的方法来科学地解决这个问题。从共同申请人以前的研究成果中汲取，该项目将创建有效的新颖分析，学习和交互技术，揭示编排的潜在理论基础。这些技术包括音乐信号的多变量时间序列分析和通过深度表征学习进行的知识推理，该深度表征学习将自动破译多模态音乐表征（音乐符号，声学特性，感知结果）的结构，以便为理解配器原理提供最佳描述符。该项目将依靠坚实的感知原则和经验特征的配器例子，从现有的大型多模态数据库中建立一个科学的音乐配器理论，并在研究过程中进行扩展。通过与法国ANR的强有力的国际合作来解决这些复杂的问题，也将导致通用的多模态学习和分析技术广泛适用于机器学习和计算感知领域，这将使加拿大处于跨学科创新的前沿。

This project addresses scientifically one of the most complex aspects of music: the use of timbre to shape music through various modes of orchestration. In close interaction with a Canadian company, OrchPlayMusic Inc., this first-of-its-kind project will lead to the creation of information technologies for human interaction with digital media that will radically change orchestration pedagogy, provide better tools for the computer-aided interactive creation of musical content, and lead to a better understanding of perceptual principles underlying orchestration practice. Timbre is the complex set of tone colours that distinguish sounds emanating from different instruments or their blended combinations. Orchestration is the art of writing music that combines different instruments to achieve various sonic goals. Orchestration pedagogy focuses solely on describing how composers score instruments rather than understanding why they made such choices. We will address the why scientifically with the methods of computer science, digital signal processing, and experimental psychology. Drawing from the results of the co-applicants' previous research, this project will create efficient novel analysis, learning, and interaction techniques that reveal the underlying theoretical bases for orchestration. These techniques include multivariate time series analysis of musical signals and knowledge inference through deep representational learning that will decipher automatically the structure of multimodal musical representations (music symbols, acoustic properties, perceptual results) in order to provide optimal descriptors for the understanding of orchestration principles. The project will rely on both solid perceptual principles and empirically characterized orchestration examples to build a scientifically grounded theory of musical orchestration from a large existing multimodal database, to be extended within the course of the research. Solving these complex issues in a strong international collaboration with the French ANR will also lead to generic multimodal learning and analysis techniques broadly applicable to the fields of machine learning and computational perception that will place Canada at the forefront of interdisciplinary innovation.