ANR Multimodal analysis and knowledge inference for musical orchestration (MAKIMOno)

这个项目科学地解决了音乐最复杂的方面之一：使用音色通过各种管弦乐模式塑造音乐。在与加拿大公司OrchPlayMusic Inc.的密切互动中，这一首个此类项目将导致人类与数字媒体互动的信息技术的创建，这将从根本上改变管弦乐教学方法，为音乐内容的计算机辅助交互创作提供更好的工具，并导致对管弦乐实践背后的感知原则的更好理解。音色是一组复杂的音色，用来区分不同乐器发出的声音或它们的混合组合。管弦乐是一门创作音乐的艺术，它将不同的乐器结合在一起，以实现各种音响目标。管弦乐教育学只关注于描述作曲家如何为乐器配乐，而不是理解他们为什么做出这样的选择。我们将用计算机科学、数字信号处理和实验心理学的方法科学地解决这一问题。借鉴共同申请者之前的研究结果，该项目将创建高效的新颖分析、学习和交互技术，揭示编排的潜在理论基础。这些技术包括音乐信号的多变量时间序列分析和通过深度表征学习进行知识推理，这些知识推理将自动破译多模式音乐表征的结构(音乐符号、声学特性、感知结果)，以便为理解管弦乐原理提供最佳描述符。该项目将依靠坚实的感知原则和经验性的管弦乐实例，从现有的大型多模式数据库中建立起具有科学基础的音乐管弦乐理论，并在研究过程中进行扩展。在与法国ANR的强有力的国际合作中解决这些复杂问题还将导致广泛适用于机器学习和计算感知领域的通用多模式学习和分析技术，这将使加拿大处于跨学科创新的前沿。

This project addresses scientifically one of the most complex aspects of music: the use of timbre to shape music through various modes of orchestration. In close interaction with a Canadian company, OrchPlayMusic Inc., this first-of-its-kind project will lead to the creation of information technologies for human interaction with digital media that will radically change orchestration pedagogy, provide better tools for the computer-aided interactive creation of musical content, and lead to a better understanding of perceptual principles underlying orchestration practice. Timbre is the complex set of tone colours that distinguish sounds emanating from different instruments or their blended combinations. Orchestration is the art of writing music that combines different instruments to achieve various sonic goals. Orchestration pedagogy focuses solely on describing how composers score instruments rather than understanding why they made such choices. We will address the why scientifically with the methods of computer science, digital signal processing, and experimental psychology. Drawing from the results of the co-applicants' previous research, this project will create efficient novel analysis, learning, and interaction techniques that reveal the underlying theoretical bases for orchestration. These techniques include multivariate time series analysis of musical signals and knowledge inference through deep representational learning that will decipher automatically the structure of multimodal musical representations (music symbols, acoustic properties, perceptual results) in order to provide optimal descriptors for the understanding of orchestration principles. The project will rely on both solid perceptual principles and empirically characterized orchestration examples to build a scientifically grounded theory of musical orchestration from a large existing multimodal database, to be extended within the course of the research. Solving these complex issues in a strong international collaboration with the French ANR will also lead to generic multimodal learning and analysis techniques broadly applicable to the fields of machine learning and computational perception that will place Canada at the forefront of interdisciplinary innovation.**************