Next Generation Virtual Musical Instruments: Physics-informed Neural Networks for Sound Synthesis and Digital Audio Effects

下一代虚拟乐器：用于声音合成和数字音频效果的物理信息神经网络

• 批准号: 2710512
• 金额: --
• 依托单位: University of Edinburgh
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2022
• 资助国家: 英国
• 起止时间: 2022 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2710512
• 关键词: Next; Generation; Virtual; Musical; Instruments

Physics-informed neural networks are an emerging class of algorithm in computational physics which combine traditional physics-based simulation methods with modern machine learning approaches. This project aims to develop physics-informed neural networks for the unique purpose of sound synthesis and audio effects modelling - emulating the sonic characteristics of musical instruments and equipment through digital signal processing. The motivation is to create next-generation digital instruments which are more realistic and therefore more inspiring to musicians. Specifically, the focus will be on vocal synthesizers, guitar amplifier simulations and virtual room environments. Context: Digital sound synthesis has been an active area of research since the 1960s when John Chowning pioneered a frequency modulation algorithm, later commercialised by Yamaha to develop the first digital synthesiser. Digital instruments and audio effects are now ubiquitous with music today, however a key challenge remains in their design: creating organic, realistic timbres - generally desirable qualities to musicians. Engineers and researchers therefore use two distinct approaches - physics-based modelling (white-box modelling) and machine learning (black-box modelling) - to digitally emulate the sound of musical systems including acoustic instruments, performance spaces, analogue circuitry (e.g., guitar amplifiers) or even human voices. Physics-based modelling involves analysis of the system to derive the governing differential equations, then solving these via numerical simulation to produce audio. This is often labour intensive and numerical simulation methods have inherent inaccuracies, computational costs, and stability issues. On the contrary, a machine learning approach assumes no human knowledge of the reference system and instead a general algorithm (normally a class of artificial neural network) is 'trained' to replicate a given input-output mapping. The black-box nature of neural networks makes them extremely flexible and powerful; however, disadvantages include generalization error, the reliance on large data sets and high computational requirements. In this research project, the objective is to combine the two existing approaches into hybrid 'grey-box' methods to obtain better overall performance in terms of efficiency, accuracy, and generality. This will be done through physics-informed neural networks (PINNs). A rapidly growing area of research in computational physics, PINNs are machine learning models which use the governing physical equations as training data and have been shown to perform better than purely data-driven methods. This research project will develop PINNs for the specific purpose of sound synthesis: creating end-to-end neural networks for the accurate, efficient simulation of various acoustic and electric systems. Methodology: PINNs will be developed for three sub-topics in digital audio: Virtual analogue modelling Room acoustics modelling Vocal synthesis In each area, one or more case studies will be undertaken, with the general methodology being: Determine the known physics of the system and governing equations. Gather training/testing data e.g., audio recordings. Develop pure machine learning models. Integrate physical constraints and obtain results. Investigate, design, and optimise model architectures. Evaluate and compare the models based on the following metrics: Numerical accuracy - in time and frequency domains. Perceptual accuracy - through blind listening tests with a group of volunteers. Computational demand (CPU and memory) Develop the best models into prototype instruments.

物理知识的神经网络是计算物理学中新兴算法的类别，它们将基于物理的模拟方法与现代机器学习方法结合在一起。该项目旨在开发具有物理信息的神经网络，以实现声音综合和音频效应建模的独特目的 - 通过数字信号处理模拟乐器和设备的声音特征。动机是创建下一代数字乐器，这更现实，因此对音乐家更具启发性。具体而言，重点将放在人声合成器，吉他放大器模拟和虚拟房间环境上。上下文：自1960年代以来，数字声音综合一直是研究的积极研究领域，当时约翰·乔宁（John Chowning）率先提出了一种频率调制算法，后来由雅马哈（Yamaha）商业化以开发第一个数字合成器。如今，数字乐器和音频效果对音乐无处不在，但是他们的设计仍然是一个关键的挑战：创造有机，现实的音色 - 通常对音乐家的特质。因此，工程师和研究人员使用两种不同的方法 - 基于物理的建模（白盒建模）和机器学习（Black-Box建模） - 以数字方式模仿音乐系统的声音，包括声学仪器，性能空间，模拟电路（例如吉他放大器），甚至是人类的声音。基于物理学的建模涉及对系统的分析，以得出管理差分方程，然后通过数值模拟来求解这些方程以产生音频。这通常是劳动密集型和数值模拟方法具有固有的不准确性，计算成本和稳定性问题。相反，机器学习方法没有人类对参考系统的了解，而是“训练”一般算法（通常是人工神经网络）以复制给定的输入输出映射。神经网络的黑盒本质使它们极为灵活和强大。但是，缺点包括概括错误，对大数据集的依赖和高计算要求。在该研究项目中，目标是将两种现有方法结合到混合“灰色盒”方法中，以在效率，准确性和一般性方面获得更好的总体性能。这将通过物理知识的神经网络（PINN）来完成。 Pinns是计算物理学研究的快速增长领域，是机器学习模型，它们使用管理物理方程式作为训练数据，并且已被证明比纯粹的数据驱动方法更好。该研究项目将开发出PINN的特定目的：创建端到端神经网络，以进行各种声学和电气系统的准确，有效的模拟。方法论：将针对数字音频的三个子主题开发PINN：虚拟模拟建模室声学对每个领域的声音合成，将进行一个或多个案例研究，其中一般方法论是：确定系统和管理方程的已知物理学。收集培训/测试数据，例如录音。开发纯机器学习模型。整合物理约束并获得结果。调查，设计和优化模型体系结构。根据以下指标评估和比较模型：时间和频域中的数值精度。感知准确性 - 通过与一群志愿者进行盲目听力测试。计算需求（CPU和内存）将最佳模型开发为原型仪器。