Next Generation Virtual Musical Instruments: Physics-informed Neural Networks for Sound Synthesis and Digital Audio Effects

物理信息神经网络是计算物理学中新兴的一类算法，它将传统的基于物理的仿真方法与现代机器学习方法相结合。该项目旨在开发物理信息神经网络，用于声音合成和音频效果建模的独特目的-通过数字信号处理模拟乐器和设备的声音特性。其动机是创造下一代数字乐器，这些乐器更逼真，因此对音乐家更有启发性。具体来说，重点将放在声乐合成器，吉他放大器模拟和虚拟房间环境。内容：自20世纪60年代John Chowning首创频率调制算法以来，数字声音合成一直是一个活跃的研究领域，后来由雅马哈将其商业化，开发出第一台数字合成器。数字乐器和音频效果在当今的音乐中无处不在，但在其设计中仍然存在一个关键挑战：创造有机，逼真的音色-音乐家通常希望的品质。因此，工程师和研究人员使用两种不同的方法-基于物理的建模（白盒建模）和机器学习（黑盒建模）-来数字仿真音乐系统的声音，包括声学乐器，表演空间，模拟电路（例如，吉他放大器）甚至人声。基于物理的建模包括分析系统以推导控制微分方程，然后通过数值模拟求解这些方程以产生音频。这通常是劳动密集型的，并且数值模拟方法具有固有的不准确性、计算成本和稳定性问题。相反，机器学习方法假设没有参考系统的人类知识，而是“训练”通用算法（通常是一类人工神经网络）来复制给定的输入-输出映射。神经网络的黑盒性质使其非常灵活和强大;然而，缺点包括泛化错误，对大数据集的依赖和高计算要求。在这个研究项目中，目标是将现有的两种方法联合收割机结合成混合“灰箱”方法，以获得更好的整体性能，在效率，准确性和通用性方面。这将通过物理信息神经网络（PINN）来完成。PINN是计算物理学中一个快速发展的研究领域，它是使用控制物理方程作为训练数据的机器学习模型，并且已经被证明比纯粹的数据驱动方法表现更好。该研究项目将为声音合成的特定目的开发PINN：创建端到端神经网络，以准确，高效地模拟各种声学和电气系统。方法学：PINN将为数字音频的三个子主题开发：虚拟模拟建模室内声学建模声乐合成在每个领域，将进行一个或多个案例研究，一般方法是：确定系统的已知物理和控制方程。收集培训/测试数据，例如，录音开发纯机器学习模型。整合物理约束并获得结果。调查、设计和优化模型架构。根据以下指标评估和比较模型：数值精度-时域和频域。感知准确性-通过一组志愿者的盲听测试。计算需求（CPU和内存）将最佳模型开发成原型仪器。

Physics-informed neural networks are an emerging class of algorithm in computational physics which combine traditional physics-based simulation methods with modern machine learning approaches. This project aims to develop physics-informed neural networks for the unique purpose of sound synthesis and audio effects modelling - emulating the sonic characteristics of musical instruments and equipment through digital signal processing. The motivation is to create next-generation digital instruments which are more realistic and therefore more inspiring to musicians. Specifically, the focus will be on vocal synthesizers, guitar amplifier simulations and virtual room environments. Context: Digital sound synthesis has been an active area of research since the 1960s when John Chowning pioneered a frequency modulation algorithm, later commercialised by Yamaha to develop the first digital synthesiser. Digital instruments and audio effects are now ubiquitous with music today, however a key challenge remains in their design: creating organic, realistic timbres - generally desirable qualities to musicians. Engineers and researchers therefore use two distinct approaches - physics-based modelling (white-box modelling) and machine learning (black-box modelling) - to digitally emulate the sound of musical systems including acoustic instruments, performance spaces, analogue circuitry (e.g., guitar amplifiers) or even human voices. Physics-based modelling involves analysis of the system to derive the governing differential equations, then solving these via numerical simulation to produce audio. This is often labour intensive and numerical simulation methods have inherent inaccuracies, computational costs, and stability issues. On the contrary, a machine learning approach assumes no human knowledge of the reference system and instead a general algorithm (normally a class of artificial neural network) is 'trained' to replicate a given input-output mapping. The black-box nature of neural networks makes them extremely flexible and powerful; however, disadvantages include generalization error, the reliance on large data sets and high computational requirements. In this research project, the objective is to combine the two existing approaches into hybrid 'grey-box' methods to obtain better overall performance in terms of efficiency, accuracy, and generality. This will be done through physics-informed neural networks (PINNs). A rapidly growing area of research in computational physics, PINNs are machine learning models which use the governing physical equations as training data and have been shown to perform better than purely data-driven methods. This research project will develop PINNs for the specific purpose of sound synthesis: creating end-to-end neural networks for the accurate, efficient simulation of various acoustic and electric systems. Methodology: PINNs will be developed for three sub-topics in digital audio: Virtual analogue modelling Room acoustics modelling Vocal synthesis In each area, one or more case studies will be undertaken, with the general methodology being: Determine the known physics of the system and governing equations. Gather training/testing data e.g., audio recordings. Develop pure machine learning models. Integrate physical constraints and obtain results. Investigate, design, and optimise model architectures. Evaluate and compare the models based on the following metrics: Numerical accuracy - in time and frequency domains. Perceptual accuracy - through blind listening tests with a group of volunteers. Computational demand (CPU and memory) Develop the best models into prototype instruments.