Next Generation Virtual Musical Instruments: Physics-informed Neural Networks for Sound Synthesis and Digital Audio Effects

物理信息神经网络是计算物理学中一类新兴的算法，它将传统的基于物理的模拟方法与现代机器学习方法相结合。该项目旨在开发物理信息神经网络，用于声音合成和音频效果建模的独特目的-通过数字信号处理模拟乐器和设备的声音特性。其动机是创造更逼真的下一代数字乐器，从而更能激发音乐家的灵感。具体来说，重点将是声乐合成器，吉他放大器模拟和虚拟房间环境。背景：自20世纪60年代以来，数字声音合成一直是一个活跃的研究领域，当时约翰·乔宁（John Chowning）开创了一种调频算法，后来由雅马哈（Yamaha）商业化，开发了第一台数字合成器。如今，数字乐器和音频效果在音乐中无处不在，但在它们的设计中仍然存在一个关键挑战：创造有机的、逼真的音色——通常是音乐家想要的品质。因此，工程师和研究人员使用两种不同的方法——基于物理的建模（白盒建模）和机器学习（黑盒建模）——来数字模拟音乐系统的声音，包括原声乐器、表演空间、模拟电路（例如，吉他放大器）甚至是人声。基于物理的建模包括对系统进行分析，推导出控制微分方程，然后通过数值模拟来求解这些方程，从而产生音频。这通常是劳动密集型的，数值模拟方法具有固有的不准确性、计算成本和稳定性问题。相反，机器学习方法假设人类不了解参考系统，而是“训练”通用算法（通常是一类人工神经网络）来复制给定的输入-输出映射。神经网络的黑盒特性使它们极其灵活和强大；然而，缺点包括泛化误差，对大数据集的依赖和高计算要求。在这个研究项目中，目标是将两种现有的方法结合成混合的“灰盒”方法，以在效率、准确性和通用性方面获得更好的整体性能。这将通过物理信息神经网络（pinn）来完成。作为计算物理学中一个快速发展的研究领域，pinn是一种机器学习模型，它使用控制物理方程作为训练数据，并且已被证明比纯数据驱动的方法表现得更好。该研究项目将为声音合成的特定目的开发pin：创建端到端的神经网络，以准确，有效地模拟各种声学和电气系统。方法：pinn将为数字音频的三个子主题开发：虚拟模拟建模室内声学建模声音合成在每个领域，将进行一个或多个案例研究，一般方法是：确定系统的已知物理和控制方程。收集培训/测试数据，如录音。开发纯机器学习模型。整合物理约束，得到结果。调查、设计和优化模型架构。根据以下指标评估和比较模型：时域和频域的数值精度。感知准确性——通过一组志愿者的盲听测试。计算需求（CPU和内存）将最佳模型开发为原型仪器。

Physics-informed neural networks are an emerging class of algorithm in computational physics which combine traditional physics-based simulation methods with modern machine learning approaches. This project aims to develop physics-informed neural networks for the unique purpose of sound synthesis and audio effects modelling - emulating the sonic characteristics of musical instruments and equipment through digital signal processing. The motivation is to create next-generation digital instruments which are more realistic and therefore more inspiring to musicians. Specifically, the focus will be on vocal synthesizers, guitar amplifier simulations and virtual room environments. Context: Digital sound synthesis has been an active area of research since the 1960s when John Chowning pioneered a frequency modulation algorithm, later commercialised by Yamaha to develop the first digital synthesiser. Digital instruments and audio effects are now ubiquitous with music today, however a key challenge remains in their design: creating organic, realistic timbres - generally desirable qualities to musicians. Engineers and researchers therefore use two distinct approaches - physics-based modelling (white-box modelling) and machine learning (black-box modelling) - to digitally emulate the sound of musical systems including acoustic instruments, performance spaces, analogue circuitry (e.g., guitar amplifiers) or even human voices. Physics-based modelling involves analysis of the system to derive the governing differential equations, then solving these via numerical simulation to produce audio. This is often labour intensive and numerical simulation methods have inherent inaccuracies, computational costs, and stability issues. On the contrary, a machine learning approach assumes no human knowledge of the reference system and instead a general algorithm (normally a class of artificial neural network) is 'trained' to replicate a given input-output mapping. The black-box nature of neural networks makes them extremely flexible and powerful; however, disadvantages include generalization error, the reliance on large data sets and high computational requirements. In this research project, the objective is to combine the two existing approaches into hybrid 'grey-box' methods to obtain better overall performance in terms of efficiency, accuracy, and generality. This will be done through physics-informed neural networks (PINNs). A rapidly growing area of research in computational physics, PINNs are machine learning models which use the governing physical equations as training data and have been shown to perform better than purely data-driven methods. This research project will develop PINNs for the specific purpose of sound synthesis: creating end-to-end neural networks for the accurate, efficient simulation of various acoustic and electric systems. Methodology: PINNs will be developed for three sub-topics in digital audio: Virtual analogue modelling Room acoustics modelling Vocal synthesis In each area, one or more case studies will be undertaken, with the general methodology being: Determine the known physics of the system and governing equations. Gather training/testing data e.g., audio recordings. Develop pure machine learning models. Integrate physical constraints and obtain results. Investigate, design, and optimise model architectures. Evaluate and compare the models based on the following metrics: Numerical accuracy - in time and frequency domains. Perceptual accuracy - through blind listening tests with a group of volunteers. Computational demand (CPU and memory) Develop the best models into prototype instruments.