Next Generation Virtual Musical Instruments: Physics-informed Neural Networks for Sound Synthesis and Digital Audio Effects

点击翻译按钮获取中文摘要

Physics-informed neural networks are an emerging class of algorithm in computational physics which combine traditional physics-based simulation methods with modern machine learning approaches. This project aims to develop physics-informed neural networks for the unique purpose of sound synthesis and audio effects modelling - emulating the sonic characteristics of musical instruments and equipment through digital signal processing. The motivation is to create next-generation digital instruments which are more realistic and therefore more inspiring to musicians. Specifically, the focus will be on vocal synthesizers, guitar amplifier simulations and virtual room environments. Context: Digital sound synthesis has been an active area of research since the 1960s when John Chowning pioneered a frequency modulation algorithm, later commercialised by Yamaha to develop the first digital synthesiser. Digital instruments and audio effects are now ubiquitous with music today, however a key challenge remains in their design: creating organic, realistic timbres - generally desirable qualities to musicians. Engineers and researchers therefore use two distinct approaches - physics-based modelling (white-box modelling) and machine learning (black-box modelling) - to digitally emulate the sound of musical systems including acoustic instruments, performance spaces, analogue circuitry (e.g., guitar amplifiers) or even human voices. Physics-based modelling involves analysis of the system to derive the governing differential equations, then solving these via numerical simulation to produce audio. This is often labour intensive and numerical simulation methods have inherent inaccuracies, computational costs, and stability issues. On the contrary, a machine learning approach assumes no human knowledge of the reference system and instead a general algorithm (normally a class of artificial neural network) is 'trained' to replicate a given input-output mapping. The black-box nature of neural networks makes them extremely flexible and powerful; however, disadvantages include generalization error, the reliance on large data sets and high computational requirements. In this research project, the objective is to combine the two existing approaches into hybrid 'grey-box' methods to obtain better overall performance in terms of efficiency, accuracy, and generality. This will be done through physics-informed neural networks (PINNs). A rapidly growing area of research in computational physics, PINNs are machine learning models which use the governing physical equations as training data and have been shown to perform better than purely data-driven methods. This research project will develop PINNs for the specific purpose of sound synthesis: creating end-to-end neural networks for the accurate, efficient simulation of various acoustic and electric systems. Methodology: PINNs will be developed for three sub-topics in digital audio: Virtual analogue modelling Room acoustics modelling Vocal synthesis In each area, one or more case studies will be undertaken, with the general methodology being: Determine the known physics of the system and governing equations. Gather training/testing data e.g., audio recordings. Develop pure machine learning models. Integrate physical constraints and obtain results. Investigate, design, and optimise model architectures. Evaluate and compare the models based on the following metrics: Numerical accuracy - in time and frequency domains. Perceptual accuracy - through blind listening tests with a group of volunteers. Computational demand (CPU and memory) Develop the best models into prototype instruments.