Spatial Analysis of Audio Images for Computational Musicology

点击翻译按钮获取中文摘要

Musicologists pursue the scholarly analysis of music with a variety of goals, including the desire to explore and understand creative musical processes. The evolving field of computational musicology brings computational methods to this pursuit. The proposed research program will develop computational methods around the investigation of creative musical processes in contemporary live electronic music that features spatial sound (i.e., rendered so that the listener can perceive the direction and location of a sound source, normally done with a surround speaker system) as part of the music. It builds on prior work done with musicologist colleagues (funded by SSHRC) and NSERC-DG-funded work on image and signal processing, and sound synthesis. The emphasis on spatial sound means that work will focus on sound-field data - in our case captured by a high-order ambisonic microphone (i.e., spherical array with a large number of microphone elements distributed over the sphere). To illustrate, consider the work of Clarke et al. (2013) - they analyze a spectrogram (an image showing sound decomposed in frequency and time) of a fragment of a musical performance by manually identifying parts of the image that correspond to salient musical objects, ultimately building a symbolic representation of the music. The proposed research follows Clarke's paradigm, but with the following important differences. First, we will bring our experience in computer vision and image/signal processing to automate the identification of salient musical objects. The notion of saliency is not trivial - I will be guided by both musicological knowledge of the subject matter and progress in the field of computer vision. Second, we will focus on spatial sound, building and relying on a microphone array and high-order ambisonic recordings. This allows us to follow the lead of O'Donovan et al. (2007) in using the microphone as an audio camera, making it possible to process two-dimensional audio data on a sphere as an image. Finally, we will be exploiting our growing database of live-spatial-electronic music recordings. We now have close to 80 GiB of high-order ambisonic recordings of contemporary spatial electronic music, which constitutes a unique resource in this type of research. While the focus of the research is computational musicology, the research has broader applications. HQP will be well equipped to work in any of the emerging fields that need sound field analysis.  Two important application areas are environmental noise assessment and amelioration, and virtual reality (VR) systems. A local sound engineering firm is already engaging with us to collaborate on measurement and analysis of industrial noise. Sound systems that will accompany VR graphics are in their infancy. HQP expertise in sound fields will be essential in the future of VR, and my HQP will have the added advantage of collaboration opportunities with experts in cultural applications of VR.