I-Corps: Creating Immersive 3D Audio

人类(和其他动物)通过听觉输入感知空间的过程是复杂的。在过去的几十年里，世界各地的研究人员(包括参与当前提案的研究人员)阐明了人类感知声音的机制，并能够开发用于创建沉浸式音频的算法。这些音频包括来自游戏的音频、来自音乐混合的音频以及来自现场事件的捕获的音频。由于可听声音的波长与环境物体的大小和听者身体特征的大小相当，因此听者感受到的声音通过散射这些物体而被“着色”。理解和有效地逼近分别表征这些散射过程的房间脉冲响应和头部相关传递函数(HRTF)是过去十年来PI研究的主要贡献。由于每个听众的身体特征都有不同的形状和大小，S的HRTF在人与人之间表现出相当大的差异。PI开发了非常快速的技术来测量HRTF，根据听众的身体和耳朵测量来表征HRTF，以及使用HRTF和房间响应快速创建身临其境的声音。此外，单个麦克风记录丢失了声音信号中的空间信息。为了在捕捉听觉环境的过程中保留空间信息，PI发展了新颖的球形麦克风阵列技术。根据平面波基函数对捕获的声音进行分解，可以容易地将HRTF合并到捕获的声音中。今天的消费者在他们的移动设备上享受音乐、游戏和其他媒体；大多数人满足于通过耳机发出的平淡无奇的声音。目前的耳机音效根本无法产生高端音乐系统、电影院或现场活动提供的引人入胜的声音体验。凭借之前在NSF支持下开发的技术，该团队让娱乐内容创作者在通过耳机呈现声音时实现了无与伦比的现实感。与团队的声音捕获硬件相结合，可以实现现场活动的身临其境的再现。去年，智能手机的销量有史以来首次超过了个人电脑；因此，通过耳机消费的内容数量正在大幅增加。我们的技术将使公司能够让耳机聆听变得更加身临其境，并接触到不断增长的消费者基础。有几个行业生产和消费音响产品。这些公司包括移动设备/游戏/PC硬件制造商、娱乐内容制作商和内容交付网络。他们都将受到这项技术的积极影响。由于巨大的市场和团队的实力，他们预计可以围绕这项技术建立一个大而成功的业务。如果成功，这一努力将有助于美国在移动技术、游戏、娱乐和沉浸式模拟领域继续保持领先地位。

The process by which humans (and other animals) perceive space from auditory input is complex. Over the past decades, researchers worldwide (including those involved in the current proposal) have elucidated the mechanisms by which humans perceive sound and are able to develop algorithms for the creation of immersive audio. These include audio from games, from the mixing of music, and from the capture of live events. Because the wavelengths of audible sound are comparable to the sizes of environmental objects and of features on the listeners' bodies, the sound perceived by the listener is "colored" by scattering off these objects. Understanding and efficiently approximating the room impulse response and the Head Related Transfer Function (HRTF), which respectively characterize these scattering processes, has been a major contribution of the PI's research over the past decade. As each listener?s physical body features have different shapes and sizes, the HRTF shows considerable inter-personal variation. The PI has developed extremely rapid techniques for measuring HRTFs, for characterizing HRTFs based on the body and ear measurements of listeners, and for rapid creation of immersive sound using HRTFs and room responses. Further, single microphone recordings lose the spatial information in the sound signal. To retain the spatial information during auditory environment capture, the PI has developed novel spherical microphone array technology. A decomposition of the captured sound in terms of plane-wave basis functions allows the easy incorporation of HRTFs in the captured sound. Today's consumers enjoy music, games, and other media on their mobile devices; most settle for lackluster sound produced over headphones. Current headphone sound is simply unable to produce the engaging sound experience that high-end music systems, movie theaters, or live events deliver. With the technologies developed under previous NSF support, the team has allowed entertainment content creators to achieve unmatched realism in sound presentation over headphones. Combined with the team's sound capture hardware, immersive reproduction of live events can be achieved. In the last year, the number of smartphone sales outpaced that of PCs for the first time in history; consequentially, the amount of content consumed over headphones is vastly increasing. Our technology will allow companies to make headphone listening more immersive and reach that growing consumer base. Several industries produce and consume audio products. These include mobile device / gaming / PC hardware manufacturers, entertainment content producers, and content delivery networks. They will all be positively impacted by the technology. Because of the large market, and the strength of the team, they anticipate that a large and successful business can be built around this technology. If successful, this effort will contribute to ongoing U.S. leadership in the fields of mobile technologies, gaming, entertainment, and immersive simulation.