ITR: Personalized Spatial Audio via Scientific Computing and Computer Vision

这是一个为期五年的连续奖励的第一个四年资助。 人类非常善于在从开放空间到小型拥挤房间的不同环境中使用频率相关的耳间时间差（ITD）、耳间电平差（ILD）和耳廓频谱线索的混合来辨别声音的空间来源。 这种能力可以帮助我们与他人和环境互动，从混合物中分辨出单个声音，并通过警告我们比视觉更广泛的空间区域的危险来帮助我们生存。 空间声音的这些优点对于人机交互是重要的。虽然与双耳相关的频率无关的ITD提示（延迟）相对容易通过耳机呈现，但ILD（电平差）和耳廓高度提示则不然。 对于给定的源位置和频率内容，声音被人的躯干、头部和耳廓散射，并且在两个耳朵处被不同地接收，导致接收到的声音的强度和频谱特征的差异。 这些影响被编码在一个非常个人的“头部相关传递函数”（HRTF），这取决于人的解剖特征（躯干，头部和脊椎的结构）。 这种个性使得难以在所提出的应用中使用HRTF。 最近的研究，包括这个团队的成员，已经集中在测量的HRTF为个人在特定的环境中，在构建模型的HRTF，了解身体的几何形状是如何与HRTF的特性，以及大脑如何处理的线索，以获得空间信息。 然而，这项研究也表明，大脑对错误的线索是非常敏感的，当声音被错误的HRTF渲染时，在这个项目中，PI和他的团队将使用数值方法从精确的三维身体表面模型计算个性化的HRTF。 他们将使用多视图、多帧计算视觉技术从图像中提取表面模型。 然后，他们将使用边界元方法，采用快速多极/变换技术和并行处理来计算表面模型的HRTF。 由此产生的HRTF将通过与声学测量的HRTF的客观比较和心理声学测试进行评估，并将用于虚拟现实，增强现实和电话会议的演示。 这种基于视觉的方法的一个主要优点是，它将允许PI和他的团队调查和建模HRTF随身体姿势变化的方式，从而提供跟踪动态环境的潜力。 因此，该项目将包括基础研究，以扩展静态HRTF测量在不同的环境中的动态情况下，使用视觉跟踪的组合，以定位在真实的空间的人，和建设室内HRTF从自由场HRTF使用快速迭代技术。 这将为音频渲染的HCI应用提供科学的基础。 此外，该研究还将产生算法和理解，这些算法和理解将对各个领域产生影响，包括基于计算机视觉的模型创建;科学计算;用于噪声控制和地雷探测的计算声学;对人类听觉的神经生理学理解等。

This is the first 4 years funding of a five-year continuing award. Humans are very good at discerning the spatial origin of sound using a mixture of frequency-dependent interaural time difference (ITD), interaural level difference (ILD), and pinna spectral cues in disparate environments ranging from open spaces to small crowded rooms. This ability helps us to interact with others and the environment by sorting out individual sounds from a mixture, and helps us to survive by warning us of danger over a wider region of space compared to vision. These advantages of spatial sound are important for human-computer interaction. While the frequency-independent ITD cues (delays) associated with the two ears are relatively easy to render over headphones, the ILD (level difference) and pinna elevation cues are not. For a given source location and frequency content, the sound scattered by the person's torso, head and pinnae, and is received differently at the two ears, leading to differences in the intensity and spectral features of the received sound. These effects are encoded in an extremely individual "Head Related Transfer Function" (HRTF) that depends on the person's anatomical features (structure of the torso, head and pinnae). This individuality has made it difficult to use the HRTF in the proposed applications. Recent research, including that of members of this team, has focused on measuring the HRTFs for individuals in specific environments, on constructing models of the HRTF, on understanding how the geometry of the body is related to the characteristics of HRTF, and how the brain processes the cues to derive spatial information. However, this research has also indicated that the brain is extraordinarily perceptive to errors in cues that result when sound is rendered with an incorrect HRTF.In this project the PI and his team will use numerical methods to compute individualized HRTFs from accurate 3-D surface models of the body. They will use multiview, multiframe computational vision techniques to extract the surface models from imagery. They will then use boundary element methods employing fast multipole/ transform techniques and parallel processing to compute the HRTFs from the surface models. The resulting HRTFs will be evaluated both by objective comparisons with acoustically measured HRTFs and by psychoacoustic testing, and will be used in demonstrations of virtual reality, augmented reality, and teleconferencing. A major advantage of this vision-based approach is that it will allow the PI and his team to investigate and model the way that HRTFs change with body posture, providing the potential of tracking dynamic environments. Thus, the project will include fundamental research to extend the static HRTF measurements to dynamic situations in different environments, using a combination of visual tracking to locate the person in real space, and construction of in-room HRTFs from free-field HRTFs using fast iterative techniques. This will provide a scientific foundation for HCI applications of audio rendering. The research will in addition yield algorithms and understanding that will have an impact on varied fields, including computer vision based model creation; scientific computing; computational acoustics for noise control and land mine detection; neurophysiological understanding of human audition; etc.