ITR: Personalized Spatial Audio via Scientific Computing and Computer Vision

这是一项为期5年的持续资助的头4年。人类非常擅长在不同的环境（从开放空间到拥挤的小房间）中，利用频率相关的耳间时间差（ITD）、耳间声级差（ILD）和耳廓频谱线索的混合来识别声音的空间来源。这种能力帮助我们从杂音中分辨出单独的声音，从而与他人和环境进行互动，并通过在比视觉更广阔的空间内警告我们危险，帮助我们生存。空间声音的这些优势对于人机交互非常重要。虽然与两耳相关的频率无关的ITD信号（延迟）相对容易通过耳机呈现，但ILD（电平差）和耳廓高度信号却不是这样。对于给定的声源位置和频率内容，声音由人的躯干、头部和耳廓散射，并且在两只耳朵上接收到不同的声音，导致接收声音的强度和频谱特征的差异。这些影响被编码在一个极其个体的“头部相关传递函数”（HRTF）中，该函数取决于人的解剖特征（躯干、头部和耳廓的结构）。这种个性使得在建议的应用程序中使用HRTF变得困难。最近的研究，包括该团队成员的研究，主要集中在测量特定环境中个体的HRTF，构建HRTF模型，理解身体几何形状与HRTF特征之间的关系，以及大脑如何处理线索以获得空间信息。然而，这项研究也表明，当声音的HRTF不正确时，大脑对提示中的错误非常敏感。在这个项目中，PI和他的团队将使用数值方法从精确的人体三维表面模型中计算个性化的hrtf。他们将使用多视图、多帧计算视觉技术从图像中提取表面模型。然后，他们将使用边界元方法，采用快速多极/变换技术和并行处理，从表面模型计算hrtf。由此产生的hrtf将通过与声学测量的hrtf和心理声学测试的客观比较来评估，并将用于虚拟现实、增强现实和远程会议的演示。这种基于视觉的方法的一个主要优点是，它将允许PI和他的团队调查和建模hrtf随身体姿势变化的方式，提供跟踪动态环境的潜力。因此，该项目将包括基础研究，将静态HRTF测量扩展到不同环境中的动态情况，使用视觉跟踪组合来定位真实空间中的人，以及使用快速迭代技术从自由场HRTF构建室内HRTF。这将为音频渲染的HCI应用提供科学基础。此外，该研究还将产生算法和理解，这将对各个领域产生影响，包括基于计算机视觉的模型创建；科学计算;噪声控制和地雷探测的计算声学；人听觉的神经生理学认识等。

This is the first 4 years funding of a five-year continuing award. Humans are very good at discerning the spatial origin of sound using a mixture of frequency-dependent interaural time difference (ITD), interaural level difference (ILD), and pinna spectral cues in disparate environments ranging from open spaces to small crowded rooms. This ability helps us to interact with others and the environment by sorting out individual sounds from a mixture, and helps us to survive by warning us of danger over a wider region of space compared to vision. These advantages of spatial sound are important for human-computer interaction. While the frequency-independent ITD cues (delays) associated with the two ears are relatively easy to render over headphones, the ILD (level difference) and pinna elevation cues are not. For a given source location and frequency content, the sound scattered by the person's torso, head and pinnae, and is received differently at the two ears, leading to differences in the intensity and spectral features of the received sound. These effects are encoded in an extremely individual "Head Related Transfer Function" (HRTF) that depends on the person's anatomical features (structure of the torso, head and pinnae). This individuality has made it difficult to use the HRTF in the proposed applications. Recent research, including that of members of this team, has focused on measuring the HRTFs for individuals in specific environments, on constructing models of the HRTF, on understanding how the geometry of the body is related to the characteristics of HRTF, and how the brain processes the cues to derive spatial information. However, this research has also indicated that the brain is extraordinarily perceptive to errors in cues that result when sound is rendered with an incorrect HRTF.In this project the PI and his team will use numerical methods to compute individualized HRTFs from accurate 3-D surface models of the body. They will use multiview, multiframe computational vision techniques to extract the surface models from imagery. They will then use boundary element methods employing fast multipole/ transform techniques and parallel processing to compute the HRTFs from the surface models. The resulting HRTFs will be evaluated both by objective comparisons with acoustically measured HRTFs and by psychoacoustic testing, and will be used in demonstrations of virtual reality, augmented reality, and teleconferencing. A major advantage of this vision-based approach is that it will allow the PI and his team to investigate and model the way that HRTFs change with body posture, providing the potential of tracking dynamic environments. Thus, the project will include fundamental research to extend the static HRTF measurements to dynamic situations in different environments, using a combination of visual tracking to locate the person in real space, and construction of in-room HRTFs from free-field HRTFs using fast iterative techniques. This will provide a scientific foundation for HCI applications of audio rendering. The research will in addition yield algorithms and understanding that will have an impact on varied fields, including computer vision based model creation; scientific computing; computational acoustics for noise control and land mine detection; neurophysiological understanding of human audition; etc.