PFI-TT: Cooperative Listening with Networked Audio Devices

该创新-技术转换合作伙伴关系（PFI-TT）项目的更广泛影响/商业潜力是使多个收听设备能够一起工作，并通过协作信号处理提高其各自的性能。对于人类和机器来说，在具有多个竞争声源的嘈杂复杂环境中，语音理解可能很困难。这些环境中的许多环境可以具有多个设备，这些设备又各自包含多个麦克风，诸如智能扬声器、智能家用电器、移动的设备和可穿戴设备。不幸的是，这些设备当前不协作以执行空间声音捕获。如果它们可以连接在一起执行大规模空间信号处理，这样的分布式阵列可以显着提高机器听力，人机交互和人类感官增强任务的性能。在复杂环境中精确定位、分离和增强声源的能力将使当前技术无法实现的新应用成为可能。这些技术可以应用于许多已经部署的声学系统，具有最小的额外带宽和计算要求。这项研究有可能极大地影响这些和其他应用领域，同时培训来自一系列代表性不足的群体的研究生和本科生以及博士后研究人员，以精益创业的方式实现技术商业化。拟议项目将开发技术，以聚合来自包含声学阵列的多个设备（如智能家居设备和可穿戴设备）的数据，以提高每个设备的空间声音处理性能。所提出的技术使用分层的分布式处理方法来有效地跨联网设备聚合信息，而无需传输完全同步的音频数据。由此产生的系统可以利用分布式阵列的空间多样性，以在收听任务中实现比单个设备更好的性能，特别是在具有强噪声和干扰的恶劣环境中，当前技术往往失败。该项目的研究目标是探索大规模系统的设计权衡和缩放行为;确定如何最好地聚合分布式阵列数据，如采样时钟不匹配，网络延迟和带宽限制的非理想性;表征系统在真实世界环境中以及在各种声学和网络条件下的性能缩放和设计权衡;并在嵌入式硬件上实现和演示新算法。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact/commercial potential of this Partnerships for Innovation - Technology Translation (PFI-TT) project is to enable multiple listening devices to work together and improve their individual performance through collaborative signal processing. For both humans and machines, speech comprehension can be difficult in noisy, complex environments with several competing sound sources. Many of these environments may have multiple devices that in turn each contain multiple microphones, such as smart speakers, smart-home appliances, mobile devices, and wearables. Unfortunately, these devices do not currently cooperate to perform spatial sound capture. If they could be connected together to perform large-scale spatial signal processing, such a distributed array could dramatically improve the performance of machine listening, human-computer interaction, and human sensory augmentation tasks. The ability to precisely localize, separate, and enhance sound sources in complex environments would enable new applications that are impossible with current technology. These technologies could be applied to many already-deployed acoustic systems with minimal additional bandwidth and computation requirements. This research has the potential to dramatically impact these and other application areas while training graduate and undergraduate students and post-doctoral researchers from a range of underrepresented groups in lean-startup approaches to technology commercialization. The proposed project will develop technologies to aggregate data from multiple devices containing acoustic arrays, such as smart-home devices and wearables, to improve the spatial sound processing performance of each individual device. The proposed technology uses a hierarchical, distributed processing approach to efficiently aggregate information across networked devices without transmitting full synchronous audio data. The resulting system can leverage the spatial diversity of the distributed array to achieve better performance than a single device in listening tasks, especially in adverse environments with strong noise and interference where current technology often fails. The research objectives of this project are to explore the design trade-offs and scaling behavior of systems at large-scale; determine how best to aggregate distributed array data in the presence of non-idealities such as sample clock mismatch, network latency, and bandwidth constraints; to characterize the performance scaling and design tradeoffs of the system in real-world environments and under various acoustic and network conditions; and to implement and demonstrate the new algorithms on embedded hardware. Source separation and speech recognition experiments will be conducted in both controlled laboratory conditions and real-world environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.