PFI-TT: Cooperative Listening with Networked Audio Devices

这项创新-技术转化伙伴关系（PFI-TT）项目的更广泛影响/商业潜力是使多个监听设备能够协同工作，并通过协作信号处理提高其单个性能。对于人类和机器来说，在有几个相互竞争的声源的嘈杂、复杂的环境中，语音理解都是困难的。许多这样的环境可能有多个设备，每个设备又包含多个麦克风，例如智能扬声器、智能家用电器、移动设备和可穿戴设备。不幸的是，这些设备目前不能合作执行空间声音捕获。如果它们可以连接在一起进行大规模的空间信号处理，这种分布式阵列可以显著提高机器聆听、人机交互和人类感官增强任务的性能。在复杂环境中精确定位、分离和增强声源的能力将使当前技术无法实现的新应用成为可能。这些技术可以应用于许多已经部署的声学系统，只需要很少的额外带宽和计算需求。这项研究有可能对这些和其他应用领域产生巨大影响，同时从一系列代表性不足的群体中培养研究生和本科生以及博士后研究人员，使其采用精益创业方法实现技术商业化。拟议的项目将开发技术，从包含声学阵列的多个设备（如智能家居设备和可穿戴设备）收集数据，以提高每个单独设备的空间声音处理性能。该技术采用分层分布式处理方法，在不传输完全同步音频数据的情况下，有效地跨网络设备聚合信息。由此产生的系统可以利用分布式阵列的空间分集，在侦听任务中获得比单个设备更好的性能，特别是在具有强噪声和干扰的恶劣环境中，当前技术通常无法实现。该项目的研究目标是探索大规模系统的设计权衡和缩放行为；确定在存在非理想情况（如采样时钟不匹配、网络延迟和带宽限制）的情况下，如何最好地聚合分布式数组数据；描述系统在真实环境和各种声学和网络条件下的性能缩放和设计权衡；并在嵌入式硬件上对新算法进行了实现和演示。源分离和语音识别实验将在受控的实验室条件和现实环境中进行。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact/commercial potential of this Partnerships for Innovation - Technology Translation (PFI-TT) project is to enable multiple listening devices to work together and improve their individual performance through collaborative signal processing. For both humans and machines, speech comprehension can be difficult in noisy, complex environments with several competing sound sources. Many of these environments may have multiple devices that in turn each contain multiple microphones, such as smart speakers, smart-home appliances, mobile devices, and wearables. Unfortunately, these devices do not currently cooperate to perform spatial sound capture. If they could be connected together to perform large-scale spatial signal processing, such a distributed array could dramatically improve the performance of machine listening, human-computer interaction, and human sensory augmentation tasks. The ability to precisely localize, separate, and enhance sound sources in complex environments would enable new applications that are impossible with current technology. These technologies could be applied to many already-deployed acoustic systems with minimal additional bandwidth and computation requirements. This research has the potential to dramatically impact these and other application areas while training graduate and undergraduate students and post-doctoral researchers from a range of underrepresented groups in lean-startup approaches to technology commercialization. The proposed project will develop technologies to aggregate data from multiple devices containing acoustic arrays, such as smart-home devices and wearables, to improve the spatial sound processing performance of each individual device. The proposed technology uses a hierarchical, distributed processing approach to efficiently aggregate information across networked devices without transmitting full synchronous audio data. The resulting system can leverage the spatial diversity of the distributed array to achieve better performance than a single device in listening tasks, especially in adverse environments with strong noise and interference where current technology often fails. The research objectives of this project are to explore the design trade-offs and scaling behavior of systems at large-scale; determine how best to aggregate distributed array data in the presence of non-idealities such as sample clock mismatch, network latency, and bandwidth constraints; to characterize the performance scaling and design tradeoffs of the system in real-world environments and under various acoustic and network conditions; and to implement and demonstrate the new algorithms on embedded hardware. Source separation and speech recognition experiments will be conducted in both controlled laboratory conditions and real-world environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.