PFI-TT: Cooperative Listening with Networked Audio Devices

PFI-TT：与网络音频设备协作收听

• 批准号: 1919257
• 负责人: Andrew Singer
• 金额: $ 25万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-01 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1919257&HistoricalAwards=false
• 关键词: PFI; TT; Cooperative; Listening; Networked

The broader impact/commercial potential of this Partnerships for Innovation - Technology Translation (PFI-TT) project is to enable multiple listening devices to work together and improve their individual performance through collaborative signal processing. For both humans and machines, speech comprehension can be difficult in noisy, complex environments with several competing sound sources. Many of these environments may have multiple devices that in turn each contain multiple microphones, such as smart speakers, smart-home appliances, mobile devices, and wearables. Unfortunately, these devices do not currently cooperate to perform spatial sound capture. If they could be connected together to perform large-scale spatial signal processing, such a distributed array could dramatically improve the performance of machine listening, human-computer interaction, and human sensory augmentation tasks. The ability to precisely localize, separate, and enhance sound sources in complex environments would enable new applications that are impossible with current technology. These technologies could be applied to many already-deployed acoustic systems with minimal additional bandwidth and computation requirements. This research has the potential to dramatically impact these and other application areas while training graduate and undergraduate students and post-doctoral researchers from a range of underrepresented groups in lean-startup approaches to technology commercialization. The proposed project will develop technologies to aggregate data from multiple devices containing acoustic arrays, such as smart-home devices and wearables, to improve the spatial sound processing performance of each individual device. The proposed technology uses a hierarchical, distributed processing approach to efficiently aggregate information across networked devices without transmitting full synchronous audio data. The resulting system can leverage the spatial diversity of the distributed array to achieve better performance than a single device in listening tasks, especially in adverse environments with strong noise and interference where current technology often fails. The research objectives of this project are to explore the design trade-offs and scaling behavior of systems at large-scale; determine how best to aggregate distributed array data in the presence of non-idealities such as sample clock mismatch, network latency, and bandwidth constraints; to characterize the performance scaling and design tradeoffs of the system in real-world environments and under various acoustic and network conditions; and to implement and demonstrate the new algorithms on embedded hardware. Source separation and speech recognition experiments will be conducted in both controlled laboratory conditions and real-world environments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该创新-技术转换合作伙伴关系（PFI-TT）项目的更广泛影响/商业潜力是使多个收听设备能够一起工作，并通过协作信号处理提高其各自的性能。对于人类和机器来说，在具有多个竞争声源的嘈杂复杂环境中，语音理解可能很困难。这些环境中的许多环境可以具有多个设备，这些设备又各自包含多个麦克风，诸如智能扬声器、智能家用电器、移动的设备和可穿戴设备。不幸的是，这些设备当前不协作以执行空间声音捕获。如果它们可以连接在一起执行大规模空间信号处理，这样的分布式阵列可以显着提高机器听力，人机交互和人类感官增强任务的性能。在复杂环境中精确定位、分离和增强声源的能力将使当前技术无法实现的新应用成为可能。这些技术可以应用于许多已经部署的声学系统，具有最小的额外带宽和计算要求。这项研究有可能对这些和其他应用领域产生巨大影响，同时为来自一系列代表性不足群体的研究生和本科生以及博士后研究人员提供精益创业方法以实现技术商业化的培训。拟议项目将开发技术，以聚合来自包含声学阵列的多个设备（如智能家居设备和可穿戴设备）的数据，以提高每个设备的空间声音处理性能。所提出的技术使用分层的分布式处理方法来有效地跨联网设备聚合信息，而无需传输完全同步的音频数据。由此产生的系统可以利用分布式阵列的空间多样性，以在收听任务中实现比单个设备更好的性能，特别是在具有强噪声和干扰的恶劣环境中，当前技术往往失败。该项目的研究目标是探索大规模系统的设计权衡和缩放行为;确定如何最好地聚合分布式阵列数据，如采样时钟不匹配，网络延迟和带宽限制的非理想性;表征系统在真实世界环境中以及在各种声学和网络条件下的性能缩放和设计权衡;并在嵌入式硬件上实现和演示新算法。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Immersive Enhancement and Removal of Loudspeaker Sound Using Wireless Assistive Listening Systems and Binaural Hearing Devices

使用无线助听系统和双耳听力设备对扬声器声音进行沉浸式增强和消除

• DOI: 10.1109/icassp49357.2023.10096123
• 发表时间: 2023
• 期刊: ICASSP 2023
• 作者: Corey, Ryan M.;Singer, Andrew C.
• 通讯作者: Singer, Andrew C.

Development of an Underground Through-Soil Wireless Acoustic Communication System

• DOI: 10.1109/mwc.2019.1900030
• 发表时间: 2020-02
• 期刊: IEEE Wireless Communications
• 影响因子: 12.9
• 作者: Sijung Yang;Omar Baltaji;A. Singer;Y. Hashash

Cooperative Audio Source Separation and Enhancement Using Distributed Microphone Arrays and Wearable Devices

使用分布式麦克风阵列和可穿戴设备进行协作音频源分离和增强

• DOI: 10.1109/camsap45676.2019.9022475
• 发表时间: 2019
• 期刊: IEEE International Workshop on Computational Advances in Multisensor Adaptive Processing (CAMSAP
• 作者: Corey, Ryan M.;Skarha, Matthew D.;Singer, Andrew C.
• 通讯作者: Singer, Andrew C.

High/Low Model for Scalable Multimicrophone Enhancement of Speech Mixtures

用于可扩展多麦克风增强语音混合物的高/低模型

• DOI: 10.23919/eusipco54536.2021.9616105
• 发表时间: 2021
• 期刊: 2021 29th European Signal Processing Conference (EUSIPCO
• 作者: Corey, Ryan M.;Singer, Andrew C.
• 通讯作者: Singer, Andrew C.

Adaptive Crosstalk Cancellation and Spatialization for Dynamic Group Conversation Enhancement Using Mobile and Wearable Devices

使用移动和可穿戴设备增强动态群组对话的自适应串扰消除和空间化

• DOI: 10.1109/iwaenc53105.2022.9914773
• 发表时间: 2022
• 期刊: 2022 International Workshop on Acoustic Signal Enhancement (IWAENC
• 作者: Corey, Ryan M.;Mittal, Manan;Sarkar, Kanad;Singer, Andrew C.
• 通讯作者: Singer, Andrew C.