CAREER: Optimizing Human Speech Perception in Noisy Environments with User-Guided Machine Learning

在每年近200亿次的视频会议通话中，不必要的背景噪音经常阻碍设备介导的通信，并影响数百万助听器用户。方法被开发来去除不需要的噪声，但不幸的是，它们在许多真实的环境中表现不佳。随后，噪声去除方法通常提供低质量和难以理解的收听体验，这导致用户不满意和沮丧。这个教师早期职业发展项目将开发降噪和评估方法来解决这些问题，从而改善用户的听力体验。经常使用数字手段（例如语音会议和助听器）进行人与人之间沟通的个人和公司将成为这项工作的主要受益者。这项研究产生的数据和算法将使来自不同和跨学科领域的科学家和研究人员受益。此外，基于这项研究的教育活动将被整合到各种努力中，以增加这些研究领域中代表性不足的参与者的数量。该项目的主要目标是开发用户引导的机器学习算法，从而改善现实世界嘈杂环境中的听力体验。在包含许多竞争的谈话者的环境中，降噪系统无意中去除或保留了非预期的语音信号。拟议的研究活动将通过以下方式解决这一问题：（1）开发多模态计算方法，以识别特定用户想要听到的语音信号。研究人员通常使用计算评估指标来评估性能，但它们并不总是与个人用户情绪相关，这意味着研究人员的评估结果不准确。这个项目将（2）开发一个有效的界面，用于捕获和预测短期用户对质量和清晰度的评估。模拟和真实世界的语音数据在扬声器，噪声和环境特性方面有所不同，但目前的降噪方法无法适应这些差异。这是一个主要的缺点，因为部署的降噪系统将遇到未知的扬声器和噪声。研究人员将（3）开发一类新的用户引导机器学习算法，该算法利用近实时的真实和预测用户评估进行系统优化。成功完成这些任务将有助于更好地理解语音感知和提高降噪系统的可用性。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Unwanted background noise often hinders device-mediated communication during the nearly 20 billion yearly video conference calls and for millions of hearing aid users. Approaches are developed to remove unwanted noise, but unfortunately, they do not perform well in many real environments. Subsequently, the noise-removal approaches often provide low quality and unintelligible listening experiences, which results in dissatisfied and frustrated users. This Faculty Early Carrer Development project will develop noise-reduction and assessment approaches that address these issues, resulting in improved listening experiences for users. Individuals and companies that regularly use digital means (e.g. voice conferencing and hearing aids) for person-to-person communication will be major beneficiaries of this work. The data and algorithms that result from this research will be made available to benefit scientists and researchers from diverse and interdisciplinary fields. Additionally, educational activities based on this research will be integrated into various efforts to increase the number of underrepresented participants in these research areas.The main objective of this project is to develop user-guided machine-learning algorithms that result in improved listening experiences in real-world noisy environments. In environments that contain many competing talkers, noise-reduction systems inadvertently remove or retain unintended speech signals. The proposed research activities will address this by (1) developing multi-modal computational approaches that identify the speech signal that a specific user wants to hear. Computational assessment metrics are generally used by researchers to assess performance, but they do not always correlate with individual user sentiment, meaning investigators have inaccurate assessment results. This project will (2) develop an effective interface for capturing and predicting short-time user assessment of quality and intelligibility. Simulated and real-world speech data differ in terms of speaker, noise and environmental characteristics, but current noise-reduction approaches are incapable of adapting to these differences on the fly. This is a major shortcoming as deployed noise-reduction systems will encounter unknown speakers and noises. The investigator will (3) develop a novel class of user-guided machine learning algorithms that utilize true and predicted user assessment in near-real time for system optimization. Successfully completing these tasks will help better understand speech perception and increase the usability of noise-reduction systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.