CAREER: Optimizing Human Speech Perception in Noisy Environments with User-Guided Machine Learning

在每年近200亿次的视频电话会议和数百万助听器用户中，不必要的背景噪音经常阻碍设备介导的通信。人们开发了一些方法来去除不必要的噪声，但不幸的是，它们在许多实际环境中表现不佳。随后，降噪方法往往提供低质量和难以理解的聆听体验，这导致用户不满和沮丧。这个教师早期职业发展项目将开发降噪和评估方法来解决这些问题，从而改善用户的聆听体验。经常使用数字手段（例如语音会议和助听器）进行人对人交流的个人和公司将成为这项工作的主要受益者。这项研究产生的数据和算法将使来自不同和跨学科领域的科学家和研究人员受益。此外，以这项研究为基础的教育活动将纳入各种努力，以增加这些研究领域代表性不足的参与者的人数。该项目的主要目标是开发用户引导的机器学习算法，从而改善现实嘈杂环境中的听力体验。在包含许多相互竞争的说话者的环境中，降噪系统会无意中去除或保留无意的语音信号。提出的研究活动将通过(1)开发多模态计算方法来解决这个问题，该方法可以识别特定用户想要听到的语音信号。研究人员通常使用计算评估指标来评估性能，但它们并不总是与个人用户的情绪相关，这意味着研究人员的评估结果不准确。该项目将(2)开发一个有效的界面，用于捕获和预测短期用户对质量和可理解性的评估。模拟和真实语音数据在说话者、噪声和环境特征方面存在差异，但目前的降噪方法无法适应这些差异。这是一个主要的缺点，因为部署的降噪系统会遇到未知的扬声器和噪音。研究者将(3)开发一种新型的用户引导机器学习算法，该算法利用近乎实时的真实和预测的用户评估进行系统优化。成功完成这些任务将有助于更好地理解语音感知并提高降噪系统的可用性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Unwanted background noise often hinders device-mediated communication during the nearly 20 billion yearly video conference calls and for millions of hearing aid users. Approaches are developed to remove unwanted noise, but unfortunately, they do not perform well in many real environments. Subsequently, the noise-removal approaches often provide low quality and unintelligible listening experiences, which results in dissatisfied and frustrated users. This Faculty Early Carrer Development project will develop noise-reduction and assessment approaches that address these issues, resulting in improved listening experiences for users. Individuals and companies that regularly use digital means (e.g. voice conferencing and hearing aids) for person-to-person communication will be major beneficiaries of this work. The data and algorithms that result from this research will be made available to benefit scientists and researchers from diverse and interdisciplinary fields. Additionally, educational activities based on this research will be integrated into various efforts to increase the number of underrepresented participants in these research areas.The main objective of this project is to develop user-guided machine-learning algorithms that result in improved listening experiences in real-world noisy environments. In environments that contain many competing talkers, noise-reduction systems inadvertently remove or retain unintended speech signals. The proposed research activities will address this by (1) developing multi-modal computational approaches that identify the speech signal that a specific user wants to hear. Computational assessment metrics are generally used by researchers to assess performance, but they do not always correlate with individual user sentiment, meaning investigators have inaccurate assessment results. This project will (2) develop an effective interface for capturing and predicting short-time user assessment of quality and intelligibility. Simulated and real-world speech data differ in terms of speaker, noise and environmental characteristics, but current noise-reduction approaches are incapable of adapting to these differences on the fly. This is a major shortcoming as deployed noise-reduction systems will encounter unknown speakers and noises. The investigator will (3) develop a novel class of user-guided machine learning algorithms that utilize true and predicted user assessment in near-real time for system optimization. Successfully completing these tasks will help better understand speech perception and increase the usability of noise-reduction systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.