Leveraging Natural Language Processing for Reverberant Speech Enhancement in Cochlear Implants

摘要 这个项目的首要目标是开发算法来解决人工耳蜗植入的困难 (Ci)用户在教堂、礼堂和教堂等有回响的聆听环境中体验口译 教室。最近的研究利用时频掩蔽技术在这一领域取得了进展，但 这些算法通常在变化的声学环境中不是很健壮，或者不适合实时 正在处理。机器学习(ML)和人工智能(AI)技术在许多领域蓬勃发展 应用领域最近，但到目前为止，AI/ML方法在CI用户中的混响显示有限 成功。我们的方法是研究几种AI/ML语音增强方法，该方法基于 自然语言处理(NLP)领域，本质上识别混响中的语音，然后对其进行清理。我们 将使用NIH支持的开源CCI-MOBILE对算法性能进行最终评估 CI研究平台的易用性和灵活性是开发CI信号处理和构建CI信号处理原型所必需的 算法。我们建议使用基于音素的识别和自动语音识别(ASR) 开发和测试我们的混响缓解算法的方法。目标1将调查实时 在基于ML的T-F掩蔽中开发音素识别的可行性。我们将写一部小说 基于音素的T-F掩码估计算法和离线算法进行语音识别测试 模式来比较传统的和基于音素的T-F掩蔽。这项工作将决定音素是否 知识对于顺序词中的语音增强是有益的。目标2将研究实时T-F掩模的用途 对CI用户的估计。我们将实施各种T-F掩码估计算法来缓解混响 来自CCI中实时的文献(包括我们在AIM 1中开发的基于音素的T-F算法)- 莫比尔县。除了对语音清晰度的影响外，算法还将根据CI进行基准测试 视听异步化的计算极限和可容忍时间延迟。这项工作将评估 T-F掩码估计算法在实时运行条件下的有效性。AIM 3将调查 通过ASR和文本到语音合成(ASR-TTS)提高CI用户的语音清晰度。我们会 研究各种前端语音增强策略以改进ASR预测和TTS引擎 用普通的和熟悉的合成声音。本工作将使用CCI-MOBILE来评估ASR-TTS的效用 以及说话人熟悉度对CI使用者混响语音清晰度的影响。我们团队带来了AI/ML， 硬件，实验测试和听力学经验，这将是成功的研究需要。CCI- 云是CCI-Mobile的一项云功能，将用于促进远程和协作的CI用户研究。 我们的工作具有很高的创新性，有可能推动一种向AI/ML驱动的听觉的范式转变 利用NLP使语音处理策略适应声学设置的假体，以最大限度地提高用户 福利。被证明的成功将改善CI用户的生活质量。

ABSTRACT The overarching goal of this project is to develop algorithms to address the difficulties that cochlear implant (CI) users experience interpreting speech in reverberant listening environments like churches, auditoriums and classrooms. Recent research has made progress in this area using time-frequency masking techniques, but these algorithms are often not robust in changing acoustic environments or are not amenable to real time processing. Machine learning (ML) and artificial intelligence (AI) techniques are burgeoning in many applications areas recently, but to date, AI/ML approaches for reverberation in CI users have shown limited success. Our proposed approach is to investigate several AI/ML speech enhancement methods based on the natural language processing (NLP) field to essentially recognize speech in reverberation and then clean it. We will provide final assessment of algorithm performance by using the open-source NIH-supported CCi-MOBILE CI research platform for its ease and flexibility necessary for developing and prototyping CI signal processing algorithms. We propose to use phoneme-based recognition and automatic speech recognition (ASR) approaches to develop and test our reverberation mitigation algorithms. Aim 1 will investigate the real-time feasibility of exploiting phoneme recognition for ML-based T-F masking in CIs. We will develop a novel phoneme-based T-F mask estimation algorithm and conduct speech recognition tests with an offline algorithm mode to compare conventional and phoneme-based T-F masking. This work will determine whether phoneme knowledge is beneficial for speech enhancement in CIs. Aim 2 will investigate the utility of real-time T-F mask estimation in CI users. We will implement various T-F mask estimation algorithms to mitigate reverberation from the literature (including our novel phoneme-based T-F algorithm developed in Aim 1) in real-time in CCi- MOBILE. In addition to their impact on speech intelligibility, algorithms will be benchmarked against CI computational limits and tolerable time delays of audiovisual asynchrony. This work will evaluate the effectiveness of T-F mask estimation algorithms in real-time operational conditions. Aim 3 will investigate advancing speech intelligibility for CI users via ASR and text-to-speech synthesis (ASR-TTS). We will investigate various front-end speech enhancement strategies to improve ASR predictions and TTS engines with generic and familiar synthetic voices. This work will use CCi-MOBILE to evaluate the utility of ASR-TTS and the effect of speaker familiarity on reverberant speech intelligibility in CI users. Our team brings AI/ML, hardware, experimental testing and audiology experience that will be needed for successful research. CCi- CLOUD, a cloud feature of CCI-MOBILE, will be used to facilitate remote and collaborative CI user studies. Our work is highly innovative and has the potential to instigate a paradigm shift towards AI/ML-driven auditory protheses that leverage NLP to adapt speech processing strategies to acoustic settings to maximize user benefits. Demonstrated success will improve the quality of life of CI users.