US-German Research Proposal: ADaptive low-latency SPEEch Decoding and synthesis using intracranial signals (ADSPEED)

最近的研究表明，通过对大脑活动的侵入性测量，直接合成可理解的语音是可能的。然而，这些方法在大脑活动和可听语音输出之间存在可察觉的延迟，从而阻碍了自然的语音交流。此外，这些方法通常需要预先录制语音，因此不能直接应用于无法说话和产生此类录音的人。该项目旨在开发从大脑活动合成语音的方法，而不会产生可察觉的处理延迟，而不依赖于用户预先录制的语音。最终目标是开发一种系统，使数百万患有严重语言障碍的人，包括那些完全丧失语言能力的人，恢复自然的口语交流。该项目分为三个研究重点。第一个重点是异步和无声学模型训练，其中将使用基于动态时间翘曲和从相应文本表示中推断预期的内部语音声学的方法创建用户发声语音的新替代品。第二个重点是在线验证和用户自适应，其中现有的低延迟语音解码和合成方案不具有固有的适应性，将使用在线人体受试者实验以闭环方式进行验证。这将为用户如何响应和适应人工合成语音输出提供有价值的见解。第三个重点是低延迟系统-用户共同适应方案的开发和测试。共同适应，即用户和系统都适应以优化合成输出，对于在缺乏可靠的建模代理的情况下揭示内部（即想象或尝试）语音的难以捉摸的表示是至关重要的。因此，本研究将同时促进对内部语音的神经表征的理解，进而促进内部语音的自适应解码，从而发展实用的闭环语音神经假肢。德国联邦教育和研究部（BMBF）正在资助一个伙伴项目。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Recent research has demonstrated that it is possible to synthesize intelligible speech sounds directly from invasive measurements of brain activity. However, these approaches have a perceptible delay between brain activity and audible speech output, preventing a natural spoken communication. Furthermore, the approaches generally require pre-recorded speech and thus cannot be directly applied to people who are unable to speak and generate such recordings. This project aims to develop methods for synthesizing speech from brain activity without perceptible processing delay that do not rely on pre-recorded speech from the user. The ultimate goal is to develop a system that restores natural spoken communication to the millions of people who suffer from severe speech disorders, including those with complete loss of speech. The project is organized into three research thrusts. The first thrust focuses on asynchronous and acoustics-free model training, where novel surrogates to the user's vocalized speech will be created using approaches based on dynamic time warping and the inference of intended inner-speech acoustics from corresponding textual representations. The second thrust focuses on online validation and user adaptation, where the existing low-latency speech decoding and synthesis scheme, which is not inherently adaptable, will be validated in a closed-loop fashion using online human-subject experiments. This will provide valuable insights into how the user responds and adapts to the artificial, synthesized speech output. The third thrust focuses on the development and testing of low-latency system-user co-adaptation schemes. Co-adaptation, where both the user and system adapt to optimize the synthesized output, is crucial for revealing the elusive representations of inner (i.e., imagined or attempted) speech in the absence of a reliable surrogate for modeling. As a result, this research will simultaneously advance the understanding of the neural representations of inner speech and, in turn, co-adaptive inner speech decoding toward the development of practical closed-loop speech neuroprosthetics.A companion project is being funded by the Federal Ministry of Education and Research, Germany (BMBF).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.