US-German Research Proposal: ADaptive low-latency SPEEch Decoding and synthesis using intracranial signals (ADSPEED)

最近的研究表明，通过对大脑活动的侵入性测量，直接合成可理解的语音是可能的。然而，这些方法在大脑活动和可听语音输出之间存在明显的延迟，阻碍了自然的口头交流。此外，这些方法通常需要预先录制的语音，因此不能直接应用于不能说话和生成此类录音的人。这个项目的目标是开发从大脑活动合成语音的方法，而不会出现明显的处理延迟，并且不依赖于用户预先录制的语音。最终目标是开发一种系统，恢复数百万患有严重言语障碍的人的自然口语交流，包括那些完全丧失言语的人。该项目被组织成三个研究项目。第一个重点是异步和无声学模型训练，其中将使用基于动态时间弯曲的方法和从对应的文本表示中推断预期的语音内部声学的方法来创建用户发声语音的新替代品。第二个重点是在线验证和用户自适应，现有的低延迟语音解码和合成方案本身并不具有自适应性，将使用在线人体实验以闭环方式进行验证。这将为用户如何响应和适应人工合成语音输出提供有价值的见解。第三个重点是开发和测试低延迟系统-用户共同适应计划。协同适应，其中用户和系统都适应以优化合成输出，对于在缺乏可靠的建模代理的情况下揭示内部(即，想象的或尝试的)语音的难以捉摸的表示至关重要。因此，这项研究将同时促进对内心语言神经表征的理解，并反过来促进共适应内部语音解码，以开发实用的闭环系统语音神经假体。德国联邦教育和研究部(BMBF)正在资助一个配套项目。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Recent research has demonstrated that it is possible to synthesize intelligible speech sounds directly from invasive measurements of brain activity. However, these approaches have a perceptible delay between brain activity and audible speech output, preventing a natural spoken communication. Furthermore, the approaches generally require pre-recorded speech and thus cannot be directly applied to people who are unable to speak and generate such recordings. This project aims to develop methods for synthesizing speech from brain activity without perceptible processing delay that do not rely on pre-recorded speech from the user. The ultimate goal is to develop a system that restores natural spoken communication to the millions of people who suffer from severe speech disorders, including those with complete loss of speech. The project is organized into three research thrusts. The first thrust focuses on asynchronous and acoustics-free model training, where novel surrogates to the user's vocalized speech will be created using approaches based on dynamic time warping and the inference of intended inner-speech acoustics from corresponding textual representations. The second thrust focuses on online validation and user adaptation, where the existing low-latency speech decoding and synthesis scheme, which is not inherently adaptable, will be validated in a closed-loop fashion using online human-subject experiments. This will provide valuable insights into how the user responds and adapts to the artificial, synthesized speech output. The third thrust focuses on the development and testing of low-latency system-user co-adaptation schemes. Co-adaptation, where both the user and system adapt to optimize the synthesized output, is crucial for revealing the elusive representations of inner (i.e., imagined or attempted) speech in the absence of a reliable surrogate for modeling. As a result, this research will simultaneously advance the understanding of the neural representations of inner speech and, in turn, co-adaptive inner speech decoding toward the development of practical closed-loop speech neuroprosthetics.A companion project is being funded by the Federal Ministry of Education and Research, Germany (BMBF).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.