US-German Data Sharing Proposal: CRCNS Data Sharing: REvealing SPONtaneous Speech Processes in Electrocorticography (RESPONSE)

人类独特的说话能力使抽象和实质性信息的快速交流成为可能。目前，由于受伤或疾病导致的严重神经肌肉损伤，美国有近200万人患有严重的语言产生障碍，全世界的人数要多得多。在极端情况下，患者可能根本无法说话。这些人将极大地受益于一种可以减轻语言缺陷的设备，使他们能够更自然、更有效地交流。这个项目将探索直接从大脑的电活动中解码用户的预期语音，并将其转换为可以通过扬声器实时播放的合成语音，以模仿自然的思维。特别是本项目将独特地专注于解码连续的、自发的语音过程，为重度残疾人实现更自然、更实用的交流设备。大脑活动的复杂动态以及连续语音产生和感知的基本处理单元在很大程度上是未知的，这些动态使得用传统的神经成像技术研究这些语音过程具有挑战性。脑皮层电图（ECoG）直接测量大脑表面的电活动，覆盖的区域足够大，可以为语音产生和理解的广泛网络提供见解，同时为解码潜在语音过程的细微方面提供本地化信息。因此，ECoG在研究语言的详细时空动态方面是工具性的和无与伦比的。该研究小组之前的工作首次展示了在提示连续讲话期间大脑活动的详细时空进展，并且该团队的脑-文本系统可以模拟音素和解码单词。然而，为了实现为严重残疾人开发自然语言神经假肢的最终目标，研究必须超越对语言提示和孤立方面的研究。该项目将扩展研究小组先前的实验，以研究自发和想象语音产生的神经过程。结合对记录的神经信号的深入分析，研究人员将应用定制的基于ecog的自动语音识别（ASR）技术，以促进对连续语音中出现的大量电话的分析。最终，该项目旨在定义连续语音产生和理解的基本单元，说明这些单元之间的功能差异，并证明自发语音的表征可以直接从神经记录中合成。一个伙伴项目由德国联邦教育和研究部（BMBF）资助。

The uniquely human capability to produce speech enables swift communication of abstract and substantive information. Currently, nearly two million people in the United States, and far more worldwide, suffer from significant speech production deficits as a result of severe neuromuscular impairments due to injury or disease. In extreme cases, individuals may be unable to speak at all. These individuals would greatly benefit from a device that could alleviate speech deficits and enable them to communicate more naturally and effectively. This project will explore aspects of decoding a user's intended speech directly from the electrical activity of the brain and converting it to synthesized speech that could be played through a loudspeaker in real-time to emulate natural speaking from thought. In particular, this project will uniquely focus on decoding continuous, spontaneous speech processes to achieve more natural and practical communication device for the severely disabled.The complex dynamics of brain activity and the fundamental processing units of continuous speech production and perception are largely unknown, and such dynamics make it challenging to investigate these speech processes with traditional neuroimaging techniques. Electrocorticography (ECoG) measures electrical activity directly from the brain surface and covers an area large enough to provide insights about widespread networks for speech production and understanding, while simultaneously providing localized information for decoding nuanced aspects of the underlying speech processes. Thus, ECoG is instrumental and unparalleled for investigating the detailed spatiotemporal dynamics of speech. The research team's prior work has shown for the first time the detailed spatiotemporal progression of brain activity during prompted continuous speech, and that the team's Brain-to-text system can model phonemes and decode words. However, in pursuit of the ultimate objective of developing a natural speech neuroprosthetic for the severely disabled, research must move beyond studying prompted and isolated aspects of speech. This project will extend the research team's prior experiments to investigate the neural processes of spontaneous and imagined speech production. In conjunction with in-depth analysis of the recorded neural signals, the researchers will apply customized ECoG-based automatic speech recognition (ASR) techniques to facilitate the analysis of the large amount of phones occurring in continuous speech. Ultimately, the project aims to define fundamental units of continuous speech production and understanding, illustrate functional differences between these units, and demonstrate that representations of spontaneous speech can be synthesized directly from the neural recordings. A companion project is being funded by the Federal Ministry of Education and Research, Germany (BMBF)