US-German Data Sharing Proposal: CRCNS Data Sharing: REvealing SPONtaneous Speech Processes in Electrocorticography (RESPONSE)

美德数据共享提案：CRCNS 数据共享：揭示皮层电图记录中的自发言语过程 (RESPONSE)

• 批准号: 1608140
• 负责人: Dean Krusienski
• 金额: $ 55.23万
• 依托单位: Old Dominion University Research Foundation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-08-01 至 2019-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1608140&HistoricalAwards=false
• 关键词: US; German; Data; Sharing; Proposal

The uniquely human capability to produce speech enables swift communication of abstract and substantive information. Currently, nearly two million people in the United States, and far more worldwide, suffer from significant speech production deficits as a result of severe neuromuscular impairments due to injury or disease. In extreme cases, individuals may be unable to speak at all. These individuals would greatly benefit from a device that could alleviate speech deficits and enable them to communicate more naturally and effectively. This project will explore aspects of decoding a user's intended speech directly from the electrical activity of the brain and converting it to synthesized speech that could be played through a loudspeaker in real-time to emulate natural speaking from thought. In particular, this project will uniquely focus on decoding continuous, spontaneous speech processes to achieve more natural and practical communication device for the severely disabled.The complex dynamics of brain activity and the fundamental processing units of continuous speech production and perception are largely unknown, and such dynamics make it challenging to investigate these speech processes with traditional neuroimaging techniques. Electrocorticography (ECoG) measures electrical activity directly from the brain surface and covers an area large enough to provide insights about widespread networks for speech production and understanding, while simultaneously providing localized information for decoding nuanced aspects of the underlying speech processes. Thus, ECoG is instrumental and unparalleled for investigating the detailed spatiotemporal dynamics of speech. The research team's prior work has shown for the first time the detailed spatiotemporal progression of brain activity during prompted continuous speech, and that the team's Brain-to-text system can model phonemes and decode words. However, in pursuit of the ultimate objective of developing a natural speech neuroprosthetic for the severely disabled, research must move beyond studying prompted and isolated aspects of speech. This project will extend the research team's prior experiments to investigate the neural processes of spontaneous and imagined speech production. In conjunction with in-depth analysis of the recorded neural signals, the researchers will apply customized ECoG-based automatic speech recognition (ASR) techniques to facilitate the analysis of the large amount of phones occurring in continuous speech. Ultimately, the project aims to define fundamental units of continuous speech production and understanding, illustrate functional differences between these units, and demonstrate that representations of spontaneous speech can be synthesized directly from the neural recordings. A companion project is being funded by the Federal Ministry of Education and Research, Germany (BMBF)

人类独特的说话能力使抽象和实质性信息的快速交流成为可能。目前，美国有近200万人，而全世界有更多的人，由于受伤或疾病导致的严重神经肌肉损伤而遭受严重的言语产生缺陷。在极端情况下，患者可能根本无法说话。这些人将大大受益于一种可以减轻语言缺陷并使他们能够更自然和有效地沟通的设备。该项目将探索直接从大脑的电活动中解码用户的预期语音，并将其转换为合成语音的各个方面，这些语音可以通过扬声器实时播放，以模仿思想中的自然语音。特别是，这个项目将专注于解码连续的，自发的语音过程，以实现更自然和实用的通信设备，为严重残疾人。大脑活动的复杂动力学和连续语音产生和感知的基本处理单元在很大程度上是未知的，这样的动力学使得用传统的神经成像技术来研究这些语音过程具有挑战性。皮层电图（ECoG）直接从大脑表面测量电活动，覆盖的区域足够大，可以提供有关语音产生和理解的广泛网络的见解，同时提供本地化信息，用于解码潜在语音过程的细微差别。因此，ECoG是工具和无与伦比的调查详细的时空动态的讲话。该研究小组先前的工作首次显示了在提示连续语音期间大脑活动的详细时空进展，并且该团队的Brain-to-text系统可以对音素进行建模并解码单词。然而，在追求为严重残疾人开发自然语音神经假体的最终目标时，研究必须超越对语音提示和孤立方面的研究。该项目将扩展研究团队先前的实验，以研究自发和想象语音产生的神经过程。结合对记录的神经信号的深入分析，研究人员将应用定制的基于ECoG的自动语音识别（ASR）技术，以便于分析连续语音中出现的大量音素。最终，该项目旨在定义连续语音产生和理解的基本单位，说明这些单位之间的功能差异，并证明自发语音的表示可以直接从神经记录中合成。德国联邦教育和研究部（BMBF）正在资助一个配套项目