CRCNS Research Proposal: Understanding Cortical Networks Related to Speech Using Deep Learning on ECOG Data

CRCNS 研究提案：利用 ECOG 数据的深度学习了解与语音相关的皮层网络

• 批准号: 1912286
• 负责人: Yao Wang
• 金额: $ 83.26万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1912286&HistoricalAwards=false
• 关键词: CRCNS; Research; Proposal; Understanding; Cortical

Despite significant advances in neural science, the dynamics by which neural activity propagates across cortex while we think of a word and produce it remains poorly understood. This proposal will develop novel, data-driven approaches for understanding functions and interactions of various brain regions by leveraging rare neural recordings obtained with electrocorticography (ECoG) sensors while neurosurgical patients participate in tasks involving language perception, semantic access and word production. This project will produce a set of validated novel computational tools for estimating neural representations and their dynamics as well as elucidate the cortical networks subserving perception, semantic access, and production of speech. Although these tools will be developed for ECoG data, the proposed frameworks are applicable to other neural data modalities including fMRI and EEG, and thus have broad applications in neuroscience. The ability to robustly translate between speech and its neural representations is vital to the development of speech prosthetics, which would allow patients with degenerative conditions (Amyotrophic Lateral Sclerosis) or neurological damage (locked-in syndrome) to drive a speech synthesizer via control from intact cortical structures. The network connectivity tools could shed light on the propagation dynamics of epileptic seizures as well as on how cortical communication, when impaired, gives rise to language aphasias and disconnection syndromes. Furthermore, the decoding and network connectivity tools could help develop novel language mapping approaches for brain surgery without the associated risks of electrical stimulation mapping.The project consists of three core thrusts: developing neural decoders for language processing, developing directed connectivity models, and experimental validation. The neural decoders will be based on deep-learning architectures able to learn a transformation between neural signals and the speech heard by the patient, the speech produced by the patient, or the semantic concept represented by the stimulus word. The connectivity models will generalize and coalesce current approaches for estimating the task-dependent, time-varying directed connectivity between cortical regions. Lastly, these findings will be experimentally validated via clinical electrical stimulation data and cortico-cortico evoked potential (CCEP) stimulation experiments. Current modeling approaches of ECoG data have mostly focused on variants of linear models and on speech acoustics. This project will harness the potential of highly non-linear and deep networks for modeling neural responses to both speech acoustics and access to semantics. Additionally, tools for inferring direct connectivity and interactions among neural regions will provide a detailed characterization of the network dynamics, which is largely overlooked by most ECoG decoding studies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管神经科学取得了重大进展，但当我们想到一个词并产生它时，神经活动在大脑皮层中传播的动力学仍然知之甚少。当神经外科患者参与涉及语言感知、语义获取和单词生成的任务时，该提案将开发新颖的、数据驱动的方法，通过利用皮质电图（ECoG）传感器获得的罕见神经记录，来理解各种大脑区域的功能和相互作用。该项目将产生一套经过验证的新型计算工具，用于估计神经表征及其动态，并阐明支持感知、语义访问和语音产生的皮质网络。虽然这些工具将为ECoG数据开发，但提议的框架适用于其他神经数据模式，包括fMRI和EEG，因此在神经科学中具有广泛的应用。语音与其神经表征之间的强大翻译能力对于语音假肢的发展至关重要，这将允许患有退行性疾病（肌萎缩侧索硬化症）或神经损伤（闭锁综合征）的患者通过控制完整的皮质结构来驱动语音合成器。网络连接工具可以揭示癫痫发作的传播动态，以及皮层通信受损时如何导致语言失语和断开连接综合征。此外，解码和网络连接工具可以帮助开发新的脑外科语言映射方法，而不会带来电刺激映射的相关风险。该项目包括三个核心重点：开发用于语言处理的神经解码器、开发定向连接模型和实验验证。神经解码器将基于深度学习架构，能够学习神经信号与患者听到的语音、患者产生的语音或刺激词所代表的语义概念之间的转换。连接模型将概括和整合目前的方法来估计皮层区域之间依赖于任务的、时变的定向连接。最后，这些发现将通过临床电刺激数据和皮质诱发电位（CCEP）刺激实验进行实验验证。目前ECoG数据的建模方法主要集中在线性模型的变体和语音声学上。该项目将利用高度非线性和深度网络的潜力，对语音声学和语义访问的神经反应进行建模。此外，用于推断神经区域之间直接连接和相互作用的工具将提供网络动态的详细特征，这在很大程度上被大多数ECoG解码研究所忽视。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Stimulus Speech Decoding from Human Cortex with Generative Adversarial Network Transfer Learning

使用生成对抗网络迁移学习从人类皮层进行刺激语音解码

• DOI: 10.1109/isbi45749.2020.9098589
• 发表时间: 2020
• 期刊: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI
• 作者: Wang, Ran;Chen, Xupeng;Khalilian-Gourtani, Amirhossein;Chen, Zhaoxi;Yu, Leyao;Flinker, Adeen;Wang, Yao
• 通讯作者: Wang, Yao

Distinct prefrontal networks for semantic integration and articulatory planning

用于语义整合和发音规划的独特前额叶网络

• DOI: 10.32470/ccn.2022.1224-0
• 发表时间: 2022
• 期刊: Conference on Cognitive Computational Neuroscience
• 作者: Yu, Leyao;Chapochnikov, Nikolai;Flinker, Adeen
• 通讯作者: Flinker, Adeen