CRCNS Research Proposal: Novel computational approaches for neural speech prostheses and causal dynamics of language processing

该项目旨在开发计算方法，以便更深入地了解语言的神经生物学，并将其转化为语音和语言的新临床应用。这项计算研究对于实现未来的语音神经假体是必要的，这将允许患有退行性疾病或神经损伤的患者使用他们完整的皮层结构来驱动语音合成器。此外，分析个体大脑中连接和语言关键区域的工具组合将提供对语言皮层网络动态的洞察，并为取代使用电刺激来绘制语言关键区域的临床实践打开大门，这并不是所有患者都能很好地耐受。这项研究有三个相互交织的重点。第一个重点将探索使用深度学习解码由颅内深度和表面电极捕获的各种神经信号产生的语音。这一推动力将在多个患者内部和跨多个患者开发模型，在克服当前该领域的局限性的同时，强大地解码语音，从而潜在地整合到语音神经假肢中。第二个重点将探索从记录的信号中估计大脑连接动态的有效算法。这项研究将开发新的技术来估计大量记录位点之间的直接连接，这对于理解认知过程中皮层之间的动态相互作用至关重要。第三个重点是开发深度学习模型，仅根据神经记录就可以预测对语言处理有因果关系的大脑区域。这些模型将有助于确定大脑中对语言处理至关重要的区域，而不需要电刺激大脑。综上所述，拟议的研究利用了深度学习的最新创新（例如变压器，图神经网络，自监督学习）来克服来自患者电极的非结构化和不同位置，数据稀缺，可扩展性和稳定性的挑战。这些新方法将作为开源和可复制的技术与科学界共享。该项目由计算神经科学合作研究（CRCNS）项目和残疾与康复工程项目（DARE）共同资助。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to develop computational approaches that will allow for a deeper understanding of the neurobiology of language and translate to novel clinical applications for speech and language. The computational research is necessary to enable future speech neuroprostheses, which would allow patients with degenerative conditions or neurological damage to drive a speech synthesizer using their intact cortical structures. Additionally, the combination of tools for analyzing connectivity and regions critical for language in individual brains will provide insight into the network dynamics of language cortex and open the door to replacing the clinical practice of using electrical stimulation to map language-critical regions, which is not well tolerated by all patients.The research is framed across three intertwining thrusts. The first thrust will explore the use of deep learning for decoding produced speech from various neural signals captured by intracranial depth and surface electrodes. This thrust will develop models within and across multiple patients that robustly decode speech while overcoming current limitations in the field, leading to potential integration into speech neuroprostheses. The second thrust will explore efficient algorithms for estimating brain connectivity dynamics from the recorded signals. This thrust will develop novel techniques that estimate directed connectivity among a large number of recording sites, which is essential for understanding the dynamic interactions across cortex during cognitive processing. The third thrust will develop deep-learning models that can predict brain regions that are causally critical for language processing based on the neural recordings alone. These models will serve to pinpoint brain regions that are critical for language processing without needing to electrically stimulate the brain. Taken together, the proposed research leverages recent innovations in deep learning (e.g. transformers, graph neural networks, self-supervised learning) to overcome challenges stemming from the non-structured and varied placements of electrodes across patients, data scarcity, scalability and stability. These new approaches will be shared with the scientific community as open-source and replicable technologies. This project is jointly funded by the Collaborative Research in Computational Neuroscience (CRCNS) program and the Disabilities and Rehabilitation Engineering Program (DARE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.