CRCNS Research Proposal: Novel computational approaches for neural speech prostheses and causal dynamics of language processing

该项目旨在开发计算方法，使人们能够更深入地了解语言的神经生物学，并将其转化为语音和语言的新临床应用。计算研究对于实现未来的语音神经假体是必要的，这将允许患有退行性疾病或神经损伤的患者使用其完整的皮质结构来驱动语音合成器。此外，结合分析个体大脑中连接和语言关键区域的工具，将提供对语言皮质网络动态的洞察，并为取代使用电刺激来绘制语言关键区域的临床实践打开大门，这并不是所有患者都能很好地耐受。这项研究分三个相互交织的方向进行。第一个推力将探索使用深度学习来解码由颅内深度和表面电极捕获的各种神经信号产生的语音。这一推力将在多个患者内部和跨多个患者开发模型，在克服该领域当前限制的同时，有力地解码语音，导致潜在的整合到语音神经假体中。第二个推力将探索从记录的信号中估计大脑连接动力学的有效算法。这一推力将开发新的技术来估计大量记录站点之间的定向连接，这对于理解认知加工过程中跨皮质的动态相互作用至关重要。第三个推力将开发深度学习模型，该模型仅基于神经记录就可以预测对语言处理至关重要的大脑区域。这些模型将用于精确定位对语言处理至关重要的大脑区域，而不需要对大脑进行电刺激。综上所述，这项拟议的研究利用深度学习领域的最新创新(如变压器、图形神经网络、自我监督学习)来克服电极在患者之间的非结构化和多样化放置、数据稀缺性、可扩展性和稳定性带来的挑战。这些新方法将作为开源和可复制的技术与科学界分享。该项目由计算神经科学合作研究(CRCNS)计划和残疾与康复工程计划(DARE)联合资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to develop computational approaches that will allow for a deeper understanding of the neurobiology of language and translate to novel clinical applications for speech and language. The computational research is necessary to enable future speech neuroprostheses, which would allow patients with degenerative conditions or neurological damage to drive a speech synthesizer using their intact cortical structures. Additionally, the combination of tools for analyzing connectivity and regions critical for language in individual brains will provide insight into the network dynamics of language cortex and open the door to replacing the clinical practice of using electrical stimulation to map language-critical regions, which is not well tolerated by all patients.The research is framed across three intertwining thrusts. The first thrust will explore the use of deep learning for decoding produced speech from various neural signals captured by intracranial depth and surface electrodes. This thrust will develop models within and across multiple patients that robustly decode speech while overcoming current limitations in the field, leading to potential integration into speech neuroprostheses. The second thrust will explore efficient algorithms for estimating brain connectivity dynamics from the recorded signals. This thrust will develop novel techniques that estimate directed connectivity among a large number of recording sites, which is essential for understanding the dynamic interactions across cortex during cognitive processing. The third thrust will develop deep-learning models that can predict brain regions that are causally critical for language processing based on the neural recordings alone. These models will serve to pinpoint brain regions that are critical for language processing without needing to electrically stimulate the brain. Taken together, the proposed research leverages recent innovations in deep learning (e.g. transformers, graph neural networks, self-supervised learning) to overcome challenges stemming from the non-structured and varied placements of electrodes across patients, data scarcity, scalability and stability. These new approaches will be shared with the scientific community as open-source and replicable technologies. This project is jointly funded by the Collaborative Research in Computational Neuroscience (CRCNS) program and the Disabilities and Rehabilitation Engineering Program (DARE).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.