Brain Commands and Beyond: Decoding Inner Speech for Neural Prosthetics

全世界数以百万计的人因为创伤性脑损伤、脑干中风或运动神经元疾病等神经系统疾病而被剥夺了简单的说话能力。在后一种情况下，言语丧失通常被认为是疾病进展的最坏结果。目前的辅助通信技术(如史蒂芬·霍金使用的技术)可以提供一些缓解。然而，它们需要残留的运动控制，如脸颊或眼睛的运动。目前的技术还存在延迟低得令人沮丧的问题，用户每分钟只产生20个单词。相比之下，自然语言是以每分钟数百个单词的速度产生的。由于所有这些原因，一种新型的言语神经假体--能够直接从大脑读出(或“解码”)预期的言语--将为社会上一些最孤立的人带来显著的好处。语言神经假体的完善也将代表着我们在理解语言和语言在大脑中的表现方式方面的一个科学里程碑。2021年夏天，第一个言语神经假体在一名瘫痪(无关节)患者身上完成。与最近的许多工作一样，这项里程碑式的研究使用了植入感觉运动皮质的电极的数据。尽管这些数据有优势，但它们也有局限性，超出了手术和将电子设备安装到大脑中的风险。要获得大量的手术数据是非常困难的，这限制了我们利用深度学习的能力。外科手术数据的另一个重要限制是，语音神经假体专注于解码“内心”的语音。与外在言语不同，人们对内在言语的潜在神经生物学知之甚少。它更像是想象中的发音(“运动想象”)还是想象中的试听(“听觉想象”)？外科手术数据通常针对感觉运动皮质，这对公开表达的语音的解码是有意义的。但这对于解码内部语音来说可能是次优的。在这里，我们重点研究MRI和脑磁图(MEG)的非侵入性内部语音解码。非侵入性神经成像至少为外科数据提供了补充的见解。因此，该项目的第一个目标是解决关于内心语言本质的问题：我们在大脑的什么地方可以解码它？内心语言的神经组织是否因个体不同而不同？解码员从一个人转移到另一个人的效果如何？回答这样的问题将有助于在任何成像方式下设计更好的神经假体。谈到第二个目标，我们有充分的理由相信，非侵入性方法将为瘫痪患者提供一种可行的、风险更低的语言神经假体。基于脑磁图的语音理解(即听语音)解码器产生了令人印象深刻的结果。破译内心的语言更难，但正如我们的试点数据所表明的那样，大数据和深度学习的结合可以克服这一点。因此，该项目的目标是在受试者内部获得足够范围(数百小时)的脑磁图数据集，以表明内部语音解码器原则上可以解决从关键字识别(更容易)到大词汇量连续内部语音解码(更难)的一系列任务。我们的目标不仅是为这些任务中的每一项产生最先进的结果，因为难度和实用性的增加而交错，而且还形成了一套明确的目标，供社区优化。因此，MEG数据将作为机器学习竞赛的一部分发布，灵感来自ImageNet竞赛在过去10年里推动计算机视觉领域的作用。我们的目标是推动内部语音解码的类似进步。

Millions of people worldwide are deprived of the simple ability to speak because of neurological disorders such as traumatic brain injury, brainstem stroke, or motor neurone disease. In the latter case, the loss of speech is often considered the worst outcome of disease progression. The current state of assistive communication technologies (such as those used by Stephen Hawking) can provide some relief. However, they require residual motor control such as cheek or eye movements. Current technologies also suffer from frustratingly low latencies, with users producing only 20 words per minute. Natural speech, by contrast, is produced at the rate of hundreds of words per minute. For all of these reasons, a new class of speech neuroprosthetic - capable of reading out (or "decoding") intended speech directly from the brain - would provide significant benefits to some of the most isolated people in society. The perfection of speech neuroprosthetics will also represent a scientific milestone in our understanding of how speech and language are represented in the brain. The first speech neuroprosthetic was achieved in a paralysed (anarthric) patient in the summer of 2021. Like much recent work, this landmark study used data from electrodes implanted in the sensorimotor cortex. Although there are advantages to such data, they also have limitations beyond the risk of surgery and installation of electronics into the brain. It is very difficult to obtain large amounts of these surgical data, which limits our ability to leverage the power of deep learning. Another important limitation of surgical data is that speech neuroprosthetics focus on decoding "inner" speech. Unlike overt speech, much less is known about the underlying neurobiology of inner speech. Is it more like imagined articulation ("motor imagery") or imagined audition ("auditory imagery")? Surgical data often targets the sensorimotor cortex, which makes sense for the decoding of overtly articulated speech. But this may be suboptimal for decoding inner speech.Here, we focus on non-invasive inner speech decoding with MRI and magnetoencephalography (MEG). Non-invasive neuroimaging provides, at least, complementary insights to surgical data. The first objective of the project thus seeks to address questions about the nature of inner speech: Where in the brain can we decode it? Does the neural organisation of inner speech differ between individuals? How well can decoders be transferred from one person to another? Answering questions like these will help to design better neuroprosthetics in any imaging modality. Turning to the second objective, there are good reasons to believe that non-invasive methods will produce a viable and less risky speech neuroprosthetic for paralysed patients. MEG-based decoders for speech comprehension (i.e. listening to speech) produce impressive results. Decoding inner speech is harder but - as our pilot data suggests - can be overcome by a combination of big data and deep learning. Thus the project aims to acquire a MEG dataset of sufficient scope (hundreds of hours) within-subject to show that inner speech decoders can, in principle, solve a sequence of tasks from keyword spotting (easier) to large-vocabulary continuous inner speech decoding (harder). The goal is not only to produce state-of-the-art results for each of these tasks, staggered by increasing difficulty and usefulness, but to shape a clear set of objectives for the community to optimise. Thus the MEG data will be released as part of a machine learning competition, inspired by the role that the ImageNet competitions have had in driving the field of computer vision over the past 10 years. We aim to drive similar advances for inner speech decoding.