Brain Commands and Beyond: Decoding Inner Speech for Neural Prosthetics

全世界有数百万人由于创伤性脑损伤、脑干中风或运动神经元疾病等神经系统疾病而丧失了简单的说话能力。在后一种情况下，丧失语言能力通常被认为是疾病进展的最坏结果。目前的辅助通信技术（比如斯蒂芬·霍金使用的那些）可以提供一些缓解。然而，他们需要残余的运动控制，如脸颊或眼睛的运动。当前的技术还存在令人沮丧的低延迟，用户每分钟只能输出20个单词。相比之下，自然语言以每分钟数百个单词的速度产生。基于所有这些原因，一种新型的语言神经假肢——能够直接从大脑读出（或“解码”）预定的语言——将为社会上一些最孤立的人带来巨大的好处。语音神经假肢的完善也将成为我们理解语音和语言如何在大脑中表现的科学里程碑。第一个语音神经假肢于2021年夏天在一位瘫痪（无关节炎）患者身上实现。像最近的许多工作一样，这项具有里程碑意义的研究使用了植入感觉运动皮层的电极的数据。尽管这些数据有其优势，但除了手术风险和在大脑中安装电子设备之外，它们也有局限性。获得大量的手术数据是非常困难的，这限制了我们利用深度学习的能力。手术数据的另一个重要限制是语言神经修复术专注于解码“内部”语言。与公开语言不同，人们对内在语言的潜在神经生物学知之甚少。它更像是想象的发音（“运动意象”）还是想象的听音（“听觉意象”）？手术数据通常以感觉运动皮层为目标，这对于解码明显清晰的语言是有意义的。但这可能不是解码内心语言的最佳方法。在这里，我们的重点是非侵入性内部语音解码与MRI和脑磁图（MEG）。非侵入性神经影像学至少为外科数据提供了补充见解。因此，该项目的第一个目标是寻求解决有关内心语言本质的问题：我们可以在大脑的哪里解码它？个体之间的内在言语的神经组织不同吗？解码器能在多大程度上从一个人转移到另一个人？回答这些问题将有助于在任何成像模式下设计更好的神经义肢。至于第二个目标，有充分的理由相信，非侵入性方法将为瘫痪患者生产一种可行的、风险更小的言语神经假肢。基于mb的语音理解（即听语音）解码器产生了令人印象深刻的结果。解码内心语音比较困难，但正如我们的试点数据所表明的那样，可以通过大数据和深度学习的结合来克服。因此，该项目旨在获得主题内足够范围（数百小时）的MEG数据集，以表明内部语音解码器原则上可以解决从关键字发现（更容易）到大词汇连续内部语音解码（更难）的一系列任务。我们的目标不仅是为每一项任务产生最先进的结果，而且是为社区制定一套明确的目标来优化。因此，MEG数据将作为机器学习竞赛的一部分发布，灵感来自ImageNet竞赛在过去10年推动计算机视觉领域发展的作用。我们的目标是在内部语音解码方面取得类似的进展。

Millions of people worldwide are deprived of the simple ability to speak because of neurological disorders such as traumatic brain injury, brainstem stroke, or motor neurone disease. In the latter case, the loss of speech is often considered the worst outcome of disease progression. The current state of assistive communication technologies (such as those used by Stephen Hawking) can provide some relief. However, they require residual motor control such as cheek or eye movements. Current technologies also suffer from frustratingly low latencies, with users producing only 20 words per minute. Natural speech, by contrast, is produced at the rate of hundreds of words per minute. For all of these reasons, a new class of speech neuroprosthetic - capable of reading out (or "decoding") intended speech directly from the brain - would provide significant benefits to some of the most isolated people in society. The perfection of speech neuroprosthetics will also represent a scientific milestone in our understanding of how speech and language are represented in the brain. The first speech neuroprosthetic was achieved in a paralysed (anarthric) patient in the summer of 2021. Like much recent work, this landmark study used data from electrodes implanted in the sensorimotor cortex. Although there are advantages to such data, they also have limitations beyond the risk of surgery and installation of electronics into the brain. It is very difficult to obtain large amounts of these surgical data, which limits our ability to leverage the power of deep learning. Another important limitation of surgical data is that speech neuroprosthetics focus on decoding "inner" speech. Unlike overt speech, much less is known about the underlying neurobiology of inner speech. Is it more like imagined articulation ("motor imagery") or imagined audition ("auditory imagery")? Surgical data often targets the sensorimotor cortex, which makes sense for the decoding of overtly articulated speech. But this may be suboptimal for decoding inner speech.Here, we focus on non-invasive inner speech decoding with MRI and magnetoencephalography (MEG). Non-invasive neuroimaging provides, at least, complementary insights to surgical data. The first objective of the project thus seeks to address questions about the nature of inner speech: Where in the brain can we decode it? Does the neural organisation of inner speech differ between individuals? How well can decoders be transferred from one person to another? Answering questions like these will help to design better neuroprosthetics in any imaging modality. Turning to the second objective, there are good reasons to believe that non-invasive methods will produce a viable and less risky speech neuroprosthetic for paralysed patients. MEG-based decoders for speech comprehension (i.e. listening to speech) produce impressive results. Decoding inner speech is harder but - as our pilot data suggests - can be overcome by a combination of big data and deep learning. Thus the project aims to acquire a MEG dataset of sufficient scope (hundreds of hours) within-subject to show that inner speech decoders can, in principle, solve a sequence of tasks from keyword spotting (easier) to large-vocabulary continuous inner speech decoding (harder). The goal is not only to produce state-of-the-art results for each of these tasks, staggered by increasing difficulty and usefulness, but to shape a clear set of objectives for the community to optimise. Thus the MEG data will be released as part of a machine learning competition, inspired by the role that the ImageNet competitions have had in driving the field of computer vision over the past 10 years. We aim to drive similar advances for inner speech decoding.