Decoding Speech using Invasive Brain-Computer Interfaces based on Intracranial Brain Signals (dSPEECH)

一些患者不能说话是因为在言语产生或运动中招募的神经通路受损或退化，例如运动神经元疾病（MND）或肌萎缩性侧索硬化症（ALS）。然而，如果他们负责语言或认知的大脑结构完好无损，脑机接口（BCI）可能会使他们受益，因为BCI有可能通过直接将大脑信号解码为语音来绕过受损的神经通路。脑机接口可以使用侵入性或非侵入性的方式来记录脑信号。一般来说，非侵入式脑机接口（如脑电图）记录的脑信号质量差，信噪比低，因此目前非侵入式脑机接口无法以可接受的性能解码语音。目前，脑皮层电描记术（ECoG）和立体脑电描记术（SEEG）等侵入性脑机接口技术能够采集到高质量的颅内脑信号，并且具有足够的空间和时间分辨率，因此利用侵入性脑机接口技术对语音进行解码是很有前景的。（ECoG和SEEG）近年来发展迅速，并产生了许多优异的结果，隐蔽（想象的）语音解码仍然具有挑战性。这种情况背后有几个原因。第一个也是主要的原因是因为相关联的神经信号与公开语音相比是弱的和可变的，因此通过机器学习算法解码隐蔽语音是非常困难的。第二个原因是侵入性地记录的有限的想象语音数据集。这种有限的数据集不能使用动物模型来缓解，因为动物使用的通信系统被认为仅限于表达有限数量的话语，而这些话语主要是由基因决定的。此外，对人体的记录通常仅限于非侵入性技术。颅内数据只能在临床环境中从耐药性癫痫或其他神经相关疾病患者中获得。纳入标准，如正常的认知和发音能力，为语音研究进一步减少了潜在的受试者的数量。本项目（dSPEECH）旨在针对上述因素进行突破性的研究。在dSPEECH中，我们雄心勃勃地使用侵入性BCI（ECoG和SEEG）解码隐藏的语音。我们将为新一代脑到文本BCI建立新的范式，开发用于解码隐蔽语音的先进机器学习/深度学习算法，并构建世界上第一个用于隐蔽语音的ECoG/SEEG数据集。我们亦会处理有关道德操守及资料管理方面可能出现的问题。有了这样的ECoG/SEEG数据集和合适的解码方法，我们有信心在解码隐蔽语音的研究上取得重大进展。dSPEECH是一个由多学科成员组成的联合项目，包括来自神经工程的研究人员和来自临床医学的神经外科医生。我们也得到了合作伙伴的大力支持，包括著名的BCI公司和海外经验丰富的神经外科医生。基于紧密和坚实的合作，我们相信dSPEECH可以产生世界领先的结果。

Some patients cannot speak because of impairment or degeneration of neural pathways recruited in speech production or movement such as motor neurone disease (MND) or amyotrophic lateral sclerosis (ALS). However, brain-computer interfaces (BCIs) may benefit them if their brain structure responsible for language or cognition is intact, as BCIs have the potential to bypass damaged neural pathways by decoding brain signals into speech directly. BCIs may use invasive or non-invasive ways for brain signal recording. In general, the brain signals recorded by non-invasive BCIs such as electroencephalography (EEG) is of poor quality with low signal-to-noise ratio, so non-invasive BCIs cannot decode speech with acceptable performance at present. Differently, invasive BCIs such as electrocorticography (ECoG) and stereo-electroencephalography (SEEG) can collect high-quality intracranial brain signals with adequate spatial and temporal resolution, so it is promising to use invasive BCIs to decode speech.Through overt speech using invasive BCIs (ECoG and SEEG) have been developed rapidly and many excellent results are generated in recent years, the covert (imagined) speech decoding is still challenging. There are several reasons behind this situation. The first and major reason is because the associated neural signals are weak and variable compared to overt speech, hence it is very difficult to decode covert speech by machine learning algorithms The second reason is the limited imagined speech dataset recorded invasively. This limited dataset cannot be relieved using an animal model because animals use a system of communication that is believed to be limited to expression of a finite number of utterances that is mostly determined genetically. In addition, recordings on humans are generally restricted to non-invasive techniques. Intracranial data can only be obtained in a clinical environment from patients with drug resistant epilepsy or other neural related conditions. Inclusion criteria, such as normal cognition and the ability to articulate, for speech study further decrease the number of potential subjects. This project (dSPEECH) aims to make some breakthrough regarding the above factors and do a novel study. In dSPEECH, we are ambitious to decode covert speech using invasive BCIs (ECoG and SEEG). We will establish new paradigms for a new generation of brain-to-text BCIs, develop advanced machine learning/deep learning algorithms for decoding covert speech, and construct the world's first ECoG/SEEG dataset for covert speech. We will also tackle the possible problems on ethical issues and data management. With the available of such ECoG/SEEG dataset and the proper decoding methods, we are confident to make big progress on research of decoding covert speech. dSPEECH is a joint project that comprises of multidisciplinary members including researchers from neural engineering and neurosurgeons from clinical medicine. We also have got strong support from partners including famous BCI companies and oversea experienced neurosurgeons. Based on the close and solid collaboration, we believe the world-leading results can be generated in dSPEECH.