Decoding inner speech: An AI approach to transcribing thoughts via EEG & EMG

解码内心言语：一种通过脑电图转录思想的人工智能方法

• 批准号: 10058047
• 负责人: Jose A CORTES-BRIONES
• 金额: $ 52.36万
• 依托单位: YALE UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-15 至 2024-09-14
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10058047
• 关键词: ALS2 gene; Algorithms; American; Architecture; Area; Articulation; Artificial Intelligence; Attention; Brain; Clinical; Communication; Complex; Data; Data Collection; Data Set; Development; Electroencephalography; Electromyography; Electrophysiology (science); Expert Systems; Fostering; Gender; Genetic Transcription; Hand; Impairment; Individual; Language; Lead; Learning; Letters; Linguistics; Location; Machine Learning; Maps; Measures; Mental Health; Mental disorders; Methods; Modeling; Muscle; Occupations; Output; Patients; Pattern; Performance; Persons; Psyche structure; Psychiatry; Quality of life; Reading; Signal Transduction; Social Interaction; Speech; Stimulus; Sum; System; Technology; Text; Thinking; Time; Training; Translating; Vocabulary; Voice; Writing; algorithm training; blind; computerized data processing; deep learning; deep neural network; design; digital; healthy volunteer; improved; innovation; large datasets; machine learning algorithm; multidisciplinary; performance tests; sound; speech accuracy

ABSTRACT Losing the capacity to communicate through language has a significant negative impact on a person’s autonomy, social interactions, occupation, mental health, and overall quality of life. Many people lose the capacity to speak and write but keep their thinking intact. Inner speech is internally and willfully generated, non-articulated verbal thoughts (e.g., reading in silence). Changes in the activation patterns of the brain’s language-related areas co-occur with inner speech and can be detected with electroencephalography (EEG). Furthermore, while inner speech doesn’t lead to any discernible voice sound or articulation, co-occurring low amplitude electrical discharges in the articulatory muscles can be detected with electromyography (EMG). The information about ongoing inner speech reflected in electrophysiological signals (EEG and EMG) can be used to transcribe inner speech into text or voice. Machine learning algorithms have been used for this purpose, however, the resulting systems have low accuracy and/or are constrained by very small vocabularies (~10 words). Furthermore, these systems need to be trained anew for each user, which significantly increases individual data-collection time. The development of ready-to-use/minimal-training (fine tuning) systems requires large training datasets that algorithms can use to learn high-level features capable of being transferred between individuals. Unfortunately, to date there are no available datasets that are large enough to train these systems. To tackle these issues, I have assembled a multidisciplinary team of collaborators from Google AI, Yale linguistics, and Yale Psychiatry to develop a state-of-the-art deep neural network to transcribe inner speech to text using EEG and EMG signals. This system will incorporate some of the latest advances in artificial intelligence and data processing developed by Google AI. It will be designed to transcribe phonemes, thus, in principle, will be able to transcribe any word. Furthermore, we will collect the largest (x120 times) multi-subject (n=150) electrophysiological (EEG+EMG) inner speech dataset to date (300 hrs. in total) to train the first ready- to-use/minimal-training inner speech transcriber system. The technology resulting from this study has the potential to radically improve the quality of life of thousands of patients by providing them with a fast method of communicating their verbal thoughts. Furthermore, by combining this system with one of the many text-to-speech AIs that are currently available, our system could potentially restore the patients’ capacity to produce conversational speech.

摘要