BrainSign: Recognizing American Sign Language from Brain Signals

美国有近200万人患有严重的运动障碍，使他们无法与外界交流。这些病例中有许多是逐渐残疾的，这为辅助技术提供了希望。尤其是肌萎缩性侧索硬化症（ALS），这是一种进行性肌肉疾病，它会慢慢侵蚀人的运动能力，最终使患者处于完全瘫痪的状态，无法交流。直接脑接口（DBIs）是一种基于神经激活测量的新兴技术，有可能为严重运动障碍患者提供与世界其他地方交流的另一种方式。但是到目前为止，DBI通信系统的最佳性能大约是每分钟68位（每分钟超过8个字符），与说话者和手语者达到的传输速率（每分钟175-200个单词）相比，这就相形见绌了。先前的研究结果表明，想象的运动产生的神经激活与实际的运动相似（尽管程度较小），基于这一结果，pi假设，通过识别来自运动皮层的美国手语（ASL）短语，可以提高DBI的交流率，并且因为完全闭锁的人仍然能够想象运动运动，尽管他们的身体无法实际执行这些运动。一个可以识别想象运动的DBI可能会被闭锁的受试者完全使用。pi设想了一种他们称之为BrainSign的DBI系统，该系统将逐步作为一种替代通信设备，用于被诊断为进行性肌肉疾病（如ALS）早期阶段的患者。在初步诊断后，患者将学会执行有用的手势和手势短语；在疾病发展的早期阶段，BrainSign将了解患者在执行每个手势时表现出的心理活动。随着病情的发展和患者活动能力的丧失，BrainSign会进行调整，以识别运动图像的心理活动，而不是实际的运动，因此最终当患者完全锁定时，BrainSign会识别想象的符号并显示适当的英语翻译，为与护理人员，朋友和家人交流提供了一种有效的方法。这种情况是否真的可以实现尚不清楚，因此这个探索性项目的目标是表征个体不同复杂性的ASL手势可以通过fMRI区分的程度，然后应用这些知识来创建第一个原型便携式系统，从大脑信号中识别ASL。更广泛的影响：这项工作将为dbi奠定基础，这些dbi将提供比以往更高的信息传输速率。这样的系统最终不仅能够帮助被锁住的人，而且还能够帮助许多在行动受限的环境中工作的人（例如，水下研究）和在无法进行声音交流的情况下工作的人。这项研究将进一步促进认知神经科学领域，提供第一个空间共定位，认知正交运动任务的全面研究。pi将通过公共数据库提供他们的数据和结果，以便其他人可以使用他们的算法改进结果。

Nearly two million people in the United States suffer from severe motor disabilities that render them incapable of communicating with the outside world. Many of these cases involve the gradual onset of disability that offers hope for assistive technology. Amyotrophic lateral sclerosis (ALS), in particular, is a progressive muscular disease that slowly erodes a person's ability to produce motor movements, ultimately leaving its victims in a locked-in state where they are completely paralyzed with no ability to communicate. Direct brain interfaces (DBIs) are an emerging technology based on measurement of neural activation that have the potential to provide sufferers of such severe motor disability with an alternative means of communicating with the rest of the world. But to date the best performance for a DBI communication system is about 68 bits per minute (just over 8 characters per minute), which pales in comparison to the transmission rates attained by speakers and signers (175-200 words per minute). Building on the results of previous research that suggests imagined movements produce neural activations similar to executed movements (although of lesser magnitude), the PIs hypothesize that DBI communication rates could be increased by recognizing phrases of American Sign Language (ASL) from the motor cortex, and that because people who are completely locked-in are still capable of imaging motor movements although their body is unable to physically execute the movements, a DBI that recognized imagined motor movements could potentially be fully accessible by locked-in subjects.The PIs envisage a DBI system they have called BrainSign, that would be phased in as an alternative communication device for patients diagnosed in the early stages of a progressive muscular disease such as ALS. Upon initial diagnosis patients would learn to execute useful signs and sign phrases; at this early stage in the disease's progression, BrainSign would learn the mental activity the patient displays while executing each sign. As the disease progresses and the patient loses mobility, BrainSign would adjust to recognize the mental activity for motor imagery rather than actual motor movement, so that eventually when the patient is completely locked-in BrainSign would recognize the imagined sign and display the appropriate English translation, providing an efficient method for communicating with caregivers, friends, and family. Whether this scenario can actually be achieved is unclear, hence this exploratory project whose objectives are to characterize the extent to which individual ASL gestures of varying complexity can be discriminated by means of fMRI, and then to apply this knowledge to create the first prototype portable system that recognizes ASL from brain signals.Broader Impacts: This work will lay the foundations for DBIs that provide much higher information transmission rates than has heretofore been achievable. Such systems will ultimately be able to assist not only locked-in people, but also many others who work in mobility-restricted environments (e.g., underwater research) and in situations where vocal communication is not possible. The research will furthermore contribute to the field of cognitive neuroscience, by providing the first comprehensive study of spatially co-located, cognitively orthogonal motor tasks. The PIs will make their data and results available via a public database, so that others can improve on the results using their algorithms.