BrainSign: Recognizing American Sign Language from Brain Signals

近200万美国人患有严重的运动障碍，使他们无法与外部世界沟通。其中许多病例涉及逐渐出现的残疾，这为辅助技术带来了希望。尤其是肌萎缩侧索硬化症(ALS)，是一种进行性肌肉疾病，会慢慢侵蚀人的运动能力，最终使患者处于锁定状态，完全瘫痪，没有沟通能力。直接脑接口(DBIS)是一项基于神经激活测量的新兴技术，有可能为严重运动障碍患者提供一种与世界其他地区交流的替代手段。但到目前为止，DBI通信系统的最佳性能约为每分钟68比特(略高于每分钟8个字符)，与说话者和签名者所达到的传输速率(每分钟175-200个字)相比相形见绌。基于先前的研究结果表明，想象的动作产生与执行的动作相似的神经激活(尽管幅度较小)，PI假设通过从运动皮质识别美国手语(ASL)的短语可以提高DBI的交流速度，并且由于完全被锁定的人仍然能够想象运动动作，尽管他们的身体无法实际执行这些动作，识别想象的运动的DBI可能潜在地被锁定的受试者完全访问。PI设想了一个他们称为BrainSign的DBI系统，这将逐步成为渐进性肌肉疾病(如ALS)早期诊断患者的替代通信设备。在初步诊断时，患者将学习执行有用的体征和手势短语；在疾病进展的早期阶段，BrainSign将学习患者在执行每个体征时所表现出的精神活动。随着疾病的发展和患者失去行动能力，BrainSign将进行调整，以识别运动想象的心理活动，而不是实际的运动运动，因此最终当患者完全锁定时，BrainSign将识别想象的迹象并显示适当的英语翻译，从而提供一种与照顾者、朋友和家人沟通的有效方法。目前尚不清楚这一场景是否能真正实现，因此这个探索性项目的目标是表征通过功能磁共振成像可以区分不同复杂性的单个ASL手势的程度，然后应用这一知识创建第一个从脑信号中识别ASL的便携式原型系统。广泛的影响：这项工作将为DBIS奠定基础，提供比迄今可实现的更高的信息传输速率。这样的系统最终将不仅能够帮助被锁住的人，而且还能够帮助在行动受限的环境(例如水下研究)和在无法进行语音通信的情况下工作的许多其他人。这项研究将进一步促进认知神经科学领域，通过提供第一个空间共同定位的认知正交运动任务的综合研究。PI将通过公共数据库提供他们的数据和结果，这样其他人就可以使用他们的算法改进结果。

Nearly two million people in the United States suffer from severe motor disabilities that render them incapable of communicating with the outside world. Many of these cases involve the gradual onset of disability that offers hope for assistive technology. Amyotrophic lateral sclerosis (ALS), in particular, is a progressive muscular disease that slowly erodes a person's ability to produce motor movements, ultimately leaving its victims in a locked-in state where they are completely paralyzed with no ability to communicate. Direct brain interfaces (DBIs) are an emerging technology based on measurement of neural activation that have the potential to provide sufferers of such severe motor disability with an alternative means of communicating with the rest of the world. But to date the best performance for a DBI communication system is about 68 bits per minute (just over 8 characters per minute), which pales in comparison to the transmission rates attained by speakers and signers (175-200 words per minute). Building on the results of previous research that suggests imagined movements produce neural activations similar to executed movements (although of lesser magnitude), the PIs hypothesize that DBI communication rates could be increased by recognizing phrases of American Sign Language (ASL) from the motor cortex, and that because people who are completely locked-in are still capable of imaging motor movements although their body is unable to physically execute the movements, a DBI that recognized imagined motor movements could potentially be fully accessible by locked-in subjects.The PIs envisage a DBI system they have called BrainSign, that would be phased in as an alternative communication device for patients diagnosed in the early stages of a progressive muscular disease such as ALS. Upon initial diagnosis patients would learn to execute useful signs and sign phrases; at this early stage in the disease's progression, BrainSign would learn the mental activity the patient displays while executing each sign. As the disease progresses and the patient loses mobility, BrainSign would adjust to recognize the mental activity for motor imagery rather than actual motor movement, so that eventually when the patient is completely locked-in BrainSign would recognize the imagined sign and display the appropriate English translation, providing an efficient method for communicating with caregivers, friends, and family. Whether this scenario can actually be achieved is unclear, hence this exploratory project whose objectives are to characterize the extent to which individual ASL gestures of varying complexity can be discriminated by means of fMRI, and then to apply this knowledge to create the first prototype portable system that recognizes ASL from brain signals.Broader Impacts: This work will lay the foundations for DBIs that provide much higher information transmission rates than has heretofore been achievable. Such systems will ultimately be able to assist not only locked-in people, but also many others who work in mobility-restricted environments (e.g., underwater research) and in situations where vocal communication is not possible. The research will furthermore contribute to the field of cognitive neuroscience, by providing the first comprehensive study of spatially co-located, cognitively orthogonal motor tasks. The PIs will make their data and results available via a public database, so that others can improve on the results using their algorithms.