BrainSign: Recognizing American Sign Language from Brain Signals

BrainSign：从大脑信号识别美国手语

• 批准号: 0836747
• 负责人: Thad Starner
• 金额: $ 3.5万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2008
• 资助国家: 美国
• 起止时间: 2008-08-01 至 2009-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0836747&HistoricalAwards=false
• 关键词: BrainSign; Recognizing; American; Sign; Language

Nearly two million people in the United States suffer from severe motor disabilities that render them incapable of communicating with the outside world. Many of these cases involve the gradual onset of disability that offers hope for assistive technology. Amyotrophic lateral sclerosis (ALS), in particular, is a progressive muscular disease that slowly erodes a person's ability to produce motor movements, ultimately leaving its victims in a locked-in state where they are completely paralyzed with no ability to communicate. Direct brain interfaces (DBIs) are an emerging technology based on measurement of neural activation that have the potential to provide sufferers of such severe motor disability with an alternative means of communicating with the rest of the world. But to date the best performance for a DBI communication system is about 68 bits per minute (just over 8 characters per minute), which pales in comparison to the transmission rates attained by speakers and signers (175-200 words per minute). Building on the results of previous research that suggests imagined movements produce neural activations similar to executed movements (although of lesser magnitude), the PIs hypothesize that DBI communication rates could be increased by recognizing phrases of American Sign Language (ASL) from the motor cortex, and that because people who are completely locked-in are still capable of imaging motor movements although their body is unable to physically execute the movements, a DBI that recognized imagined motor movements could potentially be fully accessible by locked-in subjects.The PIs envisage a DBI system they have called BrainSign, that would be phased in as an alternative communication device for patients diagnosed in the early stages of a progressive muscular disease such as ALS. Upon initial diagnosis patients would learn to execute useful signs and sign phrases; at this early stage in the disease's progression, BrainSign would learn the mental activity the patient displays while executing each sign. As the disease progresses and the patient loses mobility, BrainSign would adjust to recognize the mental activity for motor imagery rather than actual motor movement, so that eventually when the patient is completely locked-in BrainSign would recognize the imagined sign and display the appropriate English translation, providing an efficient method for communicating with caregivers, friends, and family. Whether this scenario can actually be achieved is unclear, hence this exploratory project whose objectives are to characterize the extent to which individual ASL gestures of varying complexity can be discriminated by means of fMRI, and then to apply this knowledge to create the first prototype portable system that recognizes ASL from brain signals.Broader Impacts: This work will lay the foundations for DBIs that provide much higher information transmission rates than has heretofore been achievable. Such systems will ultimately be able to assist not only locked-in people, but also many others who work in mobility-restricted environments (e.g., underwater research) and in situations where vocal communication is not possible. The research will furthermore contribute to the field of cognitive neuroscience, by providing the first comprehensive study of spatially co-located, cognitively orthogonal motor tasks. The PIs will make their data and results available via a public database, so that others can improve on the results using their algorithms.

美国有近 200 万人患有严重的运动障碍，导致他们无法与外界沟通。 其中许多案例涉及逐渐出现的残疾，这为辅助技术带来了希望。 尤其是肌萎缩侧索硬化症（ALS），是一种进行性肌肉疾病，会慢慢侵蚀人的运动能力，最终使患者处于锁定状态，完全瘫痪，无法沟通。 直接大脑接口（DBI）是一种基于神经激活测量的新兴技术，有可能为严重运动障碍患者提供与世界其他地方沟通的替代方式。 但迄今为止，DBI 通信系统的最佳性能约为每分钟 68 位（每分钟仅超过 8 个字符），与说话者和签名者所达到的传输速率（每分钟 175-200 个字）相比显得相形见绌。 先前的研究结果表明，想象的运动会产生与执行的运动类似的神经激活（尽管幅度较小），PI 假设，通过识别运动皮层的美国手语 (ASL) 短语，可以提高 DBI 沟通率，并且由于完全锁定的人仍然能够想象运动运动，尽管他们的身体无法实际执行运动，识别想象的运动运动的 DBI 可以 PI 设想了一种被他们称为 BrainSign 的 DBI 系统，该系统将分阶段作为一种替代通信设备，供诊断为 ALS 等进行性肌肉疾病早期阶段的患者使用。 初步诊断后，患者将学会执行有用的手势和手势短语；在疾病进展的早期阶段，BrainSign 将了解患者在执行每个手势时表现出的心理活动。随着疾病的进展和患者失去活动能力，BrainSign 会进行调整，以识别运动想象的心理活动，而不是实际的运动运动，这样最终当患者完全锁定时，BrainSign 会识别想象的符号并显示适当的英语翻译，为与护理人员、朋友和家人沟通提供有效的方法。 这种情况是否能够真正实现尚不清楚，因此这个探索性项目的目标是通过功能磁共振成像来表征不同复杂程度的单个 ASL 手势的识别程度，然后应用这些知识来创建第一个从大脑信号中识别 ASL 的原型便携式系统。 更广泛的影响：这项工作将为 DBI 奠定基础，提供比迄今为止可实现的更高的信息传输速率。 此类系统最终不仅能够帮助被困的人，还能帮助许多在行动受限的环境（例如水下研究）和无法进行声音交流的情况下工作的其他人。 该研究将首次对空间共置、认知正交运动任务进行全面研究，从而进一步为认知神经科学领域做出贡献。 PI 将通过公共数据库提供他们的数据和结果，以便其他人可以使用他们的算法改进结果。