EAGER: Collaborative Research: Wireless Sensing of Speech Kinematics and Acoustics for Remediation

语音是一项复杂且时间错综复杂的任务，需要众多肌肉群和生理系统的协调。虽然大多数儿童获得语言是相对容易的，但它是人类完成的最复杂的模式动作之一，因此容易受到损害。大约2%的美国人由于发育过程中的错误学习(发音障碍)或由于中风、脑损伤、帕金森病、脑瘫等神经运动疾病而说话不准确。同样多的美国人在英语发音方面有困难，因为英语是他们的第二语言。这两个用户群体都将受益于就语音产生清晰度提供明确反馈的工具。传统的语音矫正依赖于观看训练有素的临床医生的准确发音和通过镜子进行视觉反馈的重复练习。虽然这些干预措施对于容易看到的语音声音(如/b/p/m/)是有效的，但它们对口腔内产生的声音基本上是不成功的。舌头是这些受阻声音的主要发音机构，它的动作很难捕捉到。因此，临床医生使用图表和其他低技术手段(如将可食用物质放在上颚或物理操作口腔关节)来向客户展示他们的舌头放在哪里。虽然有复杂的研究工具用于测量和跟踪语音过程中的舌头运动，但这些工具昂贵得令人望而却步，令人作呕，而且对于临床和/或家庭使用来说不切实际。在这个代表着两个机构合作的探索性项目中，PI的目标是为一种轻量级、低成本、无线的语言运动学和声学传感器技术(LINKA)奠定基础，这种技术易于在临床和家庭中部署用于语音补救。PI Ghovanloo的实验室开发了一种低成本、无线和可穿戴的磁传感系统，称为舌头驱动系统(TDS)。嵌入耳机内的电磁传感器阵列检测粘在舌头上的小磁铁的位置。临床试验已经证明，通过在口腔内多达6个离散位置感应舌头运动，TDS用于计算机访问和轮椅控制是可行的。这项研究将利用TDS系统的传感能力和Pi Patel在语音障碍患者口语交互技术方面的专业知识，以及Co-PiFu在机器学习和多模式数据融合方面的工作，开发一个临床上可行的原型工具，通过结合语言-运动学和声学数据来提高语音清晰度。为此，该团队将扩展TDS，以跟踪跑步演讲过程中的舌头运动，这些运动速度快，在口腔的一小块区域内紧凑，而且经常为几个音素重叠，因此挑战将是准确地对不同声音类别的运动进行分类。为了补充这一努力，传感器时空动力学的模式识别将被嵌入到交互式游戏中，通过启用对语音修改至关重要的视听生物反馈，为语音运动(Re)学习提供激励的、个性化的环境。为了对该方法的可行性进行基准测试，该系统将在6名神经运动性言语障碍患者和6名年龄匹配的健康对照组身上进行评估。

Speech is a complex and intricately timed task that requires the coordination of numerous muscle groups and physiological systems. While most children acquire speech with relative ease, it is one of the most complex patterned movements accomplished by humans and thus susceptible to impairment. Approximately 2% of Americans have imprecise speech either due to mislearning during development (articulation disorder) or as a result of neuromotor conditions such as stroke, brain injury, Parkinson's disease, cerebral palsy, etc. An equally sizeable group of Americans have difficulty with English pronunciation because it is their second language. Both of these user groups would benefit from tools that provide explicit feedback on speech production clarity. Traditional speech remediation relies on viewing a trained clinician's accurate articulation and repeated practice with visual feedback via a mirror. While these interventions are effective for readily viewable speech sounds (visemes such as /b/p/m/), they are largely unsuccessful for sounds produced inside the mouth. The tongue is the primary articulator for these obstructed sounds and its movements are difficult to capture. Thus, clinicians use diagrams and other low-tech means (such as placing edible substances on the palate or physically manipulating the oral articulators) to show clients where to place their tongue. While sophisticated research tools exist for measuring and tracking tongue movements during speech, they are prohibitively expensive, obtrusive, and impractical for clinical and/or home use. The PIs' goal in this exploratory project, which represents a collaboration across two institutions, is to lay the groundwork for a Lingual-Kinematic and Acoustic sensor technology (LinKa) that is lightweight, low-cost, wireless and easy to deploy both clinically and at home for speech remediation.PI Ghovanloo's lab has developed a low-cost, wireless, and wearable magnetic sensing system, known as the Tongue Drive System (TDS). An array of electromagnetic sensors embedded within a headset detects the position of a small magnet that is adhered to the tongue. Clinical trials have demonstrated the feasibility of using the TDS for computer access and wheelchair control by sensing tongue movements in up to 6 discrete locations within the oral cavity. This research will leverage the sensing capabilities of the TDS system and PI Patel's expertise in spoken interaction technologies for individuals with speech impairment, as well as Co-PI Fu's work on machine learning and multimodal data fusion, to develop a prototype clinically viable tool for enhancing speech clarity by coupling lingual-kinematic and acoustic data. To this end, the team will extend the TDS to track tongue movements during running speech, which are quick, compacted within a small area of the oral cavity, and often overlap for several phonemes, so the challenge will be to accurately classify movements for different sound classes. To complement this effort, pattern recognition of sensor spatiotemporal dynamics will be embedded into an interactive game to offer a motivating, personalized context for speech motor (re)learning by enabling audiovisual biofeedback, which is critical for speech modification. To benchmark the feasibility of the approach, the system will be evaluated on six individuals with neuromotor speech impairment and six healthy age-matched controls.