EAGER: Collaborative Research: Wireless Sensing of Speech Kinematics and Acoustics for Remediation

言语是一项复杂且时间复杂的任务，需要协调众多肌肉群和生理系统。 虽然大多数儿童获得语言相对容易，但它是人类完成的最复杂的模式化运动之一，因此容易受到损害。 大约有2%的美国人有不精确的讲话，无论是由于在发展过程中的错误学习（发音障碍）或由于神经运动条件，如中风，脑损伤，帕金森氏病，脑瘫等一个同样可观的一组美国人有困难的英语发音，因为它是他们的第二语言。 这两个用户群体都将受益于提供语音清晰度明确反馈的工具。 传统的言语矫正依赖于通过镜子观察受过训练的临床医生的准确发音和重复练习以及视觉反馈。 虽然这些干预措施对容易看到的语音（视位，如/B/p/m/）是有效的，但它们对口腔内产生的声音基本上不成功。 舌头是这些受阻声音的主要发音器官，它的运动很难捕捉。 因此，临床医生使用图表和其他低科技手段（如将可食用物质放在上颚或物理操纵口腔发音器官）来向客户展示舌头的位置。 虽然存在用于测量和跟踪讲话期间的舌头运动的复杂研究工具，但它们对于临床和/或家庭使用来说过于昂贵、突兀和不切实际。 PI在这个探索性项目中的目标是为语言运动和声学传感器技术（LinKa）奠定基础，该技术重量轻，成本低，无线，易于在临床和家庭中部署，用于语音修复。PI Ghovanloo的实验室开发了一种低成本，无线和可穿戴的磁传感系统，称为舌头驱动系统（TDS）。 耳机内嵌入的电磁传感器阵列检测粘附在舌头上的小磁铁的位置。 临床试验已经证明了使用TDS进行计算机访问和轮椅控制的可行性，通过感测口腔内多达6个离散位置的舌头运动。 这项研究将利用TDS系统的传感能力和PI Patel在语音障碍个体语音交互技术方面的专业知识，以及Co-PI Fu在机器学习和多模态数据融合方面的工作，开发一种原型临床可行的工具，通过耦合语言运动学和声学数据来增强语音清晰度。 为此，该团队将扩展TDS以跟踪语音运行期间的舌头运动，这些运动在口腔的小区域内快速，紧凑，并且经常重叠几个音素，因此挑战将是准确地对不同声音类别的运动进行分类。 为了补充这一努力，传感器时空动态的模式识别将被嵌入到一个互动游戏中，通过启用视听生物反馈，这是语音修改的关键，为语音运动（再）学习提供一个激励，个性化的背景。 为了对该方法的可行性进行基准测试，该系统将在六名神经运动性语言障碍患者和六名健康年龄匹配的对照组中进行评估。

Speech is a complex and intricately timed task that requires the coordination of numerous muscle groups and physiological systems. While most children acquire speech with relative ease, it is one of the most complex patterned movements accomplished by humans and thus susceptible to impairment. Approximately 2% of Americans have imprecise speech either due to mislearning during development (articulation disorder) or as a result of neuromotor conditions such as stroke, brain injury, Parkinson's disease, cerebral palsy, etc. An equally sizeable group of Americans have difficulty with English pronunciation because it is their second language. Both of these user groups would benefit from tools that provide explicit feedback on speech production clarity. Traditional speech remediation relies on viewing a trained clinician's accurate articulation and repeated practice with visual feedback via a mirror. While these interventions are effective for readily viewable speech sounds (visemes such as /b/p/m/), they are largely unsuccessful for sounds produced inside the mouth. The tongue is the primary articulator for these obstructed sounds and its movements are difficult to capture. Thus, clinicians use diagrams and other low-tech means (such as placing edible substances on the palate or physically manipulating the oral articulators) to show clients where to place their tongue. While sophisticated research tools exist for measuring and tracking tongue movements during speech, they are prohibitively expensive, obtrusive, and impractical for clinical and/or home use. The PIs' goal in this exploratory project, which represents a collaboration across two institutions, is to lay the groundwork for a Lingual-Kinematic and Acoustic sensor technology (LinKa) that is lightweight, low-cost, wireless and easy to deploy both clinically and at home for speech remediation.PI Ghovanloo's lab has developed a low-cost, wireless, and wearable magnetic sensing system, known as the Tongue Drive System (TDS). An array of electromagnetic sensors embedded within a headset detects the position of a small magnet that is adhered to the tongue. Clinical trials have demonstrated the feasibility of using the TDS for computer access and wheelchair control by sensing tongue movements in up to 6 discrete locations within the oral cavity. This research will leverage the sensing capabilities of the TDS system and PI Patel's expertise in spoken interaction technologies for individuals with speech impairment, as well as Co-PI Fu's work on machine learning and multimodal data fusion, to develop a prototype clinically viable tool for enhancing speech clarity by coupling lingual-kinematic and acoustic data. To this end, the team will extend the TDS to track tongue movements during running speech, which are quick, compacted within a small area of the oral cavity, and often overlap for several phonemes, so the challenge will be to accurately classify movements for different sound classes. To complement this effort, pattern recognition of sensor spatiotemporal dynamics will be embedded into an interactive game to offer a motivating, personalized context for speech motor (re)learning by enabling audiovisual biofeedback, which is critical for speech modification. To benchmark the feasibility of the approach, the system will be evaluated on six individuals with neuromotor speech impairment and six healthy age-matched controls.