EAGER: Collaborative Research: Wireless Sensing of Speech Kinematics and Acoustics for Remediation

说话是一项复杂而复杂的时间任务，需要许多肌肉群和生理系统的协调。虽然大多数儿童相对容易掌握语言，但它是人类完成的最复杂的模式运动之一，因此容易受到损害。大约2%的美国人有不精确的语言，要么是由于发育过程中的错误学习（发音障碍），要么是由于神经运动疾病，如中风、脑损伤、帕金森病、脑瘫等。同样数量可观的美国人在英语发音上有困难，因为英语是他们的第二语言。这两个用户组都将受益于提供语音清晰度明确反馈的工具。传统的言语矫正依赖于观察训练有素的临床医生的准确发音，并通过镜子通过视觉反馈反复练习。虽然这些干预措施对容易看到的语音（如/b/p/m/等音）有效，但对嘴里发出的声音基本上是不成功的。舌头是这些发音障碍的主要发声器，它的动作很难捕捉。因此，临床医生使用图表和其他低技术手段（如在上颚放置可食用物质或物理操作口腔发音器）来告诉来访者舌头的放置位置。虽然有精密的研究工具来测量和追踪说话过程中的舌头运动，但它们过于昂贵、突兀，而且不适合临床和/或家庭使用。pi在这个探索性项目中的目标是为语言运动学和声学传感器技术（LinKa）奠定基础，该技术重量轻，成本低，无线，易于临床和家庭部署，用于语音修复。PI Ghovanloo的实验室开发了一种低成本、无线、可穿戴的磁传感系统，称为舌驱动系统（TDS）。一组嵌在头戴式耳机内的电磁传感器可以检测附着在舌头上的小磁铁的位置。临床试验表明，通过在口腔内多达6个离散位置感应舌头运动，TDS可以用于电脑访问和轮椅控制。这项研究将利用TDS系统的传感能力和PI Patel在语音障碍患者语音交互技术方面的专业知识，以及Co-PI Fu在机器学习和多模态数据融合方面的工作，开发一种临床可行的原型工具，通过耦合语言运动学和声学数据来提高语音清晰度。为此，该团队将扩展TDS，以跟踪运行语音过程中的舌头运动，这些运动是快速的，压缩在口腔的一小块区域内，并且经常重叠几个音素，因此挑战将是准确分类不同声音类别的运动。为了补充这一努力，传感器时空动态的模式识别将被嵌入到一个互动游戏中，通过实现视听生物反馈，为语音运动（再）学习提供一个激励的、个性化的环境，这对语音修改至关重要。为了测试该方法的可行性，该系统将在6名神经运动语言障碍患者和6名年龄匹配的健康对照者身上进行评估。

Speech is a complex and intricately timed task that requires the coordination of numerous muscle groups and physiological systems. While most children acquire speech with relative ease, it is one of the most complex patterned movements accomplished by humans and thus susceptible to impairment. Approximately 2% of Americans have imprecise speech either due to mislearning during development (articulation disorder) or as a result of neuromotor conditions such as stroke, brain injury, Parkinson's disease, cerebral palsy, etc. An equally sizeable group of Americans have difficulty with English pronunciation because it is their second language. Both of these user groups would benefit from tools that provide explicit feedback on speech production clarity. Traditional speech remediation relies on viewing a trained clinician's accurate articulation and repeated practice with visual feedback via a mirror. While these interventions are effective for readily viewable speech sounds (visemes such as /b/p/m/), they are largely unsuccessful for sounds produced inside the mouth. The tongue is the primary articulator for these obstructed sounds and its movements are difficult to capture. Thus, clinicians use diagrams and other low-tech means (such as placing edible substances on the palate or physically manipulating the oral articulators) to show clients where to place their tongue. While sophisticated research tools exist for measuring and tracking tongue movements during speech, they are prohibitively expensive, obtrusive, and impractical for clinical and/or home use. The PIs' goal in this exploratory project, which represents a collaboration across two institutions, is to lay the groundwork for a Lingual-Kinematic and Acoustic sensor technology (LinKa) that is lightweight, low-cost, wireless and easy to deploy both clinically and at home for speech remediation.PI Ghovanloo's lab has developed a low-cost, wireless, and wearable magnetic sensing system, known as the Tongue Drive System (TDS). An array of electromagnetic sensors embedded within a headset detects the position of a small magnet that is adhered to the tongue. Clinical trials have demonstrated the feasibility of using the TDS for computer access and wheelchair control by sensing tongue movements in up to 6 discrete locations within the oral cavity. This research will leverage the sensing capabilities of the TDS system and PI Patel's expertise in spoken interaction technologies for individuals with speech impairment, as well as Co-PI Fu's work on machine learning and multimodal data fusion, to develop a prototype clinically viable tool for enhancing speech clarity by coupling lingual-kinematic and acoustic data. To this end, the team will extend the TDS to track tongue movements during running speech, which are quick, compacted within a small area of the oral cavity, and often overlap for several phonemes, so the challenge will be to accurately classify movements for different sound classes. To complement this effort, pattern recognition of sensor spatiotemporal dynamics will be embedded into an interactive game to offer a motivating, personalized context for speech motor (re)learning by enabling audiovisual biofeedback, which is critical for speech modification. To benchmark the feasibility of the approach, the system will be evaluated on six individuals with neuromotor speech impairment and six healthy age-matched controls.