Multimodal Speech Translation for Assistive Communication

描述（由申请人提供）：构音障碍是一种影响超过400万美国人的神经运动语言障碍，通常非常严重，以至于言语变得难以理解，需要使用辅助和/或替代通信（AAC）设备。这些设备过时、笨重、笨重。与面对面的互动不同，AAC用户花了大量的时间浏览字母/图标菜单来撰写信息，然后通过集成的文本-语音合成系统大声朗读。因此，AAC的互动是缓慢的、费力的、不自然的，并且经常阻碍而不是支持社会、教育和职业机会。事实上，许多AAC用户继续与熟悉的护理人员发声，这意味着一致的模式必须是痛苦产生的基础。我们建议通过多模态传感器捕获这些不精确但一致的结果，并使用语音翻译的模式识别算法进行分类。虽然自动语音识别对于神经系统完好的说话者或有轻微损伤的人来说是一种可行的技术，但它在声音苛刻的说话环境中以及对那些有更严重构音障碍的人来说是失败的。相反，我们专注于语音的多模态（语言运动学和声学；LinKA）表示，因为它们为改进消歧提供了冗余和互补的输入通道。虽然其他方法使用计算机视觉、超声成像和肌电图来同时估计语音的发音和声学参数，但它们在可移植性、成本和临床应用方面受到限制。当前的

DESCRIPTION (provided by applicant): Dysarthria, a neuromotor speech disorder impacting over 4 million Americans, is often so severe that speech is rendered unintelligible, requiring the use of augmentative and/or alternative communication (AAC) devices. These devices are dated, cumbersome and bulky. Rather than engaging in face-to-face interaction, AAC users spend a disproportionate amount of time navigating through menus of letters/icons to compose a message, which can then be spoken aloud by an integrated text-to-speech synthesis system. Thus AAC interactions are slow, effortful, unnatural, and often hinder rather than support social, educational and vocational opportunities. In fact, many AAC users continue to vocalize with familiar caregivers implying that consistent patterns must underlie dysarthric productions. It is these imprecise yet consistent productions that we propose to capture via multimodal sensors and classify using pattern recognition algorithms for speech translation. While automatic speech recognition is a viable technology for neurologically intact speakers or those with mild impairments, it fails in acoustically harsh speaking contexts and for those with more severe dysarthria. Instead, we focus on multimodal (lingual kinematic and acoustic; LinKA) representations of speech as they provide redundant and complementary channels of input for improved disambiguation. While other approaches have used computer vision, ultrasound imaging and electromyography to simultaneously estimate articulatory and acoustic parameters of speech, they are limited in portability, cost, and application to clinical settings. The current proposal leverages a novel, lightweight, wearable and low-cost array of magnetic sensors near the cheeks that can recognize the magnetic field patterns generated by a small magnetic tracer placed on the tongue to capture lingual kinematics during speech. Coupling tongue movements with the acoustic signal, captured via microphones mounted on the same headset, provides a multidimensional representation of speech that can then be translated into clear understandable speech for a new generation of wearable, speech-driven AAC devices. The proposed work will optimize the efficiency and robustness of lingual-kinematic and acoustic sensing for mobile speech translation (Aim 1), yield a standardized implementation protocol for training and independent use of the LinKA system (Aim 2), and culminate in a 2-week field test of the LinKA translator with 12 potential users with speech impairment (Aim 3). The current proposal is a first and essential step toward a low-cost, wearable, personalized communication enhancement system that can broaden communication opportunities and networks for individuals with speech impairment and thereby increase communication participation, independence and overall quality of life.