Multimodal Speech Translation for Assistive Communication

描述(申请人提供)：构音障碍，一种影响超过400万美国人的神经运动性言语障碍，通常非常严重，以至于言语变得难以理解，需要使用增强和/或替代通信(AAC)设备。这些设备陈旧、笨重、笨重。AAC用户不是进行面对面的交互，而是花费不成比例的时间在字母/图标菜单中导航以撰写消息，然后可以通过集成的文本到语音合成系统大声朗读。因此，AAC的互动是缓慢、费力、不自然的，而且往往阻碍而不是支持社会、教育和职业机会。事实上，许多AAC用户继续与熟悉的照顾者交谈，暗示一致的模式必须是关节功能障碍产生的基础。正是这些不精确但一致的结果，我们建议通过多模式传感器捕获，并使用用于语音翻译的模式识别算法进行分类。虽然自动语音识别对于神经学完好的说话者或有轻度障碍的人来说是一项可行的技术，但它在听觉刺耳的说话环境中和那些有更严重构音障碍的人身上失败了。相反，我们专注于语音的多模式(语言、运动学和声学；Linka)表示，因为它们提供了冗余和互补的输入通道，以改进歧义消除。虽然其他方法已经使用计算机视觉、超声成像和肌电图术来同时估计语音的发音和声学参数，但它们在便携性、成本和临床应用方面受到限制。海流 该提案利用了一种新颖、轻便、可穿戴和低成本的靠近脸颊的磁性传感器阵列，可以识别放置在舌头上的小型磁性跟踪器产生的磁场模式，以捕捉语音过程中的语言运动学。将舌头运动与通过安装在同一耳机上的麦克风捕获的声音信号相结合，可以提供语音的多维表示，然后可以转换为新一代可穿戴、语音驱动的AAC设备的清晰可理解语音。拟议的工作将优化移动语音翻译的语言运动学和声学感知的效率和稳健性(目标1)，为培训和独立使用Linka系统(目标2)产生标准化的实施协议，并最终对Linka翻译机进行为期两周的实地测试，有12个潜在的言语障碍用户(目标3)。目前的提议是迈向低成本、可穿戴的个性化交流增强系统的第一步，也是必不可少的一步，该系统可以拓宽言语障碍个人的交流机会和网络，从而提高交流的参与度、独立性和整体生活质量。

DESCRIPTION (provided by applicant): Dysarthria, a neuromotor speech disorder impacting over 4 million Americans, is often so severe that speech is rendered unintelligible, requiring the use of augmentative and/or alternative communication (AAC) devices. These devices are dated, cumbersome and bulky. Rather than engaging in face-to-face interaction, AAC users spend a disproportionate amount of time navigating through menus of letters/icons to compose a message, which can then be spoken aloud by an integrated text-to-speech synthesis system. Thus AAC interactions are slow, effortful, unnatural, and often hinder rather than support social, educational and vocational opportunities. In fact, many AAC users continue to vocalize with familiar caregivers implying that consistent patterns must underlie dysarthric productions. It is these imprecise yet consistent productions that we propose to capture via multimodal sensors and classify using pattern recognition algorithms for speech translation. While automatic speech recognition is a viable technology for neurologically intact speakers or those with mild impairments, it fails in acoustically harsh speaking contexts and for those with more severe dysarthria. Instead, we focus on multimodal (lingual kinematic and acoustic; LinKA) representations of speech as they provide redundant and complementary channels of input for improved disambiguation. While other approaches have used computer vision, ultrasound imaging and electromyography to simultaneously estimate articulatory and acoustic parameters of speech, they are limited in portability, cost, and application to clinical settings. The current proposal leverages a novel, lightweight, wearable and low-cost array of magnetic sensors near the cheeks that can recognize the magnetic field patterns generated by a small magnetic tracer placed on the tongue to capture lingual kinematics during speech. Coupling tongue movements with the acoustic signal, captured via microphones mounted on the same headset, provides a multidimensional representation of speech that can then be translated into clear understandable speech for a new generation of wearable, speech-driven AAC devices. The proposed work will optimize the efficiency and robustness of lingual-kinematic and acoustic sensing for mobile speech translation (Aim 1), yield a standardized implementation protocol for training and independent use of the LinKA system (Aim 2), and culminate in a 2-week field test of the LinKA translator with 12 potential users with speech impairment (Aim 3). The current proposal is a first and essential step toward a low-cost, wearable, personalized communication enhancement system that can broaden communication opportunities and networks for individuals with speech impairment and thereby increase communication participation, independence and overall quality of life.