Multimodal Speech Translation for Assistive Communication

描述（由申请人提供）：构音障碍是一种影响超过400万美国人的神经运动言语障碍，通常非常严重，以至于言语无法理解，需要使用增强和/或替代通信（AAC）设备。这些设备陈旧、笨重、体积大。AAC用户不是进行面对面的互动，而是花费不成比例的时间浏览字母/图标菜单来撰写消息，然后可以通过集成的文本到语音合成系统大声说出。因此，AAC的互动是缓慢的，费力的，不自然的，往往阻碍而不是支持社会，教育和职业机会。事实上，许多AAC用户继续发声与熟悉的照顾者，这意味着一致的模式必须根据构音障碍的生产。正是这些不精确但一致的产品，我们建议通过多模态传感器捕获并使用语音翻译的模式识别算法进行分类。虽然自动语音识别对于神经系统完好的说话者或有轻度损伤的人来说是一种可行的技术，但它在声学上苛刻的说话环境中以及对于那些患有更严重的构音障碍的人来说是失败的。相反，我们专注于多模态（语言运动学和声学; LinKA）表示的语音，因为它们提供了冗余和互补的输入通道，以改善消歧。虽然其他方法已经使用计算机视觉、超声成像和肌电图来同时估计语音的发音和声学参数，但它们在便携性、成本和临床环境的应用方面受到限制。当前 该提案利用了脸颊附近的新型、轻量、可穿戴且低成本的磁传感器阵列，其可以识别由放置在舌头上的小磁示踪剂产生的磁场图案，以捕获讲话期间的舌运动学。将舌头运动与通过安装在同一耳机上的麦克风捕获的声学信号相耦合，提供了语音的多维表示，然后可以将其转换为新一代可穿戴的语音驱动AAC设备的清晰可理解的语音。拟议的工作将优化的效率和鲁棒性的语言运动学和声学传感移动的语音翻译（目标1），产生一个标准化的实施协议的培训和独立使用的LinKA系统（目标2），并最终在为期2周的现场测试的LinKA翻译与12个潜在的用户与语音障碍（目标3）。目前的提案是迈向低成本、可穿戴、个性化通信增强系统的第一步，也是必不可少的一步，该系统可以为有语言障碍的个人拓宽通信机会和网络，从而提高通信参与度、独立性和整体生活质量。

DESCRIPTION (provided by applicant): Dysarthria, a neuromotor speech disorder impacting over 4 million Americans, is often so severe that speech is rendered unintelligible, requiring the use of augmentative and/or alternative communication (AAC) devices. These devices are dated, cumbersome and bulky. Rather than engaging in face-to-face interaction, AAC users spend a disproportionate amount of time navigating through menus of letters/icons to compose a message, which can then be spoken aloud by an integrated text-to-speech synthesis system. Thus AAC interactions are slow, effortful, unnatural, and often hinder rather than support social, educational and vocational opportunities. In fact, many AAC users continue to vocalize with familiar caregivers implying that consistent patterns must underlie dysarthric productions. It is these imprecise yet consistent productions that we propose to capture via multimodal sensors and classify using pattern recognition algorithms for speech translation. While automatic speech recognition is a viable technology for neurologically intact speakers or those with mild impairments, it fails in acoustically harsh speaking contexts and for those with more severe dysarthria. Instead, we focus on multimodal (lingual kinematic and acoustic; LinKA) representations of speech as they provide redundant and complementary channels of input for improved disambiguation. While other approaches have used computer vision, ultrasound imaging and electromyography to simultaneously estimate articulatory and acoustic parameters of speech, they are limited in portability, cost, and application to clinical settings. The current proposal leverages a novel, lightweight, wearable and low-cost array of magnetic sensors near the cheeks that can recognize the magnetic field patterns generated by a small magnetic tracer placed on the tongue to capture lingual kinematics during speech. Coupling tongue movements with the acoustic signal, captured via microphones mounted on the same headset, provides a multidimensional representation of speech that can then be translated into clear understandable speech for a new generation of wearable, speech-driven AAC devices. The proposed work will optimize the efficiency and robustness of lingual-kinematic and acoustic sensing for mobile speech translation (Aim 1), yield a standardized implementation protocol for training and independent use of the LinKA system (Aim 2), and culminate in a 2-week field test of the LinKA translator with 12 potential users with speech impairment (Aim 3). The current proposal is a first and essential step toward a low-cost, wearable, personalized communication enhancement system that can broaden communication opportunities and networks for individuals with speech impairment and thereby increase communication participation, independence and overall quality of life.