Wearable silent speech technology to enhance impaired oral communication

可穿戴式无声语音技术可增强受损的口语交流

• 批准号: 10412276
• 负责人: Jun Wang
• 金额: $ 29.28万
• 依托单位: UNIVERSITY OF TEXAS AT AUSTIN
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-15 至 2024-07-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10412276
• 关键词: Acceleration; Acoustics; Address; Affect; Alaryngeal Speech; Algorithms; Amyotrophic Lateral Sclerosis; Articulation; Articulators; Benchmarking; California; Characteristics; Communication; Communities; Consumption; Data; Data Set; Devices; Disease; Documentation; Engineering; Enhancement Technology; Ensure; Excision; Feedback; Felis catus; Goals; Health; Hoarseness; Human; Impairment; Individual; Label; Laryngectomy; Larynx; Lip structure; Machine Learning; Magnetism; Malignant neoplasm of larynx; Maps; Measures; Mechanics; Mental Depression; Modeling; Motion; Motor; Movement; Nature; Output; Parents; Parkinson Disease; Patients; Pattern; Physiological; Positioning Attribute; Process; Production; Publishing; Quality of life; Readiness; Research; Research Personnel; Robotics; Rotation; Route; Sampling; Science; Self-Help Devices; Series; Signal Transduction; Site; Speech; Speech Acoustics; Speech Disorders; Speech Intelligibility; Speech Sound; Speech Therapy; Standardization; Technology; Testing; Time; Tongue; Tracer; Translating; United States; Universities; Voice; Voice Disorders; Wireless Technology; archive data; archived data; base; community planning; data archive; data mining; improved; innovation; interest; kinematics; language training; multimodal data; multimodality; new technology; novel; oral communication; public repository; recruit; repository; sensor; social exclusion; sound; tool; vibration; visual feedback; vocal cord; wearable device

Project Summary/Abstract The long-term objectives of the parent R01 project (R01DC016621, 08/15/2019 – 06/31/2024) are to obtain a deeper understanding how articulatory movement patterns are mapped to speech particularly when there is no vocal fold vibration (silent speech) and then to develop a novel, wearable assistive technology called silent speech interface (SSI) to assist the impaired oral communication for individuals in need (e.g., individuals after laryngectomy, surgical removal of larynx to treat advanced laryngeal cancer). The parent R01 project aims to (1) determine the articulatory patterns of normal (vocalized) and silent speech, produced by both healthy talkers and people after laryngectomy, (2) develop a wearable device for real-time tongue and lip motion tracking, and (3) synthesize speech from articulation directly. If successful, the proposed research will enhance human health by making an impact on individuals after laryngectomy and potentially to a broader range of other speech and voice disorders as well as visual feedback-based secondary language training and speech therapy. Through the parent R01 project, we are collecting a unique multi-modal speech dataset from patients following laryngectomy and healthy controls. This data set includes speech kinematics from multiple tracers attached on the tongue and the lips, and speech acoustics. Each tracer comprises multiple sensors that measure inertial and magnetic information that can provide additional information to assist the speech acoustic and the articulation-to-speech algorithm. Making such data ML ready for others to consume was out of scope due to required effort and complexity needed to pre-process and synchronize sampling between kinematic and acoustic data. The specific goal of this supplemental project is to make data AI/ML ready by developing the pre-processing algorithms needed to generate a set of features, along with proper formatting and labeling, that can be more easily shared through repositories and used by others to evaluate different ML algorithms. New ML models that will be tested on these ML-ready shared datasets will significantly advance our capabilities to translate articulatory motion into speech sounds, which will not only improve the quality of life for people affected by laryngectomy but also for the millions of individuals living with speech sound disorders such as Parkinson’s disease, and amyotrophic lateral sclerosis.

项目摘要/摘要 母公司R01项目(R01DC016621,2019年8月15日-2019年6月31日)的长期目标是获得 更深入地理解发音运动模式是如何映射到言语中的，特别是当有 没有声带振动(无声语音)，然后开发一种新型的可穿戴辅助技术，称为无声 语音接口(SSI)，以帮助有需要的个人(例如，之后的个人)进行口头交流 喉切除术、手术切除喉以治疗晚期喉癌)。父R01项目旨在 (1)确定正常(有声)和无声语音的发音模式，两者都是健康的 说话者和喉切除术后的人，(2)开发一种可穿戴的设备，用于实时舌动和唇动 跟踪，以及(3)直接从发音合成语音。如果成功，拟议的研究将加强 通过对喉切除术后的个体以及可能对更广泛范围的 其他言语和声音障碍以及基于视觉反馈的中级语言训练和言语 心理治疗。 通过父R01项目，我们正在收集来自以下患者的独特的多模式语音数据集 喉切除组和健康对照组。该数据集包括来自附加在 舌头和嘴唇，以及语言声学。每个示踪器由多个测量惯性的传感器组成 以及可以提供附加信息以辅助语音和声音的磁性信息 清晰度到语音转换算法。使这样的数据ML准备好供其他人使用是超出范围的，因为 所需的工作量和复杂性，需要对运动学和 声学数据。这个补充项目的具体目标是通过开发 需要预处理算法来生成一组要素，以及适当的格式化和标注， 可以更容易地通过存储库共享，并被其他人用来评估不同的ML算法。新的 将在这些支持ML的共享数据集上测试的ML模型将显著提升我们的能力 将发音动作转换为语音，这不仅将提高人们的生活质量 受喉切除的影响，但也对数百万患有语音障碍的人来说，例如 帕金森氏症和肌萎缩侧索硬化症。