Multimodal Speech Translation for Assistive Communication
用于辅助沟通的多模态语音翻译
基本信息
- 批准号:8913172
- 负责人:
- 金额:$ 21.79万
- 依托单位:
- 依托单位国家:美国
- 项目类别:
- 财政年份:2014
- 资助国家:美国
- 起止时间:2014-08-15 至 2018-07-31
- 项目状态:已结题
- 来源:
- 关键词:AcousticsActivities of Daily LivingAdherenceAlgorithmsAmericanArticular Range of MotionAugmentative and Alternative Communication deviceCaregiversCheek structureClinicalCommunicationCommunication Aids for DisabledCommunication impairmentComputer InterfaceComputer Vision SystemsCoupledCouplingDataDetectionDevicesDiseaseDysarthriaElectromyographyEmploymentEnsureEyeGenerationsHawksHealthImpairmentIndividualLaboratoriesLettersLifeMagnetismModificationMorphologic artifactsMotorMovementMuscleOutcome MeasurePatternPattern RecognitionPerformanceProductionProtocols documentationQuality of lifeRecruitment ActivityRunningSelf-Help DevicesSeriesSignal TransductionSocial supportSpeechSpeech AcousticsSpeech DisordersSystemTechnologyTestingTextTimeTongueTracerTrainingTranslatingTranslationsUltrasonographyWheelchairsWireless TechnologyWorkalternative communicationcopingcostdesigndeviantimprovediterative designkinematicsmagnetic fieldnovelphrasesportabilityresearch studysensorsocialspeech recognitionsuccessusability
项目摘要
DESCRIPTION (provided by applicant): Dysarthria, a neuromotor speech disorder impacting over 4 million Americans, is often so severe that speech is rendered unintelligible, requiring the use of augmentative and/or alternative communication (AAC) devices. These devices are dated, cumbersome and bulky. Rather than engaging in face-to-face interaction, AAC users spend a disproportionate amount of time navigating through menus of letters/icons to compose a message, which can then be spoken aloud by an integrated text-to-speech synthesis system. Thus AAC interactions are slow, effortful, unnatural, and often hinder rather than support social, educational and vocational opportunities. In fact, many AAC users continue to vocalize with familiar caregivers implying that consistent patterns must underlie dysarthric productions. It is these imprecise yet consistent productions that we propose to capture via multimodal sensors and classify using pattern recognition algorithms for speech translation. While automatic speech recognition is a viable technology for neurologically intact speakers or those with mild impairments, it fails in acoustically harsh speaking contexts and for those with more severe dysarthria. Instead, we focus on multimodal (lingual kinematic and acoustic; LinKA) representations of speech as they provide redundant and complementary channels of input for improved disambiguation. While other approaches have used computer vision, ultrasound imaging and electromyography to simultaneously estimate articulatory and acoustic parameters of speech, they are limited in portability, cost, and application to clinical settings. The current
proposal leverages a novel, lightweight, wearable and low-cost array of magnetic sensors near the cheeks that can recognize the magnetic field patterns generated by a small magnetic tracer placed on the tongue to capture lingual kinematics during speech. Coupling tongue movements with the acoustic signal, captured via microphones mounted on the same headset, provides a multidimensional representation of speech that can then be translated into clear understandable speech for a new generation of wearable, speech-driven AAC devices. The proposed work will optimize the efficiency and robustness of lingual-kinematic and acoustic sensing for mobile speech translation (Aim 1), yield a standardized implementation protocol for training and independent use of the LinKA system (Aim 2), and culminate in a 2-week field test of the LinKA translator with 12 potential users with speech impairment (Aim 3). The current proposal is a first and essential step toward a low-cost, wearable, personalized communication enhancement system that can broaden communication opportunities and networks for individuals with speech impairment and thereby increase communication participation, independence and overall quality of life.
描述(由申请人提供):构音障碍是一种影响超过 400 万美国人的神经运动性言语障碍,其严重程度常常导致言语难以理解,需要使用增强和/或替代交流 (AAC) 设备。这些设备过时、笨重且体积庞大。 AAC 用户不是进行面对面的交互,而是花费过多的时间浏览字母/图标菜单来撰写消息,然后通过集成的文本到语音合成系统大声朗读该消息。因此,AAC 的互动是缓慢的、费力的、不自然的,并且常常阻碍而不是支持社会、教育和职业机会。事实上,许多 AAC 用户继续向熟悉的护理人员发声,暗示构音障碍的产生必然存在一致的模式。我们建议通过多模态传感器捕获这些不精确但一致的产物,并使用语音翻译的模式识别算法进行分类。虽然自动语音识别对于神经系统完好的说话者或有轻度损伤的人来说是一项可行的技术,但它在声音恶劣的说话环境中以及对于构音障碍更严重的人来说却失败了。相反,我们专注于语音的多模态(语言运动学和声学;LinKA)表示,因为它们提供冗余和互补的输入通道以改进歧义消除。虽然其他方法使用计算机视觉、超声成像和肌电图来同时估计语音的发音和声学参数,但它们在便携性、成本和临床应用方面受到限制。目前的
该提案利用脸颊附近的新型、轻质、可穿戴且低成本的磁传感器阵列,可以识别放置在舌头上的小型磁示踪剂产生的磁场模式,以捕获言语过程中的舌运动学。将舌头运动与通过安装在同一耳机上的麦克风捕获的声学信号耦合起来,可以提供语音的多维表示,然后可以将其转换为清晰可理解的语音,用于新一代可穿戴、语音驱动的 AAC 设备。拟议的工作将优化移动语音翻译的语言运动和声学传感的效率和鲁棒性(目标 1),产生用于训练和独立使用 LinKA 系统的标准化实施协议(目标 2),并最终对 12 名有语言障碍的潜在用户进行为期 2 周的 LinKA 翻译器现场测试(目标 3)。目前的提案是朝着低成本、可穿戴、个性化通信增强系统迈出的第一步,也是至关重要的一步,该系统可以拓宽语言障碍人士的通信机会和网络,从而提高通信参与度、独立性和整体生活质量。
项目成果
期刊论文数量(2)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
Evaluation of a Wireless Tongue Tracking System on the Identification of Phoneme Landmarks.
- DOI:10.1109/tbme.2020.3023284
- 发表时间:2021-04
- 期刊:
- 影响因子:0
- 作者:Sebkhi N;Santus N;Bhavsar A;Siahpoushan S;Inan OT
- 通讯作者:Inan OT
Joint Magnetic Calibration and Localization Based on Expectation Maximization for Tongue Tracking.
- DOI:10.1109/tbme.2017.2688919
- 发表时间:2018-01
- 期刊:
- 影响因子:0
- 作者:Lu J;Yang Z;Okkelberg KZ;Ghovanloo M
- 通讯作者:Ghovanloo M
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
数据更新时间:{{ journalArticles.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ monograph.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ sciAawards.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ conferencePapers.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ patent.updateTime }}
RUPAL PATEL其他文献
RUPAL PATEL的其他文献
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
{{ truncateString('RUPAL PATEL', 18)}}的其他基金
VocaliD SBIR Phase II: Optimized Speech Corpora for Personalized Speech Synthesis
VocaliD SBIR 第二阶段:用于个性化语音合成的优化语音语料库
- 批准号:
9408604 - 财政年份:2015
- 资助金额:
$ 21.79万 - 项目类别:
Multimodal Speech Translation for Assistive Communication
用于辅助沟通的多模态语音翻译
- 批准号:
8737379 - 财政年份:2014
- 资助金额:
$ 21.79万 - 项目类别:
Prosody in Congenital and Acquired Dysarthria.
先天性和后天性构音障碍的韵律。
- 批准号:
8636737 - 财政年份:2013
- 资助金额:
$ 21.79万 - 项目类别:
Prosody in Congenital and Acquired Dysarthria.
先天性和后天性构音障碍的韵律。
- 批准号:
8763936 - 财政年份:2013
- 资助金额:
$ 21.79万 - 项目类别:
Acquisition of Prosodic Control in Typically Developing Children
典型发育儿童的韵律控制能力的获得
- 批准号:
8048490 - 财政年份:2011
- 资助金额:
$ 21.79万 - 项目类别:
Acquisition of Prosodic Control in Typically Developing Children
典型发育儿童的韵律控制能力的获得
- 批准号:
8207838 - 财政年份:2011
- 资助金额:
$ 21.79万 - 项目类别:
Identifying Communicative Signals in Dysarthric Speech
识别构音障碍言语中的交流信号
- 批准号:
6794230 - 财政年份:2004
- 资助金额:
$ 21.79万 - 项目类别:
Identifying Communicative Signals in Dysarthric Speech
识别构音障碍言语中的交流信号
- 批准号:
6866487 - 财政年份:2004
- 资助金额:
$ 21.79万 - 项目类别:
相似海外基金
Determining 4-Dimensional Foot Loading Profiles of Healthy Adults across Activities of Daily Living
确定健康成年人日常生活活动的 4 维足部负荷曲线
- 批准号:
2473795 - 财政年份:2024
- 资助金额:
$ 21.79万 - 项目类别:
Studentship
Developing a trunk function assessment for hemiplegics. -For improving activities of daily living-
开发偏瘫患者的躯干功能评估。
- 批准号:
23K10540 - 财政年份:2023
- 资助金额:
$ 21.79万 - 项目类别:
Grant-in-Aid for Scientific Research (C)
Relation with the activities of daily living and the subjective values among people with social withdrawal
社交退缩者日常生活活动与主观价值观的关系
- 批准号:
23K16596 - 财政年份:2023
- 资助金额:
$ 21.79万 - 项目类别:
Grant-in-Aid for Early-Career Scientists
CRII: RI: Understanding Activities of Daily Living in Indoor Scenarios
CRII:RI:了解室内场景中的日常生活活动
- 批准号:
2245652 - 财政年份:2023
- 资助金额:
$ 21.79万 - 项目类别:
Standard Grant
Association between Nursing Care and Prognosis and Activities of Daily Living in Acute Stroke patients by using Big Data.
利用大数据研究急性脑卒中患者的护理与预后和日常生活活动的关系。
- 批准号:
23K16412 - 财政年份:2023
- 资助金额:
$ 21.79万 - 项目类别:
Grant-in-Aid for Early-Career Scientists
Sources of vulnerability among those using homecare despite having no limitations in Activities of Daily Living. An intersectionality analysis
尽管日常生活活动没有限制,但使用家庭护理的人的脆弱性来源。
- 批准号:
499112 - 财政年份:2023
- 资助金额:
$ 21.79万 - 项目类别:
Operating Grants
Synergizing home health rehabilitation therapy to optimize patients’ activities of daily living
协同家庭健康康复治疗,优化患者的日常生活活动
- 批准号:
10429480 - 财政年份:2022
- 资助金额:
$ 21.79万 - 项目类别:
Effects of a model of nurses-occupational therapists collaborative practice on activities of daily living in elderly patients
护士-职业治疗师合作实践模式对老年患者日常生活活动的影响
- 批准号:
22K17540 - 财政年份:2022
- 资助金额:
$ 21.79万 - 项目类别:
Grant-in-Aid for Early-Career Scientists
Assessing a Novel Virtual Environment that Primes Individuals Living with AD/ADRD to Accomplish Activities of Daily Living.
评估一种新颖的虚拟环境,该环境可以帮助 AD/ADRD 患者完成日常生活活动。
- 批准号:
10668160 - 财政年份:2022
- 资助金额:
$ 21.79万 - 项目类别:
Synergizing home health rehabilitation therapy to optimize patients’ activities of daily living
协同家庭健康康复治疗,优化患者的日常生活活动
- 批准号:
10621820 - 财政年份:2022
- 资助金额:
$ 21.79万 - 项目类别: