Visual speech recognition using ultrasound tongue and video lip/face images

使用超声波舌头和视频唇/脸图像进行视觉语音识别

• 批准号: 23520467
• 负责人: WILSON Ian
• 金额: $ 3.08万
• 依托单位: The University of Aizu
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (C)
• 财政年份: 2011
• 资助国家: 日本
• 起止时间: 2011 至 2013
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/grant/KAKENHI-PROJECT-23520467/
• 关键词: ultrasound; video; tongue; articulation; jaw; 超音波; ビデオ; 舌; 調音; acoustics; optical flow; pixel-wise; computer lipreading; stress; pitch

There are three main results of our research: (1) Related to video data collection of jaw movement, when measuring the amount of skin stretching over the mandible for the vowel in a CVC syllable, the onset consonant (but not the coda consonant) has a significant effect. (2) Related to ultrasound data collection of tongue position when speaking English, native (L1) speakers rest their tongue in a more efficient location (closer to the median position for English speech sounds) than Japanese (L2) speakers do. (3) Related to our focus on how best to construct and interpret a feature space we call MUTIS (midsagittal ultrasound tongue image space), results indicated that higher dimensions of MUTIS are most effective for identifying people, and that primarily the lower dimensions of VSS (vocal sound space) data are most effective for identifying phonemes. Trajectories within the VSS data indicate clear differences between L1 and L2 speakers, but not within the MUTIS data alone.

我们的研究主要有三个结果：(1)对于下颌运动的视频数据采集，当测量CVC音节中元音在下颌骨上的皮肤伸展程度时，声母(而不是尾辅音)有显著的影响。(2)与舌位超声数据采集有关，母语为英语的人比日语为母语的人在更有效的位置(更接近英语语音的中位位置)休息舌头。(3)关于如何更好地构建和解释我们称为MUTIS(正中矢状超声舌象空间)的特征空间，结果表明，MUTIS的高维对于人的识别是最有效的，而首先VSS(声音空间)数据的低维对于音素识别是最有效的。VSS数据中的轨迹表明L1和L2说话者之间存在明显的差异，但不仅仅是MUTIS数据中的差异。

项目成果

Effect of syllable onset, coda, and nucleus on degree of skin stretching over the mandible.

音节开始、尾声和核心对下颌骨皮肤伸展程度的影响。

• DOI: 10.1121/1.4799467
• 发表时间: 2013
• 期刊: Proc. of International Congress of Acoustics.
• 作者: Wilson;I. and Erickson;D.（連携研究者）
• 通讯作者: D.（連携研究者）

発音習得のための超音波舌画像に対する音素片マッピング [Mapping phonemes to midsagittal tongue images for pronunciation learning]

将音素映射到正中矢状舌头图像以进行发音学习

• 发表时间: 2012
• 作者: Yaguchi;Y.;N.Horiguchi;& I.Wilson
• 通讯作者: & I.Wilson

Pitch and intensity in the speech of Japanese speakers of English: Comparison with L1 speakers

日本人说英语时的音调和强度：与母语人士的比较

• DOI: 10.1121/1.4755421
• 发表时间: 2012
• 期刊: Journal of the Acoustical Society of America
• 影响因子: 2.4
• 作者: J. Okada;I. Wilson;M. Yoshizawa
• 通讯作者: M. Yoshizawa

Articulatory and laryngeal contributions to rhythm in English

发音和喉部对英语节奏的贡献

• 发表时间: 2013
• 作者: Erickson;D. & I.Wilson
• 通讯作者: D. & I.Wilson

Video recordings of L1 and L2 jaw movement: Effect of syllable onset on jaw opening during syllable nucleus

L1 和 L2 下颌运动的视频记录：音节开始对音节核期间下颌张开的影响

• DOI: 10.1121/1.4755428
• 发表时间: 2012
• 期刊: Journal of the Acoustical Society of America
• 影响因子: 2.4
• 作者: Y. Abe;I. Wilson;D. Erickson
• 通讯作者: D. Erickson