Visual speech recognition using ultrasound tongue and video lip/face images

本研究的主要结果有三：（1）在下颌运动的视频数据采集中，当测量CVC音节中元音的下颌皮肤拉伸量时，起始辅音（而不是尾辅音）具有显著的影响。(2)与讲英语时舌头位置的超声数据收集相关，母语（L1）说话者将其舌头置于比日语（L2）说话者更有效的位置（更接近英语语音的中值位置）。(3)与我们专注于如何最好地构建和解释我们称为MUTIS（正中矢状超声舌图像空间）的特征空间相关，结果表明，MUTIS的高维数对于识别人最有效，而VSS（声乐空间）数据的低维数对于识别音素最有效。VSS数据内的轨迹表明L1和L2扬声器之间的明显差异，但不是单独的MUTIS数据内。

There are three main results of our research: (1) Related to video data collection of jaw movement, when measuring the amount of skin stretching over the mandible for the vowel in a CVC syllable, the onset consonant (but not the coda consonant) has a significant effect. (2) Related to ultrasound data collection of tongue position when speaking English, native (L1) speakers rest their tongue in a more efficient location (closer to the median position for English speech sounds) than Japanese (L2) speakers do. (3) Related to our focus on how best to construct and interpret a feature space we call MUTIS (midsagittal ultrasound tongue image space), results indicated that higher dimensions of MUTIS are most effective for identifying people, and that primarily the lower dimensions of VSS (vocal sound space) data are most effective for identifying phonemes. Trajectories within the VSS data indicate clear differences between L1 and L2 speakers, but not within the MUTIS data alone.