基于发音器官的运动来识别语音（静音语音识别）可应用到广泛的领域，如识别喉切除的病人说话;在会场及电影院等需要保持静音或者隐私的场所不出声音只运动发音器官来打电话;再比如在高噪声环境中进行语音识别等。本课题将用超声仪结合摄像头来采集说话人的发音器官运动，基于唇部运动图像及舌头运动图像来获取发音器官运动,从而识别相应的语音。本课题将基于实现静音语音识别中的几个关键技术点来展开研究。首先，将利用有限波尔兹曼机通过非监督学习对摄像机记录的唇部运动图像及用超声仪记录的舌头运动图像分别进行特征提取及非线性的降维。为了实现多模态发音运动数据的融合，本课题利用多个有限波尔兹曼机搭建一个深度神经网络来对多通道、多模态的发音运动数据进行数据融合研究。基于多模态融合获得的发音运动数据特征，利用隐马尔科夫模型进行静音语音识别研究。本课题将在发音运动特征提取、多通道数据融合、发音运动识别等方面有创新性成果。

Speech recognition based on articulator movement (silent speech recognition)can be applied widely in many areas, such as speech recognition for patients with laryngectomy;making calls but speaking without sound in the situations like venue and cinemas where we need to keep silent or privacy,speech recognition in high-noise environments and so on. This project will use ultrasound machine and cameras to acquire images about speaker's articulator and recognize articulator's movement to get corresponding speech by the images. The research will focus on several key technical points about silent speech recognition. First, we will use constrained Boltzmann machine and unsupervised learning to extract features and decrease dimension nonlinearly for images about lips and tongues movements recorded by camera and ultrasonund machine,respectively. The research uses multiple constrained Boltzmann machines to build a deep neural network to fuse multi-channel and multi-modal data.We will use hidden Markov model in silent speech recognition by the characteristics derived from the fusion of multi-modal data.In the research, there will be innovative achievements in the feature extraction, multi-channel data fusion, articulator recognition and so on.