基于深度学习的表情、语音和姿态多模态情感识别研究

多模态情感识别是人工智能、情感计算和图像处理等领域的研究热点。目前的多模态情感识别研究主要集中于表情和语音双模态情感识别以及表情和姿态双模态情感识别，很少有研究人员进行表情、语音和姿态多模态情感识别，但表情、语音和姿态是人类最重要的三种情感表达方式。因此，本项目旨在申请人原有单模态和多模态情感识别研究基础上，开展基于深度学习的表情、语音和姿态多模态情感识别研究。本项目的主要内容包括：1）研究表情模态和姿态模态的视频情感特征提取，提出基于立方体Attention的增强卷积和空时LSTM网络；2）研究语音模态的情感特征提取，提出基于分段语谱图和双重Attention的多路卷积神经网络；3）研究三种模态的深度情感特征融合，提出稀疏监督多类典型相关分析融合方法、监督多类核典型相关分析融合方法和稀疏监督多类核典型相关分析融合方法；4）实现基于深度学习的表情、语音和姿态多模态情感识别系统。

Multimodal emotion recognition is the research hotspots of artificial intelligence, affective computing, image processing and so on. The current multimodal emotion recognition research focuses on bimodal emotion recognition based on facial expression and speech, bimodal emotion recognition based on facial expression and body gesture, and few researchers have conducted multimodal emotion recognition based on facial expression, speech and body gesture. But facial expression, speech and body gesture are the three most important emotion expression patterns for human beings. Therefore, on the base of the original research results of monomodal and multimodal emotion recognition, this project will conduct research of multimodal emotion recognition of facial expression, speech and body gesture based on deep learning. The main contents of this project include: 1) Researching the video emotion feature extraction of facial expression and body gesture modality, as well as proposing the enhanced convolution and spatio-temporal LSTM network based on cube attention; 2) Researching the emotion feature extraction of speech modality, as well as proposing the multi-channel convolutional neural network based on subsection spectrogram and double attention; 3) Researching the deep emotion feature fusion of three modalities and proposing the sparse supervised multiset canonical correlation analysis fusion method, supervised multiset kernel canonical correlation analysis fusion method and sparse supervised multiset kernel canonical correlation analysis fusion method; 4) Realizing the multimodal emotion recognition system of facial expression, speech and body gesture based on deep learning.