基于图卷积神经网络的EEG-fNIRS双模态情感识别研究

脑电（EEG）和功能近红外成像（fNIRS）反映了大脑中枢神经活动状况，二者具有时空分辨率优势互补的特点。由于EEG/fNIRS的电极/光极排布呈不规则非欧式结构特性，使得对EEG/fNIRS情感活动的建模异常困难，成为EEG/fNIRS情感识别面临的一大挑战。图卷积是建立在图模式基础上的卷积操作，为这类不规则结构的特征提取提供了一种有效途径，可实现对EEG/fNIRS信号的动态情感建模。为此，本项目针对EEG/fNIRS情感识别问题，深入开展基于图卷积神经网络模型的识别方法研究。主要内容包括：(1)EEG-fNIRS双模态情感数据同步采集；(2)EEG/fNIRS显著通道检测与情感识别；(3)基于图卷积神经网络的个性化EEG/fNIRS情感识别；(4)EEG-fNIRS双模态情感特征动态融合与识别。通过本项目的研究，将显著提升EEG/fNIRS情感识别的泛化性、鲁棒性和准确性。

EEG and fNIRS reflect the activities of the human brain central nervous system, which have complementary advantages of spatial and temporal resolution. Because of the non-Euclidean irregular structure property of EEG/fNIRS electrode/optrode arrangement, it is a challenging task to model the emotional activities from EEG/fNIRS signals. Graph convolution is a convolution operation based on graph pattern representation, which provides an effective method for feature extraction of such irregular structures, and can realize the dynamic emotion modelling for EEG/fNIRS signals. Therefore, this project aimed at the emotional recognition of EEG/fNIRS, carried out in-depth research on the recognition method based on the graph convolutional neural network model. The major contents of this research include the following aspects: (1) Bimodal emotional data collection from both EEG and fNIRS; (2) Salient channel detection and emotion recognition from EEG/fNIRS; (3) Subject-independent emotion recognition of EEG/fNIRS based on graph convolutional neural network; (4) The fusion of bi-modal emotional features of EEG and fNIRS for emotion recognition. Through the research of this project, the generalization, robustness and accuracy of EEG/fNIRS emotion recognition would be significantly improved.