基于视觉传感器的人体行为识别容易受到遮挡和外观变化的影响，而可穿戴传感器以一维时序信号来捕捉人体运动，从而避免上述挑战。因此，本项目拟结合视觉和可穿戴传感器行为数据，实现鲁棒的多模态人体行为识别系统。但现阶段面临几个亟待解决的问题：1）可穿戴传感器数据不包含纹理和颜色信息且表达能力较弱，需要设计针对性的特征提取方法；2）如何进行有效的知识迁移来减少视觉和可穿戴传感器数据之间的模态差异；3）如何挖掘视觉和可穿戴传感器数据之间的互补信息，并进行自适应的特征融合。针对上述问题，结合申请人前期研究工作（发表在TIP、TCSVT、Complexity、SPL等主流期刊上），本项目拟构建格兰姆角度场和语义自编码器模型来学习可穿戴传感器行为特征，设计基于三通道卷积神经网络的域不变表达学习框架来减少模态差异，提出基于压缩和激励的双通道自适应特征融合网络来实现多模态特征融合，最终提升多模态人体行为识别性能。

Vision-sensor based human action recognition is susceptible to occlusion and appearance variations, while wearable sensors can capture human motion with a one-dimensional time series signal to avoid this challenge. Therefore, this project intends to leverage both vision and wearable sensor based action data to achieve a robust multi-modal human action recognition system. However, it faces several key problems: 1) wearable sensor data does not contain texture and color information and thus its discriminative power is weak, so specific feature extraction methods should be designed to solve this problem; 2) how to design effective knowledge transfer method to reduce the modality difference between the action data of vision-sensor and wearable sensor; 3) how to discover complementary information between vision-sensor and wearable sensor data and realize adaptive feature fusion. To address aforementioned problems and based on the applicant’s previous works (these works have been published on TIP, TCSVT, Complexity, SPL, etc.), this project will build feature learning model based on gramian angular field and semantic autoencoder to learn action features of wearable sensor, and construct three-stream convolutional neural networks based domain-invariant representation learning framework to reduce the modality gap, and then build two-stream adaptive feature fusion network based on squeeze and excitation to realize multi-modal feature fusion, and ultimately improve the performance of multi-modal human action recognition.