多重相关约束下的多模态学习及其在脑图像分析中的应用

多模态学习是机器学习的重要研究内容。在脑图像分析中，多模态脑图像常呈现“模态缺失严重、强弱模态性能差异大、标记不充足且获取困难”的特点，而现有多模态学习方法难以有效解决这些问题。为此，本项目以脑图像分析为应用背景，围绕模态不完整、模态不均衡、标记不充足情况下的多模态学习三个科学问题展开研究，通过挖掘并利用模态内外、标记间等多重相关，研究多重相关约束的多模态学习方法。针对模态不完整，运用模态间结构相关与层次语义相关，研究多模态样本自适应学习与深度特征学习方法；针对模态不均衡，考虑强弱模态互补学习的方向性与安全性，研究渐进相关的强指导弱学习和可靠相关的强弱联合学习方法；针对标记不充足，运用标记局部相关与模态标记相关，研究多模态半监督弱标记学习方法。同时，将上述方法分别用于解决脑功能连接分析、脑图像分割以及脑肿瘤辅助诊断应用。本项目可为多模态学习研究提供新的思路，且对脑图像分析应用有实用价值。

Multi-modal learning is one of the important research contents in machine learning. There are three following challenges in multi-modal brain image analysis, including: (1) severe modalities missing, (2) large diversity between the performance of strong and weak modalities, and (3) insufficient labels with high cost, which cannot be addressed by the existing multi-modal learning methods effectively. Therefore, with the application of brain image analysis, this project aims at performing the research of three scientific issues of multi-modal learning with (1) incomplete modalities, (2) strong and weak modalities, and (3) insufficient labels, respectively, by capturing the multiple correlations (i.e., inter-/intra-modality, inter-label, etc). Specifically, for incomplete modalities, this project focuses on multi-modal sample-adaptive learning and deep feature learning methods with the structural and semantic correlation between modalities. For strong and weak modalities, considering the direction and safety of strong-modal and weak-modal learning, this project conducts the study of both strong-modal guided weak-modal learning via gradual correlation and joint learning of strong and weak modalities with reliable correlation. For insufficient labels, based on both the local dependence between labels and the correlation between labels and modalities, this project focuses on multi-modal semi-supervised weak-label learning methods. Meanwhile, the aforementioned multi-modal learning methods will be applied in brain connectivity analysis, brain image segmentation and brain tumor diagnosis, respectively. In summary, this project can provide new ideas for multi-modal learning, and also is practically valuable for the application of brain image analysis.