从多模态数据中学习语义知识是当前人工智能领域中的前沿课题。本项目针对多模态数据特征提取与语义融合问题研究一种结合人类知识的可解释神经网络，将包括人类常识在内的层次化知识融入多模态数据特征提取的过程，使神经网络能有效学习概念和概念之间的相互关联，从而赋予特征表达很强的高层语义信息，使得与神经网络的中层表达进行语义交互变得可解释和可操控。在此基础上，研究跨模态知识迁移、任务驱动的多模态语义融合以及网络系统的模块化，实现网络的低代价灵活搭建。这项研究的意义在于提供了一种不同于端到端模式的新的网络构造和训练方法，有望大大增强网络知识提炼的精准性并缩减网络规模、降低训练和知识共享的代价，是神经网络研究的一种新思路。我们在多模态疾病诊断和多模态动作识别与检索问题中检验提出的方法，研究过程中提出的概念卷积、非线性注意力机制、跨模态知识蒸馏、自适应强化学习等新技术也会对深度学习的发展起到推动作用。

Learning semantic knowledge from multimodal data is one of the frontier topics in artificial intelligence. Aiming at multimodal feature extraction and semantic fusion, this project proposes a knowledge integrated interpretable neural network model that fuses the hierarchical knowledge, including human commonsense, into the procedure of feature extraction, such that the neural network can learn concepts as well as the interconnections among them and encode this strong semantic information into the feature representations. This model makes the semantic interaction with the network’s middle-level representations interpretable and maneuverable. Based on this model, we study the cross-modality knowledge transfer, task-driven multimodal semantic fusion and network modularization to realize the low cost and flexible construction of neural networks. The significance of this research is that it provides a new approach (rather than end-to-end pattern) to network construction and training, and this approach can improve the effect of the feature extraction and reduce the scale of networks and the cost of the training and knowledge sharing. The proposed model represents a new thought of the research about neural networks. We validate the model with the multimodal disease diagnosis and multimodal action recognition and retrieval problems. The new techniques proposed during the research, e.g., concept convolution, nonlinear attention mechanism, cross-modal knowledge distillation and adaptive reinforcement learning, will be real progresses in the development of deep learning.