基于因果分析的视觉内容描述研究

近年来，得益于深度学习强大的特征感知和表示能力，视觉内容描述研究取得了显著进展。然而，面对物理世界中的有偏数据，深度学习方法由于缺乏归纳一般性规则和推理能力，无法正确区分有用的特征模式和无用的偏差，从而导致模型鲁棒性和泛化能力较差。基于此，本项目引入因果分析方法，结合深度学习技术研究视觉内容描述。本项目重点研究“基于图像语义编辑的视觉内容描述方法”、“基于因果混杂因子分析的视觉内容描述方法”和“基于场景图的反事实视觉内容描述方法”三方面内容，分别从浅层数据分布干预、中层潜在因果混杂因子分析以及深层的反事实因果推理三个层次，层层递进，提升视觉内容描述系统性能。本项目的研究成果能为如何建立具有简单认知推理能力的视觉内容描述系统提供新方法和新思路，并推动视觉内容理解及语义分析在复杂真实环境的应用。

Visual captioning has made great progress in recent years, thanks to the powerful capacities in feature perception and representation of deep learning. However, deep learning, poor in reasoning and induction, cannot discriminate positive patterns and negative patterns (biases) effectively, resulting in models with poor robustness and generalization for real-world biased data. Towards this end, this project introduces causal analysis and combines it with deep learning for visual captioning. Specifically, this project focuses on the research of three aspects: (1) visual captioning based on image semantics editing, (2) visual captioning via causal confounder analysis, and (3) scene graph based counterfactual visual captioning. These research aim to improve the effectiveness of visual captioning system from shallow data distribution intervention, middle-level latent causal confounder analysis, and deep causal reasoning with counterfactuals. The research findings of this project can provide new strategies and new ideas for how to build an intelligent visual captioning system with the ability of reasoning, and promote the application of visual content understanding and semantic analysis in complex real-world scenarios.