Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
复制标题
DOI：
10.1109/tpami.2023.3284038
复制
发表时间：
2022-07
影响因子：
23.6
通讯作者：
Yang Liu;Guanbin Li;Liang Lin
中科院分区：
文献类型：
--
作者：
Yang Liu;Guanbin Li;Liang Lin
现有的视觉问题回答方法往往遭受跨模态虚假相关性和过度简化的事件级推理过程，无法捕捉事件的时间性，因果关系和动态跨越视频。在这项工作中，以解决任务的事件级视觉问答，我们提出了一个跨模态因果关系推理的框架。特别是，引入了一组因果干预操作，以发现跨视觉和语言模态的潜在因果结构。跨模态因果关系推理（Cross-Modal Causal RelatIonal Reasoning，CMCIR）包括三个模块：1）感知因果关系的视觉-语言推理（CVLR）模块，用于通过前门和后门因果干预来协同地解开视觉和语言之间的虚假关联; 2）时空Transformer（STT）模块，用于捕获视觉和语言语义之间的细粒度交互; iii）视觉语言特征融合（VLFF）模块，用于自适应地学习全局语义感知视觉语言表示。在四个事件级数据集上的大量实验表明，我们的CMCIR在发现视觉语言因果结构和实现鲁棒的事件级视觉问答方面具有优越性。
Existing visual question answering methods often suffer from cross-modal spurious correlations and oversimplified event-level reasoning processes that fail to capture event temporality, causality, and dynamics spanning over the video. In this work, to address the task of event-level visual question answering, we propose a framework for cross-modal causal relational reasoning. In particular, a set of causal intervention operations is introduced to discover the underlying causal structures across visual and linguistic modalities. Our framework, named Cross-Modal Causal RelatIonal Reasoning (CMCIR), involves three modules: i) Causality-aware Visual-Linguistic Reasoning (CVLR) module for collaboratively disentangling the visual and linguistic spurious correlations via front-door and back-door causal interventions; ii) Spatial-Temporal Transformer (STT) module for capturing the fine-grained interactions between visual and linguistic semantics; iii) Visual-Linguistic Feature Fusion (VLFF) module for learning the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on four event-level datasets demonstrate the superiority of our CMCIR in discovering visual-linguistic causal structures and achieving robust event-level visual question answering.