Inconsistent Matters: A Knowledge-Guided Dual-Consistency Network for Multi-Modal Rumor Detection
Inconsistent Matters: A Knowledge-Guided Dual-Consistency Network for Multi-Modal Rumor Detection
复制标题
DOI：
10.1109/tkde.2023.3275586
复制
发表时间：
2023-06
影响因子：
8.9
通讯作者：
Mengzhu Sun;Xi Zhang;Jianqiang Ma;Yazheng Liu
中科院分区：
文献类型：
--
作者：
Mengzhu Sun;Xi Zhang;Jianqiang Ma;Yazheng Liu
谣言传播者越来越多地利用多媒体内容来吸引新闻消费者的注意力和信任。目前已有的谣言检测模型虽然利用了多模态数据，但很少考虑图像和文本之间的语义不一致，也很少发现帖子内容和背景知识之间的不一致。此外，它们通常假设多个模态的完整性，因此无法处理在现实生活场景中丢失的模态。基于社交媒体中谣言更容易出现语义不一致的现象，提出了一种新的基于知识引导的双一致性网络（Knowledge-guided Dual-consistency Network，KNN）来检测多媒体内容的谣言。该算法使用两个一致性检测子网络同时捕获跨模态和内容知识两个层次的不一致性。它还可以在不同的缺失视觉模态条件下实现鲁棒的多模态表征学习，使用特殊的令牌来区分具有视觉模态的帖子和没有视觉模态的帖子。在三个公开的真实世界多媒体数据集上进行的大量实验表明，我们的框架在完整和不完整模态条件下都可以超越最先进的基线。
Rumor spreaders are increasingly utilizing multimedia content to attract the attention and trust of news consumers. Though quite a few rumor detection models have exploited the multi-modal data, they seldom consider the inconsistent semantics between images and texts, and rarely spot the inconsistency among the post contents and background knowledge. In addition, they commonly assume the completeness of multiple modalities and thus are incapable of handling handle missing modalities in real-life scenarios. Motivated by the intuition that rumors in social media are more likely to have inconsistent semantics, a novel Knowledge-guided Dual-consistency Network is proposed to detect rumors with multimedia contents. It uses two consistency detection subnetworks to capture the inconsistency at the cross-modal level and the content-knowledge level simultaneously. It also enables robust multi-modal representation learning under different missing visual modality conditions, using a special token to discriminate between posts with visual modality and posts without visual modality. Extensive experiments on three public real-world multimedia datasets demonstrate that our framework can outperform the state-of-the-art baselines under both complete and incomplete modality conditions.