SFusion: Self-attention Based N-to-One Multimodal Fusion Block
SFusion: Self-attention Based N-to-One Multimodal Fusion Block
复制标题
DOI：
10.1007/978-3-031-43895-0_15
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Ze Liu;Jia Wei;Rui Li;Jianlong Zhou
中科院分区：
文献类型：
--
作者：
Ze Liu;Jia Wei;Rui Li;Jianlong Zhou
人们用不同的感官来感知世界，如视觉、听觉、嗅觉和触觉。处理和融合来自多种模态的信息使人工智能能够更容易地理解我们周围的世界。然而，当存在缺失模态时，可用模态的数量在不同情况下是不同的，这导致N到一个融合问题。为了解决这个问题，我们提出了一个自我关注的融合块称为SFusion。不同于预设公式或基于卷积的方法，所提出的块自动学习融合可用的模态，而不合成或补零丢失的模态。具体来说，从上游处理模型中提取的特征表示被投影为令牌并馈送到自我注意模块以生成潜在的多模态相关性。然后，引入模态注意机制来构建共享表示，该共享表示可被下游决策模型应用。所提出的SFusion可以很容易地集成到现有的多模态分析网络。在这项工作中，我们将SFusion应用于不同的骨干网络，用于人类活动识别和脑肿瘤分割任务。大量的实验结果表明，SFusion块实现了更好的性能比竞争的融合策略。我们的代码可在https://github.com/scut-cszcl/SFusion上获得。
People perceive the world with different senses, such as sight, hearing, smell, and touch. Processing and fusing information from multiple modalities enables Artificial Intelligence to understand the world around us more easily. However, when there are missing modalities, the number of available modalities is different in diverse situations, which leads to an N-to-One fusion problem. To solve this problem, we propose a self-attention based fusion block called SFusion. Different from preset formulations or convolution based methods, the proposed block automatically learns to fuse available modalities without synthesizing or zero-padding missing ones. Specifically, the feature representations extracted from upstream processing model are projected as tokens and fed into self-attention module to generate latent multimodal correlations. Then, a modal attention mechanism is introduced to build a shared representation, which can be applied by the downstream decision model. The proposed SFusion can be easily integrated into existing multimodal analysis networks. In this work, we apply SFusion to different backbone networks for human activity recognition and brain tumor segmentation tasks. Extensive experimental results show that the SFusion block achieves better performance than the competing fusion strategies. Our code is available at https://github.com/scut-cszcl/SFusion.