IBPNet: a multi-resolution and multi-modal image fusion network via iterative back-projection
IBPNet: a multi-resolution and multi-modal image fusion network via iterative back-projection
复制标题
IBPNet：通过迭代反投影的多分辨率和多模态图像融合网络
DOI：
10.1007/s10489-022-03375-w
复制
发表时间：
2022
影响因子：
5.3
通讯作者：
庞利会
中科院分区：
文献类型：
--
作者：
刘畅;杨斌;张小志;庞利会
大多数多模态图像融合方法都是以源图像具有相同分辨率为前提的。然而，由于环境和硬件设施的限制，多模态图像的分辨率往往是不同的。例如，红外图像的空间分辨率通常低于对应的可见光图像的空间分辨率。因此，我们的动机是解决融合图像中容易出现的细节模糊或一定程度的信息丢失问题。在此基础上，提出了一种基于深度学习的迭代反投影多分辨率多模态图像融合网络（IBPNet），以获得高质量的融合图像。我们的IBPNet的关键贡献是设计了上投影和下投影块，以实现高分辨率和低分辨率图像之间的特征映射转换。交替过程中产生的反馈误差在重构过程中得到自校正。此外，设计了一种有效的组合损失函数，能够适应不同的多分辨率、多模态图像融合任务。实验结果表明，该方法在视觉感知和客观评价方面均优于其他最先进的融合方法，具有上级。
Most multi-modal image fusion methods are based on the prerequisite that the source images have the same resolution. However, due to the limitations of the environment and hardware facilities, the resolution of multi-modal images is always distinct. For example, spatial resolution of infrared images is usually lower than that of the corresponding visible images. Therefore, our motivation is to solve the problem of blurred details or a certain degree of information loss that are prone to appear in the fusion images. Under this motivation, a novel deep learning-based multi-resolution multi-modal image fusion network via iterative back-projection (IBPNet) is constructed to get high quality fused images. The key contribution of our IBPNet is to design up-projection and down-projection blocks to realize the feature mapping conversion between high and low-resolution images. The feedback errors generated in the alternation process are self-corrected in the reconstruction process. In addition, an effective combined loss function is designed, which can adapt to different multi-resolution and multi-modal image fusion tasks. Experimental results show that our method is superior to other state-of-the-art fusion methods in terms of both visual perception and objective evaluation.