Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation
Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation
复制标题
用于多模态先验引导磁共振成像分割的基于互信息的图共同关注网络
DOI：
10.1109/tcsvt.2021.3112551
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
and Yen-Wei Chen
中科院分区：
文献类型：
--
作者：
Shaocong Mo;Ming Cai;Lanfen Lin;Ruofeng Tong;Qingqing Chen;Fang Wang;Hongjie Hu;Yutaro Iwamoto;Xian-Hua Han;and Yen-Wei Chen
多模态磁共振成像（MRI）提供了有关目标的补充信息，多模态磁共振成像的分割被广泛用作临床初步诊断、分期区分和治疗后疗效评估的重要预处理步骤。对于主要模态或每种模态，通过建模连接并有效融合它们之间的特征来增强视觉信息非常重要。然而，现有的多模态分割方法有一个缺点：它们在融合过程中巧合地丢弃了个体模态的信息。最近，基于图学习的方法已应用于分割，这些方法通过对特征区域之间的关系进行建模并使用全局信息进行推理，取得了相当大的改进。在本文中，我们提出了一种基于图学习的方法来有效地提取特定于模态的特征并在所有模态之间有效地建立区域对应关系。具体来说，在将特征投影到图域并利用图卷积在所有区域传播信息以学习全局模态特定特征之后，我们提出了一种基于相互信息的图共同注意模块，通过选择性地融合节点特征来学习由图域中具有不同模态的全连接图构造的二分图的权重系数。基于空间图空间之间的变形图和我们提出的图共同注意模块，我们提出了一种多模态先验引导分割框架，该框架针对两种临床情况使用两种策略：特定模态学习策略和共模态学习策略。此外，改进的共模态学习策略与多任务损失中的可训练权重一起使用，以优化所提出的框架。我们在两个多模态 MRI 数据集上验证了我们提出的模块和框架：我们的私人肝脏病变数据集和公共前列腺区域数据集。我们在两个数据集上的实验结果证明了我们提出的方法的优越性。
Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations:Modality-Specific Learning StrategyandCo-Modality Learning Strategy. Besides, the improvedCo-Modality Learning Strategyis used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.