Disentangle, Align and Fuse for Multimodal and Semi-Supervised Image Segmentation.
Disentangle, Align and Fuse for Multimodal and Semi-Supervised Image Segmentation.
复制标题
用于多模式和半监督图像分割的解开，对齐和融合。
DOI：
10.1109/tmi.2020.3036584
复制
发表时间：
2021-03
影响因子：
10.6
通讯作者：
Tsaftaris SA
中科院分区：
文献类型：
--
作者：
Chartsias A;Papanastasiou G;Wang C;Semple S;Newby DE;Dharmakumar R;Tsaftaris SA
磁共振（MR）协议依赖于几个序列，以正确评估病理和器官状态。尽管在图像分析的进步，我们倾向于处理每个序列，这里称为模态，在隔离。利用模态之间共享的共同信息（器官的解剖结构）有利于多模态处理和学习。然而，我们必须克服固有的解剖配准错误和信号强度的差异，以获得这种好处。我们提出了一种方法，通过学习利用其他模态中存在的信息，即使很少（半监督）或没有（无监督）注释可用于此特定模态，也可以提高感兴趣模态（在单个输入模型上）的分割准确性。我们的方法的核心是学习解剖和成像因素的分解。来自不同输入的共享解剖因素被联合处理和融合以提取更准确的分割掩模。使用空间Transformer网络校正图像配准错误，该网络非线性地对齐解剖因素。成像因子捕获不同模态数据的信号强度特征，并用于图像重建，从而实现半监督学习。动态学习输入之间的时间和切片配对。我们展示了在晚期钆增强（LGE）和血氧水平依赖（BOLD）心脏分割，以及在T2腹部分割的应用。代码可在https://github.com/vios-s/multimodal_segmentation上获得。
Magnetic resonance (MR) protocols rely on several sequences to assess pathology and organ status properly. Despite advances in image analysis, we tend to treat each sequence, here termed modality, in isolation. Taking advantage of the common information shared between modalities (an organ’s anatomy) is beneficial for multi-modality processing and learning. However, we must overcome inherent anatomical misregistrations and disparities in signal intensity across the modalities to obtain this benefit. We present a method that offers improved segmentation accuracy of the modality of interest (over a single input model), by learning to leverage information present in other modalities, even if few (semi-supervised) or no (unsupervised) annotations are available for this specific modality. Core to our method is learning a disentangled decomposition into anatomical and imaging factors. Shared anatomical factors from the different inputs are jointly processed and fused to extract more accurate segmentation masks. Image misregistrations are corrected with a Spatial Transformer Network, which non-linearly aligns the anatomical factors. The imaging factor captures signal intensity characteristics across different modality data and is used for image reconstruction, enabling semi-supervised learning. Temporal and slice pairing between inputs are learned dynamically. We demonstrate applications in Late Gadolinium Enhanced (LGE) and Blood Oxygenation Level Dependent (BOLD) cardiac segmentation, as well as in T2 abdominal segmentation. Code is available at https://github.com/vios-s/multimodal_segmentation.