Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis.
Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis.
复制标题
病理融合：融合肿瘤诊断和预后的组织病理学和基因组特征的综合框架。
DOI：
10.1109/tmi.2020.3021387
复制
发表时间：
2022-04
影响因子：
10.6
通讯作者：
中科院分区：
文献类型：
--
作者：
癌症诊断、预后和治疗反应的预测是基于来自组织学切片的形态信息和来自基因组数据的分子图谱。然而，大多数基于深度学习的客观结果预测和分级范例仅基于组织学或基因组学，并没有以直观的方式利用补充信息。在这项工作中，我们提出了一种可解释的组织学图像和基因组(突变，CNV，RNA-Seq)特征端到端多模式融合的策略，用于预测生存结果。我们的方法通过取单峰特征表征的Kronecker积来模拟跨通道的成对特征交互作用，并通过基于门控的注意机制来控制每个表征的表达能力。在有监督的学习之后，我们能够解释并显著地定位每种通道的特征，并理解当条件作用于多通道输入时，特征重要性如何变化。我们使用来自癌症基因组图谱(TCGA)的胶质瘤和肾透明细胞癌数据集来验证我们的方法，该数据集包含配对的完整幻灯片图像、基因和转录组数据，以及基本事实生存和组织学级别标签。在15倍的交叉验证中，我们的结果表明，所提出的多模式融合范例改善了根据地面真相分级和分子亚型以及仅根据组织学和基因组数据训练的单峰深度网络的预后决定。提出的方法为如何以直观的方式训练多模式生物医学数据上的深层网络建立了洞察力和理论，这将对医学中寻求结合异质数据流以了解疾病并预测治疗反应和耐药性的其他问题有用。代码和经过训练的模型可在以下网址获得：https://github.com/mahmoodlab/PathomicFusion.
Cancer diagnosis, prognosis, and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for survival outcome prediction. Our approach models pairwise feature interactions across modalities by taking the Kronecker product of unimodal feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism. Following supervised learning, we are able to interpret and saliently localize features across each modality, and understand how feature importance shifts when conditioning on multimodal input. We validate our approach using glioma and clear cell renal cell carcinoma datasets from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. In a 15-fold cross-validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from ground truth grading and molecular subtyping, as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment. Code and trained models are made available at: https://github.com/mahmoodlab/PathomicFusion.