Machine learning for medical imaging: automated disease diagnosis and prognosis

深度学习是一类机器学习算法，在我们今年最近的几项研究中取得了令人印象深刻的结果。除了应用于自然语言处理之外，我们还看到了它在医学图像分析中的成功，例如处理胸部X射线，CT图像和各种视网膜图像，用于自主疾病诊断和预后。与我们在NIH校园的临床合作者一起，我们之前已经对超过100，000份放射学报告进行了文本挖掘，我们的算法生成了弱训练标签，从而能够开发用于自动阅读和分类胸部X射线图像的高级深度学习方法。这项工作的结果是向科学界发布了ChestX-ray 14（https：//nihcc.app.box.com/v/ChestXray-NIHCC），这是最大的公开可用胸部X射线数据集之一，下载量超过10，000次。我们还进行了研究，以协助筛查年龄相关性黄斑变性（AMD），这是60岁及以上美国人视力丧失的主要原因。通过利用尖端的深度学习技术和重新利用来自主要AMD临床试验的大成像数据，我们之前开发了一种用于自主AMD诊断的新型数据驱动方法（DeepSeeNet），其性能超过了人类眼科医生（在这种情况下是视网膜专家）。这一结果凸显了深度学习系统在帮助早期疾病检测和增强临床决策过程方面的潜力。2023年，我们继续进行这方面的研究，重点是其他疾病，例如一种称为地图状萎缩（GA）的晚期AMD的特定形式，以及放射学中的新成像模式，例如光学相干断层扫描（OCT）图像。 地图状萎缩（GA）是晚期AMD萎缩形式的定义性病变。据预测，到2040年，萎缩性AMD将影响全球> 500万人。GA的检测在临床实践和研究中具有重要意义。GA的诊断和严重程度分类通常需要眼科医生亲自评估。这可能会限制GA评估的可及性，特别是对于生活在偏远地区或眼科医生很少的国家的个人。在这种情况下，GA诊断和分类的自动化方法具有重要的潜在优势，包括速度和可访问性，以及准确性和一致性。与NEI的Chew博士和她的同事一起，我们使用了来自311名参与者的1284次SD-OCT扫描来开发Deep-GA-Net：一种具有3D注意力层的三维（3d）深度学习网络，用于检测谱域OCT（SD-OCT）扫描的GA。交叉验证用于评估Deep-GA-Net，其中每个测试集不包含来自相应训练集的参与者。使用B扫描水平的正面热图和重要区域来可视化Deep-GA-Net的输出，并且3名眼科医生对其中GA的存在或不存在进行分级以评估可解释性（即，可理解性和可解释性）。与其他网络相比，Deep-GA-Net实现了最佳指标，准确率为0.93，AUC为0.94，APR为0.91，并且在正面热图和B扫描分级任务中分别获得了0.98和0.68的最佳分级。总之，Deep-GA-Net能够从SD-OCT扫描中准确检测GA。Deep-GA-Net的可视化更易于解释，如3名眼科医生所建议的。 除了自动疾病诊断，我们的深度学习方法还可以为临床研究做出重大贡献。例如，我们之前的DeepSeeNet软件在最近的临床研究中发挥了关键作用，以确定网状假性玻璃疣（RPD）状态，ARMS 2/HTRA 1基因型或两者是否与改变的地图状萎缩（GA）扩大率相关，并分析RPD状态对遗传效应的潜在介导。 在放射学方面，我们专注于自动预填充放射学报告，这是一项重要的临床任务，尽管过去进行了各种尝试，但仍然具有挑战性。与美国国立卫生研究院临床中心的萨默斯博士和他的团队一起，我们提议使用纵向多模态数据，即，既往患者访视CXR、当前访视CXR和既往访视报告，以预先填写当前患者访视报告的结果部分。我们首先从MIMIC-CXR数据集中收集了26，625名患者的纵向访视信息，并创建了一个名为Longitudinal-MIMIC的新数据集。利用该新数据集，训练基于变换器的模型，以通过基于交叉注意的多模态融合模块和分层存储器驱动的解码器从包含多模态数据（CXR图像+报告）的纵向患者就诊记录中捕获信息。与以前仅使用当前访问数据作为输入来训练模型的工作相比，我们的工作利用了可用于预填充放射学报告的发现部分的纵向信息。实验结果表明，该方法在F1评分、BLEU-4评分、METEOR评分和ROUGE-L评分上均明显优于现有的几种方法。

Deep learning, a class of machine learning algorithms, has showed impressive results in several of our recent studies this year. In addition to its application to natural language processing, we have also seen its success in our medical image analysis such as processing chest X-rays, CT images, and various kinds of retinal images for autonomous disease diagnosis and prognosis. Together with our clinical collaborators on the NIH campus, we have previously text-mined over 100,000 radiology reports where our algorithm generated weak training labels to enable the development of advanced deep learning methods for automatically reading and classifying chest X-ray images. This work resulted in the release to the scientific community of ChestX-ray14 (https://nihcc.app.box.com/v/ChestXray-NIHCC), one of the largest publicly available chest x-ray datasets with over 10,000 downloads. We have also conducted research to assist in the screening of age-related macular degeneration (AMD), a leading cause of vision loss in Americans 60 and older. By leveraging cutting-edge deep learning techniques and repurposing big imaging data from a major AMD clinical trial, we previously developed a novel data-driven approach (DeepSeeNet) for autonomous AMD diagnosis that exceeded the performance of human ophthalmologists (retinal specialists in this case). Such a result highlights the potential of deep learning systems to assist in early disease detection and enhance the clinical decision-making processes. In 2023, we continued this line of research with an emphasis on other diseases such as a specific form of late AMD called geographic atrophy (GA) and new imaging modalities in radiology such as optical coherence tomograph (OCT) images. Geographic atrophy (GA) is the defining lesion of the atrophic form of late AMD. It is predicted that, by 2040, atrophic AMD will affect > 5 million people worldwide. The detection of GA has important implications in both clinical practice and research. The diagnosis and severity classification of GA typically require in-person evaluation by an ophthalmologist. This may limit the accessibility of GA assessment, particularly for individuals living in remote areas or in countries with few ophthalmologists. In this context, automated approaches to GA diagnosis and classification have important potential advantages, including speed and accessibility, as well as accuracy and consistency. With Dr. Chew and her colleagues at NEI, we used 1284 SD-OCT scans from 311 participants to develop Deep-GA-Net: a 3-dimentaional (3d) deep learning network with 3D attention layer, for the detection of GA on spectral domain OCT (SD-OCT) scans. Cross-validation was used to evaluate Deep-GA-Net, where each testing set contained no participant from the corresponding training set. En face heatmaps and important regions at the B-scan level were used to visualize the outputs of Deep-GA-Net, and 3 ophthalmologists graded the presence or absence of GA in them to assess the explainability (i.e., understandability and interpretability) of its detections. Compared with other networks, Deep-GA-Net achieved the best metrics, with accuracy of 0.93, AUC of 0.94, and APR of 0.91, and received the best gradings of 0.98 and 0.68 on the en face heatmap and B-scan grading tasks, respectively. In summary, Deep-GA-Net was able to detect GA accurately from SD-OCT scans. The visualizations of Deep-GA-Net were more explainable, as suggested by 3 ophthalmologists. In addition to automatic disease diagnosis, our deep learning approach can significantly contribute to clinical research. For example, our previous DeepSeeNet software played a critical role in the recent clinical investigation to determine whether reticular pseudodrusen (RPD) status, ARMS2/HTRA1 genotype, or both are associated with altered geographic atrophy (GA) enlargement rate and to analyze potential mediation of genetic effects by RPD status. In radiology, we focused on automatically pre-filling radiology reports, an important clinical task that remains challenging despite various attempts in the past. Together with Dr. Summers and his team at the NIH clinical center, we proposed to use longitudinal multi-modal data, i.e., previous patient visit CXR, current visit CXR, and previous visit report, to pre-fill the findings section of a current patient visit report. We first gathered the longitudinal visit information for 26,625 patients from the MIMIC-CXR dataset and created a new dataset called Longitudinal-MIMIC. With this new dataset, a transformer-based model was trained to capture the in- formation from longitudinal patient visit records containing multi-modal data (CXR images + reports) via a cross-attention-based multi-modal fusion module and a hierarchical memory-driven decoder. In contrast to previous work that only uses current visit data as input to train a model, our work exploits the longitudinal information available to pre-fill the findings section of radiology reports. Experiments show that our approach outperforms several recent approaches significantly on F1 score, BLEU-4, METEOR and ROUGE-L respectively.