CAREER: Knowledge-enhanced and interpretable radiology report generation

职业：知识增强和可解释的放射学报告生成

• 批准号: 2145640
• 负责人: Yifan Peng
• 金额: $ 59.7万
• 依托单位: Joan and Sanford I. Weill Medical College of Cornell University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-07-15 至 2027-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2145640&HistoricalAwards=false
• 关键词: CAREER; Knowledge; enhanced; interpretable; radiology

The radiology report is the primary mean of communication between radiologists and referring physicians, and which also serve as a legal document. To date, many studies have demonstrated the feasibility of using deep learning to automatically generate radiology reports from chest x-rays. However, existing approaches utilize only current chest x-ray images and do not consider historical images, associated electronic health records (EHRs), and domain-specific prior knowledge. Therefore, the current computer-generated reports are far from accurate and complete. To bridge this gap, there is a critical need to study new report generation techniques to handle large-scale, real-world healthcare data. This project will employ novel informatics and data science techniques to automatically generate clinical reports to improve workflow efficiency and improve healthcare outcomes. From the perspectives of biomedical informatics, our approach will leverage the wealth of information from EHR to profoundly understand the role of natural language, image analysis, and deep learning in report generation. From the perspective of clinical translation, this project will facilitate radiologists’ workflow, improve clinical accuracy and efficiency, and enhance decision-making. Additionally, the project will closely integrate research with education, by launching a new graduate Natural Language Processing and Health course and supporting several capstone and specialization projects. It will also broaden the outreach from the investigators to non-computer-science graduate students, who will be exposed to working principles of NLP through our extensive collaborative efforts. This project will develop and validate a framework to automatically generate radiology reports using longitudinal, multimodal EHR data and domain knowledge. The investigator will attain the overall objective by pursuing four aims. First, the project will build a memory-enhanced report generation system to handle longitudinal chest x-rays and reports. Second, the project will build a radiology-specific knowledge graph from multimodal EHR and inject it into the report generation framework. We will employ a novel approach to construct such radiology-specific knowledge graph, by modeling heterogeneous multi-dimensional EHR data in our model. Third, we will create a new rationale-based model that supports rationale-base interpretabilityFinally, the project will build and evaluate a prototype user-centered reporting system with a user-friendly graphic user interface. The new reporting system will enhance communication between radiologists and referral physicians, particularly in large and heterogeneous EHR. The proposed research is creative and original because it represents a step towards building automatic systems with a higher-level understanding of radiology knowledge and decision-making. It is expected to open research horizons and employ techniques and theories from data science to support next-generation medical diagnostic reasoning from chest x-rays and structured EHR.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

放射学报告是放射科医师和转诊医师之间沟通的主要手段，也是一份法律的文件。到目前为止，许多研究已经证明了使用深度学习自动生成胸部X光片放射学报告的可行性。然而，现有的方法仅利用当前的胸部X射线图像，并且不考虑历史图像、相关联的电子健康记录（EHR）和特定领域的先验知识。因此，目前计算机生成的报告远远不够准确和完整。为了弥合这一差距，迫切需要研究新的报告生成技术来处理大规模的真实医疗数据。该项目将采用新的信息学和数据科学技术来自动生成临床报告，以提高工作流程效率并改善医疗结果。从生物医学信息学的角度来看，我们的方法将利用EHR的丰富信息，深刻理解自然语言、图像分析和深度学习在报告生成中的作用。从临床翻译的角度来看，该项目将方便放射科医生的工作流程，提高临床准确性和效率，并加强决策。此外，该项目将通过推出新的研究生自然语言处理和健康课程，并支持几个顶点和专业化项目，将研究与教育紧密结合起来。它还将扩大从调查人员到非计算机科学研究生的范围，通过我们广泛的合作努力，他们将接触到NLP的工作原理。该项目将开发和验证一个框架，使用纵向，多模式EHR数据和领域知识自动生成放射学报告。调查员将通过追求四个目标来实现总体目标。首先，该项目将建立一个内存增强的报告生成系统，以处理纵向胸部X光片和报告。其次，该项目将从多模式EHR构建放射学特定的知识图，并将其注入报告生成框架。我们将采用一种新的方法来构建这样的放射学特定的知识图，通过在我们的模型中建模异构多维EHR数据。第三，我们将创建一个新的基于理性的模型，支持理性的解释性最后，该项目将建立和评估一个原型用户为中心的报告系统与用户友好的图形用户界面。新的报告系统将加强放射科医生和转诊医生之间的沟通，特别是在大型和异构的电子病历。拟议的研究具有创造性和原创性，因为它代表了建立对放射学知识和决策有更高水平理解的自动系统的一步。该奖项旨在开拓研究视野，利用数据科学的技术和理论，支持下一代医学诊断推理，包括胸部X光和结构化电子病历。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

How Does Pruning Impact Long-Tailed Multi-Label Medical Image Classifiers?

• DOI: 10.48550/arxiv.2308.09180
• 发表时间: 2023-08
• 期刊: ArXiv
• 作者: G. Holste;Ziyu Jiang;A. Jaiswal;M. Hanna;Shlomo Minkowitz;A. Legasto;J. Escalon;Sharon Steinberger;M. Bittman;Thomas C. Shen;Ying Ding;R. M. Summers;G. Shih;Yifan Peng;Zhangyang Wang

Enhancing thoracic disease detection using chest X-rays from PubMed Central Open Access

• DOI: 10.1016/j.compbiomed.2023.106962
• 发表时间: 2023-04-23
• 期刊: COMPUTERS IN BIOLOGY AND MEDICINE
• 影响因子: 7.7
• 作者: Lin, Mingquan;Hou, Bojian;Peng, Yifan
• 通讯作者: Peng, Yifan

Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reports

• DOI: 10.48550/arxiv.2306.08749
• 发表时间: 2023-06
• 期刊: ArXiv
• 作者: Qingqing Zhu;T. Mathai;P. Mukherjee;Yifan Peng;R. M. Summers;Zhiyong Lu

RoS-KD: A Robust Stochastic Knowledge Distillation Approach for Noisy Medical Imaging

RoS-KD：一种用于噪声医学成像的鲁棒随机知识蒸馏方法

• DOI: 10.1109/icdm54844.2022.00118
• 发表时间: 2022
• 期刊: 2022 IEEE International Conference on Data Mining (ICDM
• 作者: Jaiswal, Ajay;Ashutosh, Kumar;Rousseau, Justin F.;Peng, Yifan;Wang, Zhangyang;Ding, Ying
• 通讯作者: Ding, Ying

Adopting and expanding ethical principles for generative artificial intelligence from military to healthcare.

采用和扩展从军事到医疗保健领域的生成人工智能的道德原则。

• DOI: 10.1038/s41746-023-00965-x
• 发表时间: 2023-12-02
• 期刊: NPJ digital medicine
• 影响因子: 15.2