CAREER: Knowledge-enhanced and interpretable radiology report generation

放射学报告是放射科医生和转诊医生之间沟通的主要手段，也是一份法律文件。到目前为止，许多研究已经证明了使用深度学习从胸部X光自动生成放射学报告的可行性。然而，现有的方法只利用当前的胸部X光图像，而没有考虑历史图像、相关的电子健康记录(EHR)和特定领域的先验知识。因此，目前的计算机生成的报告远远不准确和完整。为了弥补这一差距，迫切需要研究新的报告生成技术来处理大规模的真实医疗数据。该项目将使用新的信息学和数据科学技术来自动生成临床报告，以提高工作流程效率和改善医疗保健结果。从生物医学信息学的角度来看，我们的方法将利用来自EHR的丰富信息来深刻理解自然语言、图像分析和深度学习在报告生成中的作用。从临床翻译的角度来看，该项目将方便放射科医生的工作流程，提高临床准确性和效率，增强决策能力。此外，该项目将通过推出新的研究生自然语言处理和健康课程，并支持几个顶峰和专业化项目，将研究与教育紧密结合起来。它还将扩大调查人员对非计算机科学研究生的接触，他们将通过我们广泛的合作努力接触到NLP的工作原理。该项目将开发和验证一个框架，以使用纵向、多模式电子病历数据和领域知识自动生成放射学报告。调查者将通过追求四个目标来实现总体目标。首先，该项目将建立一个记忆增强的报告生成系统，以处理纵向胸部X光和报告。其次，该项目将从多模式电子病历中构建放射学专用知识图谱，并将其注入报告生成框架。我们将使用一种新的方法来构建这种特定于放射学的知识图，在我们的模型中，通过对异质的多维EHR数据进行建模。第三，我们将创建一个新的基于原理的模型，支持基于原理的可解释性。最后，该项目将构建并评估一个具有用户友好图形用户界面的以用户为中心的原型报告系统。新的报告系统将加强放射科医生和转诊医生之间的沟通，特别是在大型和不同种类的EHR中。这项拟议的研究具有创造性和原创性，因为它代表着朝着建立自动化系统迈出了一步，对放射学知识和决策有了更高的理解。预计它将开辟研究视野，并使用数据科学的技术和理论来支持基于胸部X光和结构化EHR的下一代医疗诊断推理。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The radiology report is the primary mean of communication between radiologists and referring physicians, and which also serve as a legal document. To date, many studies have demonstrated the feasibility of using deep learning to automatically generate radiology reports from chest x-rays. However, existing approaches utilize only current chest x-ray images and do not consider historical images, associated electronic health records (EHRs), and domain-specific prior knowledge. Therefore, the current computer-generated reports are far from accurate and complete. To bridge this gap, there is a critical need to study new report generation techniques to handle large-scale, real-world healthcare data. This project will employ novel informatics and data science techniques to automatically generate clinical reports to improve workflow efficiency and improve healthcare outcomes. From the perspectives of biomedical informatics, our approach will leverage the wealth of information from EHR to profoundly understand the role of natural language, image analysis, and deep learning in report generation. From the perspective of clinical translation, this project will facilitate radiologists’ workflow, improve clinical accuracy and efficiency, and enhance decision-making. Additionally, the project will closely integrate research with education, by launching a new graduate Natural Language Processing and Health course and supporting several capstone and specialization projects. It will also broaden the outreach from the investigators to non-computer-science graduate students, who will be exposed to working principles of NLP through our extensive collaborative efforts. This project will develop and validate a framework to automatically generate radiology reports using longitudinal, multimodal EHR data and domain knowledge. The investigator will attain the overall objective by pursuing four aims. First, the project will build a memory-enhanced report generation system to handle longitudinal chest x-rays and reports. Second, the project will build a radiology-specific knowledge graph from multimodal EHR and inject it into the report generation framework. We will employ a novel approach to construct such radiology-specific knowledge graph, by modeling heterogeneous multi-dimensional EHR data in our model. Third, we will create a new rationale-based model that supports rationale-base interpretabilityFinally, the project will build and evaluate a prototype user-centered reporting system with a user-friendly graphic user interface. The new reporting system will enhance communication between radiologists and referral physicians, particularly in large and heterogeneous EHR. The proposed research is creative and original because it represents a step towards building automatic systems with a higher-level understanding of radiology knowledge and decision-making. It is expected to open research horizons and employ techniques and theories from data science to support next-generation medical diagnostic reasoning from chest x-rays and structured EHR.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.