EAGER: Integrating Pathological Image and Biomedical Text Data for Clinical Outcome Prediction

临床结果的准确预测是个性化医疗的一个关键方面，提供重要的信息，可以制定患者的治疗计划，并最终影响患者的预后。目前的病理分级/分类系统需要人类大脑进行大量的信息处理，以解释高度复杂的数据资源。组织病理学作为疾病诊断的基石，随着技术创新的发展，以更快的速度和更高的分辨率捕获图像已经取得了重大进展。然而，目前大多数组织病理学图像分析方法往往忽略了组织的复杂层次结构。了解各种细胞类型之间复杂的相互作用，这些相互作用形成细胞成分，进而形成组织结构，对于了解生物学和疾病状态至关重要。分析病理图像是至关重要的，但有效地整合相关的生物医学文本数据，如病理报告、初步诊断报告和临床记录，提出了额外的挑战。这种类型的文本数据被定义为医学标题，类似于扩展的图像标题，它提供了必要的上下文，但也进一步增加了诊断的复杂性。同时利用病理切片及其标题的增强计算方法可以彻底改变预测临床结果的准确性和效率。该项目的目标是开发新的病理图像-文本分析工具，用于临床结果预测。该项目将重点发展病理图像分析算法，包括用于亚型细胞分割的自动提示微调框架、细胞级图学习、斑块级图学习以及用于临床结果预测的细胞级图和斑块级图的智能集成；2)利用生物医学文本数据对大型语言模型进行微调，以获得改进的文本嵌入，包括开发生物医学文本数据分析算法，将深度预训练模型进行微调，以实现精确的生物医学文本数据表示；3)整合病理图像数据与生物医学文本数据进行临床预后预测，其中包括多模态数据智能集成的新算法和跨模态学习模型，从同一患者的组织病理图像生成生物医学文本数据表示。这些目标的成功实现有望为医疗保健专业人员提供强大的工具，以增强决策过程，个性化治疗计划，并改善患者的整体结果。此外，拟议的研究将为多模式医疗数据的整合提供更广泛的见解，为如何在大数据和精准医疗时代利用医疗信息学设定新的标准。该项目的多学科性质也为将其组成部分整合到现有课程中提供了独特的机会，并激发了K-12学生和代表性不足的学生的科学兴趣。该项目的成果将以同行评审出版物、开源软件、教程、研讨会和讲习班的形式传播。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The accurate prediction of clinical outcomes is a critical aspect of personalized medicine, offering vital information that can shape patient treatment plans and ultimately affect patient prognosis. The current pathological grading/classification system requires extensive information processing by a human brain to interpret highly complex data resources. Histopathology, as the cornerstone of disease diagnosis, has advanced significantly with technological innovations allowing for the capture of images at greater speed and resolution. However, most current histopathological image analysis methods often overlook the complex hierarchical structures of tissues. Understanding the intricate interactions among various cell types, which form the cellular components and, in turn, tissue architectures, is crucial for insights into biology and disease status. Analyzing pathological images is crucial, but effectively integrating associated biomedical text data, such as pathological reports, preliminary diagnosis reports, and clinical notes, poses additional challenges. This variety of text data is defined as medical captions, akin to extended image captions, which provide necessary context but also introduce further complexity in diagnostics. Enhanced computational methods that simultaneously leverage pathological slides and their captions could revolutionize the accuracy and efficiency of predicting clinical outcomes.The goal of this project is to develop novel pathological image-text analysis tools for clinical outcome prediction. The project will focus on 1) developing algorithms for pathological image analysis, which include auto-prompting fine-tuning framework for subtype cell segmentation, cell-level graph learning, patch-level graph learning, and intelligent integration of cell-level graph and patch-level graph for clinical outcome prediction; 2) fine-tuning large language models using biomedical text data to obtain improved text embeddings, which include the development of algorithms for biomedical text data analysis, incorporating fine-tuning of deep pre-trained models for precise biomedical text data representation; 3) Integrating pathological image data with biomedical text data for clinical outcome prediction, which include novel algorithms for the intelligent integration of multi-modal data and cross-modal learning models to generate biomedical text data representation from the histopathological images of the same patient. The successful realization of these aims promises to provide healthcare professionals with powerful tools to enhance the decision-making process, personalize treatment plans, and improve overall patient outcomes. Additionally, the proposed study stands to offer broader insights into the integration of multi-modal medical data, setting a new standard for how medical informatics can be leveraged in the era of big data and precision medicine. The multidisciplinary nature of this project also provides unique opportunities for integrating its components into existing curricula, as well as inspiring scientific interests in K-12 students and underrepresented students. The results of this project will be disseminated in the form of peer-reviewed publications, open-source software, tutorials, seminars, and workshops.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.