EAGER: Integrating Pathological Image and Biomedical Text Data for Clinical Outcome Prediction

临床结果的准确预测是个性化医疗的一个重要方面，提供了重要的信息，可以制定患者的治疗计划，并最终影响患者的预后。目前的病理分级/分类系统需要人脑进行广泛的信息处理，以解释高度复杂的数据资源。作为疾病诊断的基石，组织学随着技术创新而取得了显着进步，从而可以以更快的速度和分辨率捕获图像。然而，目前大多数组织病理学图像分析方法往往忽略了组织的复杂层次结构。了解各种细胞类型之间的复杂相互作用，这些细胞类型形成细胞成分，进而形成组织结构，对于深入了解生物学和疾病状态至关重要。分析病理图像是至关重要的，但有效地集成相关的生物医学文本数据，如病理报告，初步诊断报告和临床笔记，带来了额外的挑战。这种文本数据被定义为医疗字幕，类似于扩展的图像字幕，其提供必要的上下文，但也在诊断中引入了进一步的复杂性。同时利用病理切片及其字幕的增强计算方法可以彻底改变预测临床结局的准确性和效率。本项目的目标是开发用于临床结局预测的新型病理图文分析工具。该项目将专注于1）开发用于病理图像分析的算法，包括用于亚型细胞分割的自动提示微调框架，细胞级图学习，块级图学习，以及用于临床结果预测的细胞级图和块级图的智能集成; 2）使用生物医学文本数据微调大型语言模型以获得改进的文本嵌入，其中包括开发用于生物医学文本数据分析的算法，结合深度预训练模型的微调，用于精确的生物医学文本数据表示; 3）将病理图像数据与生物医学文本数据集成，用于临床结果预测，其中包括用于智能集成多模态数据和跨模态学习模型的新算法，以从同一患者的组织病理学图像生成生物医学文本数据表示。这些目标的成功实现有望为医疗保健专业人员提供强大的工具，以增强决策过程，个性化治疗计划，并改善患者的整体预后。此外，拟议的研究将为多模态医学数据的整合提供更广泛的见解，为如何在大数据和精准医学时代利用医学信息学制定新标准。该项目的多学科性质也为将其组成部分融入现有课程提供了独特的机会，并激发了K-12学生和代表性不足的学生的科学兴趣。该项目的成果将以同行评审的出版物、开源软件、教程、研讨会和讲习班的形式传播。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

The accurate prediction of clinical outcomes is a critical aspect of personalized medicine, offering vital information that can shape patient treatment plans and ultimately affect patient prognosis. The current pathological grading/classification system requires extensive information processing by a human brain to interpret highly complex data resources. Histopathology, as the cornerstone of disease diagnosis, has advanced significantly with technological innovations allowing for the capture of images at greater speed and resolution. However, most current histopathological image analysis methods often overlook the complex hierarchical structures of tissues. Understanding the intricate interactions among various cell types, which form the cellular components and, in turn, tissue architectures, is crucial for insights into biology and disease status. Analyzing pathological images is crucial, but effectively integrating associated biomedical text data, such as pathological reports, preliminary diagnosis reports, and clinical notes, poses additional challenges. This variety of text data is defined as medical captions, akin to extended image captions, which provide necessary context but also introduce further complexity in diagnostics. Enhanced computational methods that simultaneously leverage pathological slides and their captions could revolutionize the accuracy and efficiency of predicting clinical outcomes.The goal of this project is to develop novel pathological image-text analysis tools for clinical outcome prediction. The project will focus on 1) developing algorithms for pathological image analysis, which include auto-prompting fine-tuning framework for subtype cell segmentation, cell-level graph learning, patch-level graph learning, and intelligent integration of cell-level graph and patch-level graph for clinical outcome prediction; 2) fine-tuning large language models using biomedical text data to obtain improved text embeddings, which include the development of algorithms for biomedical text data analysis, incorporating fine-tuning of deep pre-trained models for precise biomedical text data representation; 3) Integrating pathological image data with biomedical text data for clinical outcome prediction, which include novel algorithms for the intelligent integration of multi-modal data and cross-modal learning models to generate biomedical text data representation from the histopathological images of the same patient. The successful realization of these aims promises to provide healthcare professionals with powerful tools to enhance the decision-making process, personalize treatment plans, and improve overall patient outcomes. Additionally, the proposed study stands to offer broader insights into the integration of multi-modal medical data, setting a new standard for how medical informatics can be leveraged in the era of big data and precision medicine. The multidisciplinary nature of this project also provides unique opportunities for integrating its components into existing curricula, as well as inspiring scientific interests in K-12 students and underrepresented students. The results of this project will be disseminated in the form of peer-reviewed publications, open-source software, tutorials, seminars, and workshops.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.