Collaborative Research:SCH:Bimodal Interpretable Multi-Instance Medical-Image Classification

该项目专注于创建一个更智能的人工智能（AI）系统，以更好地理解和分析复杂的医学图像，例如来自患者多次扫描的图像。传统方法取得了一些成功，但在处理罕见疾病和提供医生和病人容易理解的解释方面面临挑战。该项目旨在开发一种现代人工智能方法，通过利用大量医学图像和医生笔记来克服这些限制，而不管它们属于特定的健康状况。研究团队将应对各种挑战，使人工智能系统更具可扩展性、可解释性和鲁棒性。这一创新项目将为医务工作者提供值得信赖的人工智能驱动的诊断工具，加快复杂医学图像的诊断过程。这个项目的影响将广泛地体现在人工智能研究和其他领域，因为它的基础研究可能会对各种应用产生影响，而它的使用启发性研究将使现代人工智能方法加速转化为社会效益。实现总体目标的方法是通过可扩展的预训练和微调方法开发一个双峰可解释的多实例医学图像分类框架。该框架包括基于双峰原型的可解释对比预训练，用于从不平衡的未标记数据中学习成对的图像和文本原型，以及通过深度AUC最大化方法进行多实例学习，用于从不平衡的患者级标记数据中学习。为了使对比预训练对不平衡数据具有可扩展性和鲁棒性，研究人员将开发一个基于部分AUC损失的统一框架，该框架不仅统一了现有的对比损失，而且还引入了新的高级全局对比损失。研究团队将利用新的优化工具，开发具有数学保证的改进的随机算法，而不依赖于现有方法的大量批量。为了使多实例学习对不平衡数据具有可扩展性和鲁棒性，研究人员从多级成分优化的角度出发，通过开发随机池化操作，提出了有效的多实例深度AUC最大化随机算法。研究人员不仅将采用标准的性能指标进行评估，还将利用放射科医生的领域专业知识来评估模型的性能和可解释性。调查人员将通过出版物、开源软件、教程、研讨会和课程材料传播结果，此外还将参与外展活动，以加强STEM学习并培养对该领域的更大兴趣。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project focuses on creating a smarter artificial-intelligence (AI) system to better understand and analyze complex medical images, such as those from multiple scans of a patient. Traditional methods have had some success but face challenges in dealing with rare diseases and in providing explanations that doctors and patients can easily understand. This project aims to develop a modern AI approach that overcomes these limitations by leveraging a vast collection of medical images and doctors' notes, regardless of the specific health conditions to which they pertain. The research team will tackle various challenges to make the AI system more scalable, interpretable, and robust. This innovative project will deliver trustworthy AI-driven diagnostic tools to medical workers, expediting the diagnostic process for complex medical images. The impact of this project will be felt broadly in AI research and beyond, as its foundational research is likely to have impact in various applications, and its use-inspired research will enable the accelerated transition of modern AI approaches into benefits for society. The approach to achieve the overarching goal is to develop a bimodal interpretable multi-instance medical image classification framework by a scalable pretraining and finetuning approach. The framework consists of bimodal prototype-based interpretable contrastive pretraining to learn paired image and text prototypes from imbalanced unlabeled data, and multi-instance learning by deep area-under-the-receiver-operator-curve (AUC) maximization methods to learn from imbalanced patient-level labeled data. To make contrastive pretraining scalable and robust to imbalanced data, the investigators will develop a unified framework based on partial AUC losses, which not only unifies the existing contrastive loss but also induces new advanced global contrastive losses. The team of researchers will leverage new optimization tools and develop improved stochastic algorithms with mathematical guarantee without dependence on the large batch size of existing methods. To make multi-instance learning scalable and robust to imbalanced data, the investigators propose efficient stochastic algorithms for multi-instance deep AUC maximization by developing stochastic pooling operations from the lens of multi-level compositional optimization. The investigators will not only employ standard performance metrics for evaluation but will also leverage the domain expertise from radiologists to evaluate model performance and interpretability. The investigators will disseminate results through publications, open-source software, tutorials, workshops, and course materials, additionally engaging in outreach initiatives to enhance STEM learning and foster greater interest in the field.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.