CISE-MSI: DP: IIS:III: Deep Learning Based Automated Concept and Caption Generation of Medical Images Towards Developing an Effective Decision Support System (DSS)

该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。识别和标记医学图像中的重要特征，如X射线和超声波，对于诊断本身和建立支持医疗质量教育、培训和审计的图像库都是基本的。这项工作即使对于训练有素的专家来说也是非常耗时的，这使得它成为计算机视觉、机器学习(ML)和自然语言处理(NLP)研究人员研究的一个有影响力的重要问题领域。这些基于人工智能(AI)的技术在日常相机图像的对象识别和标记方面取得了很大进展；然而，由于需要考虑图像中子结构之间的细节和关系，需要生成不仅适用于整个图像而且适用于这些重要子结构的字幕，以及需要处理医学图像处理中产生的噪声和伪影，医学图像带来了额外的挑战。此外，对错误的容忍度很低；解释需要连贯、语法和语义正确，才能有用。该项目专注于生物医学信息学和成像科学的交叉，致力于开发出现在开放获取生物医学期刊等公共收藏中的图像中人类注释视觉概念的高质量数据集，然后使用这些数据集训练新的视觉、ML和NLP算法。这项工作将支持三所为少数族裔服务的机构之间的多机构研究和教育合作，为来自计算领域历史上代表性不足的群体的学生提供人工智能、ML和云计算方面的高级研究和课堂培训。为了提高图像解释和检索的有效性，本项目将(1)创建一个基于众包的标注系统来对图像的重要感兴趣区域(ROI)进行临床标注；(2)改进对象检测模型来分割图像并映射医学图像ROI；(3)通过考虑概念之间的相关性来改进多标签概念分类技术；(4)通过深层语言模型应用上下文嵌入来生成字幕。将通过与基准数据集中的当前方法进行比较，包括为本项目构建的方法，对拟议的方法进行评估。最终目标是开发一种基于人工智能的原型，帮助医生专注于感兴趣的图像区域，找到相关的比较图像，并以正确和标准的方式描述结果，所有这些都可以减少医疗差错，并通过降低每次检查的成本造福于医疗部门和社会。除了研究目标，该项目还将实施研究教育医学人工智能培训计划，包括支持云的课堂、跨机构指导，以及与现有的行业实习生“成功之路”计划合作，以建设未来的科学技术队伍。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Identifying and labeling important features in medical images such as X-rays and ultrasounds is fundamental to both diagnosis itself and to building libraries of images that support education, training, and auditing of medical quality. This work is time-consuming even for trained experts, making it an impactful and important problem domain to study for researchers in computer vision, machine learning (ML), and natural language processing (NLP). These artificial intelligence (AI)-based techniques have made great progress in object recognition and labeling for everyday camera images; however, medical images pose additional challenges because of the need to account for detail and relationships between substructures in the image, the need to generate captions that apply not just to the whole image but to these important substructures, and the need to handle noise and artifacts created in medical image processing. Further, the tolerance for error is low; interpretations need to be coherent, grammatically, and semantically correct in order to be useful. This project focuses on the intersection of biomedical informatics and imaging science, working to develop high quality datasets of human-annotated visual concepts in images that appear in public collections such as open access biomedical journals, then using those datasets to train novel vision, ML, and NLP algorithms. The work will support multi-institutional research and educational collaborations between three minority-serving institutions, providing advanced research and classroom training in AI, ML, and cloud computing to students from groups historically underrepresented in computing. To improve image interpretation and retrieval effectiveness, this project will (1) create a crowdsourcing-based annotation system to clinically annotate important regions of interest (ROIs) of images; (2) advance object detection models to segment images and map medical image ROIs; (3) advance multilabel concept classification techniques by considering correlations between concepts; and (4) apply contextualized embeddings via deep language models to generate the captions. The proposed approaches will be evaluated through comparison with current methods in benchmark datasets, including the ones constructed for this project. The end goal is the development of an AI-based prototype that helps physicians focus on interesting image regions, find relevant comparison images, and describe findings in correct and standard ways, all of which can reduce medical errors and benefit both medical departments and society by reducing the cost per exam. In addition to the research objectives, the project will implement a research-education medical AI training program including cloud-enabled classrooms, cross-institutional mentoring, and partnering with an existing industry internship “pathway to success” initiative to build the science and technology workforce of the future.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.