CISE-MSI: DP: IIS:III: Deep Learning Based Automated Concept and Caption Generation of Medical Images Towards Developing an Effective Decision Support System (DSS)

CISE-MSI：DP：IIS：III：基于深度学习的医学图像自动概念和标题生成，以开发有效的决策支持系统 (DSS)

• 批准号: 2131207
• 负责人: Md Rahman
• 金额: $ 43.98万
• 依托单位: Morgan State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2131207&HistoricalAwards=false
• 关键词: CISE; MSI; DP; IIS; III

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Identifying and labeling important features in medical images such as X-rays and ultrasounds is fundamental to both diagnosis itself and to building libraries of images that support education, training, and auditing of medical quality. This work is time-consuming even for trained experts, making it an impactful and important problem domain to study for researchers in computer vision, machine learning (ML), and natural language processing (NLP). These artificial intelligence (AI)-based techniques have made great progress in object recognition and labeling for everyday camera images; however, medical images pose additional challenges because of the need to account for detail and relationships between substructures in the image, the need to generate captions that apply not just to the whole image but to these important substructures, and the need to handle noise and artifacts created in medical image processing. Further, the tolerance for error is low; interpretations need to be coherent, grammatically, and semantically correct in order to be useful. This project focuses on the intersection of biomedical informatics and imaging science, working to develop high quality datasets of human-annotated visual concepts in images that appear in public collections such as open access biomedical journals, then using those datasets to train novel vision, ML, and NLP algorithms. The work will support multi-institutional research and educational collaborations between three minority-serving institutions, providing advanced research and classroom training in AI, ML, and cloud computing to students from groups historically underrepresented in computing. To improve image interpretation and retrieval effectiveness, this project will (1) create a crowdsourcing-based annotation system to clinically annotate important regions of interest (ROIs) of images; (2) advance object detection models to segment images and map medical image ROIs; (3) advance multilabel concept classification techniques by considering correlations between concepts; and (4) apply contextualized embeddings via deep language models to generate the captions. The proposed approaches will be evaluated through comparison with current methods in benchmark datasets, including the ones constructed for this project. The end goal is the development of an AI-based prototype that helps physicians focus on interesting image regions, find relevant comparison images, and describe findings in correct and standard ways, all of which can reduce medical errors and benefit both medical departments and society by reducing the cost per exam. In addition to the research objectives, the project will implement a research-education medical AI training program including cloud-enabled classrooms, cross-institutional mentoring, and partnering with an existing industry internship “pathway to success” initiative to build the science and technology workforce of the future.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。识别和标记医学图像（如X射线和超声波）中的重要特征对于诊断本身以及构建支持教育，培训和医疗质量审计的图像库至关重要。这项工作即使对于训练有素的专家来说也是耗时的，这使得它成为计算机视觉，机器学习（ML）和自然语言处理（NLP）研究人员研究的一个有影响力和重要的问题领域。这些基于人工智能（AI）的技术在日常相机图像的物体识别和标记方面取得了很大进展;然而，由于需要考虑图像中的子结构之间的细节和关系，需要生成不仅适用于整个图像而且适用于这些重要子结构的字幕，以及需要处理医学图像处理中产生的噪声和伪影。此外，对错误的容忍度很低;为了有用，解释需要连贯，语法和语义正确。该项目专注于生物医学信息学和成像科学的交叉，致力于开发高质量的人类注释视觉概念数据集，这些图像出现在公共收藏品中，如开放获取生物医学期刊，然后使用这些数据集来训练新的视觉，ML和NLP算法。这项工作将支持三个少数民族服务机构之间的多机构研究和教育合作，为来自历史上在计算领域代表性不足的群体的学生提供人工智能，机器学习和云计算方面的高级研究和课堂培训。为了提高图像解释和检索的有效性，该项目将（1）创建一个基于众包的注释系统，以临床注释图像的重要感兴趣区域（ROI）;（2）改进对象检测模型，以分割图像并映射医学图像ROI;（3）通过考虑概念之间的相关性，改进多标签概念分类技术;以及（4）经由深度语言模型应用上下文化嵌入以生成字幕。将通过与基准数据集（包括为本项目构建的数据集）中的当前方法进行比较，对所提出的方法进行评估。 最终目标是开发一个基于AI的原型，帮助医生专注于感兴趣的图像区域，找到相关的比较图像，并以正确和标准的方式描述发现，所有这些都可以减少医疗错误，并通过降低每次检查的成本使医疗部门和社会受益。除了研究目标，该项目将实施一项研究教育医疗人工智能培训计划，包括云课堂，跨机构指导，并与现有的行业实习“成功之路”合作，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准。

项目成果

Concept Detection and Caption Prediction in ImageCLEFmedical Caption 2023 with Convolutional Neural Networks; Vision and Text-to-Text Transfer Transformers

使用卷积神经网络进行 ImageCLEFmedical Caption 2023 中的概念检测和字幕预测；

• 发表时间: 2023
• 期刊: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023
• 作者: Hasan M;Layode O;Rahman M.
• 通讯作者: Rahman M.

CS_Morgan at ImageCLEFmedical 2022 Caption Task: Deep Learning Based Multi-Label Classification and Transformers for Concept Detection & Caption Prediction

CS_Morgan 在 ImageCLEFmedical 2022 标题任务：基于深度学习的多标签分类和用于概念检测的 Transformers

• 发表时间: 2022
• 期刊: CEUR Workshop Proceedings (CEUR-WS.org
• 作者: Rahman, Md M.;Layode, O.
• 通讯作者: Layode, O.

Statistical Analysis of Imbalanced Classification with Training Size Variation and Subsampling on Datasets of Research Papers in Biomedical Literature

生物医学文献研究论文数据集训练规模变化和子采样的不平衡分类统计分析

• 发表时间: 2023
• 期刊: Machine Learning and Knowledge Extraction
• 影响因子: 3.9
• 作者: Dixon, J;Rahman, M.
• 通讯作者: Rahman, M.

Media Interestingness Prediction in ImageCLEFfusion 2023 with Dense Architecture-based Ensemble & Scaled; Gradient Boosting Regressor Model

使用基于密集架构的集成在 ImageCLEFfusion 2023 中进行媒体兴趣度预测

• 发表时间: 2023
• 期刊: CEUR workshop proceedings
• 作者: Emon, M;Rahman, M.
• 通讯作者: Rahman, M.