CISE-MSI: DP: IIS:III: Deep Learning Based Automated Concept and Caption Generation of Medical Images Towards Developing an Effective Decision Support System (DSS)

CISE-MSI：DP：IIS：III：基于深度学习的医学图像自动概念和标题生成，以开发有效的决策支持系统 (DSS)

• 批准号: 2131207
• 负责人: Md Rahman
• 金额: $ 43.98万
• 依托单位: Morgan State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2131207&HistoricalAwards=false
• 关键词: CISE; MSI; DP; IIS; III

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Identifying and labeling important features in medical images such as X-rays and ultrasounds is fundamental to both diagnosis itself and to building libraries of images that support education, training, and auditing of medical quality. This work is time-consuming even for trained experts, making it an impactful and important problem domain to study for researchers in computer vision, machine learning (ML), and natural language processing (NLP). These artificial intelligence (AI)-based techniques have made great progress in object recognition and labeling for everyday camera images; however, medical images pose additional challenges because of the need to account for detail and relationships between substructures in the image, the need to generate captions that apply not just to the whole image but to these important substructures, and the need to handle noise and artifacts created in medical image processing. Further, the tolerance for error is low; interpretations need to be coherent, grammatically, and semantically correct in order to be useful. This project focuses on the intersection of biomedical informatics and imaging science, working to develop high quality datasets of human-annotated visual concepts in images that appear in public collections such as open access biomedical journals, then using those datasets to train novel vision, ML, and NLP algorithms. The work will support multi-institutional research and educational collaborations between three minority-serving institutions, providing advanced research and classroom training in AI, ML, and cloud computing to students from groups historically underrepresented in computing. To improve image interpretation and retrieval effectiveness, this project will (1) create a crowdsourcing-based annotation system to clinically annotate important regions of interest (ROIs) of images; (2) advance object detection models to segment images and map medical image ROIs; (3) advance multilabel concept classification techniques by considering correlations between concepts; and (4) apply contextualized embeddings via deep language models to generate the captions. The proposed approaches will be evaluated through comparison with current methods in benchmark datasets, including the ones constructed for this project. The end goal is the development of an AI-based prototype that helps physicians focus on interesting image regions, find relevant comparison images, and describe findings in correct and standard ways, all of which can reduce medical errors and benefit both medical departments and society by reducing the cost per exam. In addition to the research objectives, the project will implement a research-education medical AI training program including cloud-enabled classrooms, cross-institutional mentoring, and partnering with an existing industry internship “pathway to success” initiative to build the science and technology workforce of the future.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项是根据2021年《美国救援计划法》（公法117-2）全部或部分资助的。识别和标记医学图像中的重要特征，例如X射线和超声波，对诊断本身以及支持教育，培训和审核医学质量的图像的图书馆都是至关重要的。这项工作即使对于训练有素的专家来说也很耗时，这使其成为一个有影响力且重要的问题领域，可以研究计算机视觉，机器学习（ML）和自然语言处理（NLP）的研究人员。这些基于人工智能（AI）的技术在每天的相机图像方面取得了重大进展。但是，医疗图像构成了其他挑战，因为需要考虑图像中的细节和子结构之间的关系，需要生成不仅适用于整个图像的字幕，还适用于这些重要的子结构，以及处理医学图像处理中创建的噪声和工件的需要。此外，误差的公差很低。为了有用，解释需要相干，语法和语义上正确。该项目的重点是生物医学信息和成像科学的交集，旨在在图像中开发高质量的人类宣传的视觉概念数据集，这些图像出现在公共集合中，例如开放式生物医学期刊，然后使用这些数据集来培训新颖的视觉，ML，ML和NLP算法。这项工作将支持三个少数族裔服务机构之间的多机构研究和教育合作，从而向来自历史上代表性不足的计算机的学生提供了AI，ML和云计算的高级研究和课堂培训。为了提高图像解释和检索有效性，该项目将（1）创建一个基于人群的注释系统，以临床注释图像的重要区域（ROI）； （2）预先对象检测模型分割图像并映射医疗图像ROI； （3）通过考虑概念之间的相关性来推进多标签概念分类技术； （4）通过深层语言模型应用上下文化的嵌入以生成字幕。提出的方法将通过与基准数据集中的当前方法进行比较，包括为该项目构建的方法。最终目标是开发基于AI的原型，该原型可帮助医生专注于有趣的图像区域，查找相关的比较图像，并以正确和标准的方式描述发现，所有这些都可以通过降低每次考试成本来减少医疗错误和受益的医疗部门和社会。除研究目标外，该项目还将实施一项研究教育医学AI培训计划，包括支持云的教室，跨机构心理，并与现有的行业实习“成功途径”倡议合作，以建立未来的科学和技术劳动力。这一奖项通过评估律师的构建委员会的构建授予的构建委员会的知识范围构成了诚实的范围。

项目成果

Concept Detection and Caption Prediction in ImageCLEFmedical Caption 2023 with Convolutional Neural Networks; Vision and Text-to-Text Transfer Transformers

使用卷积神经网络进行 ImageCLEFmedical Caption 2023 中的概念检测和字幕预测；

• 发表时间: 2023
• 期刊: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023
• 作者: Hasan M;Layode O;Rahman M.
• 通讯作者: Rahman M.

CS_Morgan at ImageCLEFmedical 2022 Caption Task: Deep Learning Based Multi-Label Classification and Transformers for Concept Detection & Caption Prediction

CS_Morgan 在 ImageCLEFmedical 2022 标题任务：基于深度学习的多标签分类和用于概念检测的 Transformers

• 发表时间: 2022
• 期刊: CEUR Workshop Proceedings (CEUR-WS.org
• 作者: Rahman, Md M.;Layode, O.
• 通讯作者: Layode, O.

Media Interestingness Prediction in ImageCLEFfusion 2023 with Dense Architecture-based Ensemble & Scaled; Gradient Boosting Regressor Model

使用基于密集架构的集成在 ImageCLEFfusion 2023 中进行媒体兴趣度预测

• 发表时间: 2023
• 期刊: CEUR workshop proceedings
• 作者: Emon, M;Rahman, M.
• 通讯作者: Rahman, M.

Statistical Analysis of Imbalanced Classification with Training Size Variation and Subsampling on Datasets of Research Papers in Biomedical Literature

生物医学文献研究论文数据集训练规模变化和子采样的不平衡分类统计分析

• 发表时间: 2023
• 期刊: Machine Learning and Knowledge Extraction
• 影响因子: 3.9
• 作者: Dixon, J;Rahman, M.
• 通讯作者: Rahman, M.