Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging.
Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging.
复制标题
DOI：
10.2214/ajr.23.29120
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Nooshin Abbasi;Ronilda C. Lacson;Neena Kapoor;Andro Licaros;Jeffrey P. Guenette;Kristine S. Burk;M. Hammer;Sonali Desai;S. Eappen;Sanjay Saini;R. Khorasani
中科院分区：
文献类型：
--
作者：
Nooshin Abbasi;Ronilda C. Lacson;Neena Kapoor;Andro Licaros;Jeffrey P. Guenette;Kristine S. Burk;M. Hammer;Sonali Desai;S. Eappen;Sanjay Saini;R. Khorasani
背景：在放射学报告中进行推荐成像的报告率较低。来自变形金刚的双向编码器表示（BERT）是一种经过预训练以理解语言上下文和歧义的深度学习模型，有可能识别额外成像（RAI）的建议，从而帮助大规模的质量改进工作。目的：开发并外部验证基于人工智能（AI）的模型，用于识别包含RAI的放射学报告。方法：这项回顾性研究在一个多点健康中心进行。随机选择2015年1月1日至2021年6月31日在一个研究中心生成的共计6300份放射学报告，并按4：1的比例分为训练集（n=5040）和测试集（n=1260）。随机抽取2022年4月1日至2022年4月30日期间在该中心其余研究中心（包括学术和社区医院）生成的共计1260份报告作为外部验证组。不同亚专科的转诊医生和放射科医生手动审查了RAI存在的报告印象。一个基于BERT的技术，用于识别RAI开发使用的训练集。在测试集中评估了基于BERT的模型和先前开发的传统机器学习（TLM）模型的性能。最后，在外部验证集中评估性能。模型可公开获取：https://github.com/NooshinAbbasi/Recommendation-for-Additional-Imaging。结果：在7419例独特的患者中，平均年龄为58.8岁; 4133例为女性，3286例为男性。7560份报告中共有10.0%包含RAI。在测试集上，BERT模型的准确率为94%，召回率为98%，F1得分为96%; TML模型的准确率为69%，召回率为65%，F1得分为67%。在测试集中，基于BERT的模型的准确性高于TLM模型（99% vs 93%，p<.001）。在外部验证集中，基于BERT的模型的精确率为99%，召回率为91%，F1得分为95%，准确率为99%。结论：基于BERT的AI模型准确地识别了RAI报告，优于TML模型。外部验证集的高性能表明，其他卫生系统有可能在不需要机构特定培训的情况下适应该模型。临床影响：该模型可以应用于RAI的实时EHR监测或其他改进措施，以帮助确保及时执行临床必要的建议随访。
Background: Reported rates for performing recommended imaging in radiology reports are low. Bidirectional Encoder Representations from Transformers (BERT), a deep-learning model pre-trained to understand language context and ambiguity, has potential to identify recommendations for additional imaging (RAI) and thereby assist large-scale quality improvement efforts. Objective: To develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAI. Methods: This retrospective study was performed at a multisite health center. A total of 6300 radiology reports generated at one site from January 1, 2015 to June 31, 2021 were randomly selected and split by 4:1 ratio to training (n=5040) and test (n=1260) sets. A total of 1260 reports generated at the center's remaining sites (including academic and community hospitals) from April 1, 2022 to April 30, 2022 were randomly selected as an external validation group. Referring practitioners and radiologists of varying subspecialties manually reviewed report impressions for presence of RAI. A BERT-based technique for identifying RAI was developed using the training set. Performance of BERT-based model and a previously developed traditional machine-learning (TLM) model was assessed in the test set. Finally, performance was assessed in the external validation set. Model is publicly available: https://github.com/NooshinAbbasi/Recommendation-for-Additional-Imaging. Results: Among 7419 unique patients, mean age was 58.8 years; 4133 were women, 3286 were men. Total of 10.0% of 7560 reports contained RAI. In test set, BERT-based model showed precision of 94%, recall of 98%, and F1 score of 96%; TML model showed precision of 69%, recall of 65%, and F1 score of 67%. In test set, accuracy was greater for BERT-based than TLM model (99% vs 93%, p<.001). In external validation set, BERT-based model showed precision of 99%, recall of 91%, F1 score of 95%, and accuracy of 99%. Conclusion: The BERT-based AI model accurately identified reports with RAI, outperforming TML model. High performance in the external validation set suggests the potential for other health systems to adapt the model without requiring institution-specific training. Clinical Impact: The model could potentially be applied for real-time EHR monitoring for RAI, or other improvement initiatives, to help ensure timely performance of clinically necessary recommended follow-up.