The reporting quality of natural language processing studies: systematic review of studies of radiology reports.
The reporting quality of natural language processing studies: systematic review of studies of radiology reports.
复制标题
DOI：
10.1186/s12880-021-00671-8
复制
发表时间：
2021-10-02
影响因子：
2.7
通讯作者：
Whiteley W
中科院分区：
文献类型：
--
作者：
Davidson EM;Poon MTC;Casey A;Grivas A;Duma D;Dong H;Suárez-Paniagua V;Grover C;Tobin R;Whalley H;Wu H;Alex B;Whiteley W
使用自然语言处理（NLP）对放射学报告进行自动语言分析可以提供有关患者健康和疾病的有价值信息。随着NLP研究的快速发展，NLP研究应该有透明的方法，以允许方法和再现性的比较。本系统综述旨在总结将NLP应用于放射学报告的研究的特征和报告质量。我们在Google Scholar中检索了2015年1月至2019年10月期间以英语发表的将NLP应用于任何成像模式的放射学报告的研究。至少有两名审查员独立进行筛选并完成数据提取。我们指定了15个标准，涉及数据源、数据集、地面实况、结果和质量评估的再现性。NLP的主要性能指标是精确度，召回率和F1分数。在检索到的4，836条记录中，我们纳入了164项在放射学报告中使用NLP的研究。NLP最常见的临床应用是疾病信息或分类（28%）和诊断监测（27.4%）。大多数研究使用英文放射学报告（86%）。在28%的研究中使用了来自混合成像模式的报告。肿瘤（24%）是最常见的疾病领域。大多数研究的数据集大小> 200（85.4%），但描述其注释，训练，验证和测试集的研究比例分别为67.1%，63.4%，45.7%和67.7%。大约一半的研究报告了准确率（48.8%）和召回率（53.7%）。很少有研究报告进行了外部验证（10.8%），数据可用性（8.5%）和代码可用性（9.1%）。没有与总体报告质量相关的业绩模式。在卫生服务和研究中，放射学报告的NLP有一系列潜在的临床应用。然而，我们发现次优的报告质量，排除了比较，再现性和复制。我们的研究结果支持需要制定针对临床NLP研究的报告标准。在线版本包含补充材料，可通过10.1186/s12880-021-00671-8获得。
Automated language analysis of radiology reports using natural language processing (NLP) can provide valuable information on patients’ health and disease. With its rapid development, NLP studies should have transparent methodology to allow comparison of approaches and reproducibility. This systematic review aims to summarise the characteristics and reporting quality of studies applying NLP to radiology reports. We searched Google Scholar for studies published in English that applied NLP to radiology reports of any imaging modality between January 2015 and October 2019. At least two reviewers independently performed screening and completed data extraction. We specified 15 criteria relating to data source, datasets, ground truth, outcomes, and reproducibility for quality assessment. The primary NLP performance measures were precision, recall and F1 score. Of the 4,836 records retrieved, we included 164 studies that used NLP on radiology reports. The commonest clinical applications of NLP were disease information or classification (28%) and diagnostic surveillance (27.4%). Most studies used English radiology reports (86%). Reports from mixed imaging modalities were used in 28% of the studies. Oncology (24%) was the most frequent disease area. Most studies had dataset size > 200 (85.4%) but the proportion of studies that described their annotated, training, validation, and test set were 67.1%, 63.4%, 45.7%, and 67.7% respectively. About half of the studies reported precision (48.8%) and recall (53.7%). Few studies reported external validation performed (10.8%), data availability (8.5%) and code availability (9.1%). There was no pattern of performance associated with the overall reporting quality. There is a range of potential clinical applications for NLP of radiology reports in health services and research. However, we found suboptimal reporting quality that precludes comparison, reproducibility, and replication. Our results support the need for development of reporting standards specific to clinical NLP studies. The online version contains supplementary material available at 10.1186/s12880-021-00671-8.
登录
查看更多内容
DOI：
10.1016/s2589-7500(20)30218-1
发表时间：
2020-10
期刊：
The Lancet. Digital health
影响因子：
--
作者：
Liu X;Cruz Rivera S;Moher D;Calvert MJ;Denniston AK;SPIRIT-AI and CONSORT-AI Working Group
通讯作者：
SPIRIT-AI and CONSORT-AI Working Group
DOI：
10.1093/jamia/ocv155
发表时间：
2016-04-01
影响因子：
6.4
作者：
Bates, Jonathan;Fodeh, Samah J.;Womack, Julie A.
通讯作者：
Womack, Julie A.
影响因子：
105.7
作者：
Vollmer, Sebastian;Mateen, Bilal A.;Hemingway, Harry
通讯作者：
Hemingway, Harry
影响因子：
105.7
作者：
Rivera, Samantha Cruz;Liu, Xiaoxua;Yau, Christopher
通讯作者：
Yau, Christopher
影响因子：
3.7
作者：
Kim, Chulho;Zhu, Vivienne;Lenert, Leslie
通讯作者：
Lenert, Leslie