Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements.
Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements.
复制标题
DOI：
10.1136/amiajnl-2013-001837
复制
发表时间：
2014-05
期刊：
影响因子：
--
通讯作者：
Solti I
中科院分区：
文献类型：
--
作者：
Lingren T;Deleger L;Molnar K;Zhai H;Meinzen-Derr J;Kaiser M;Stoutenborough L;Li Q;Solti I
提出一系列实验：（1）评价预注释对临床试验公告手动注释速度的影响;（2）如果使用预注释，则测试潜在偏倚。为了建立金标准，从clinicaltrials.gov网站随机选择1400个临床试验公告，并对诊断、体征、症状、统一医学语言系统（UMLS）概念唯一标识符和SNOMED CT代码进行双重注释。我们使用了两种基于词典的方法来预注释文本。我们通过F-测度和方差分析测试评估注释时间和潜在偏倚，并实施Bonferroni校正。每个实体节省的时间从13.85%到21.5%不等。注释者间一致性（IAA）范围为93.4%至95.5%。IAA和注释者在预注释中的表现没有统计学显著差异。在每一个实验对，与预注释文本的注释者需要更少的时间来注释比未标记的文本的注释者。节省的时间具有统计学意义。此外，预注释并没有降低IAA或注释器的性能。基于词典的预标注是一种可行且实用的方法，可以在不引入偏倚的情况下降低临床试验公告合格性部分中临床命名实体识别的标注成本。
To present a series of experiments: (1) to evaluate the impact of pre-annotation on the speed of manual annotation of clinical trial announcements; and (2) to test for potential bias, if pre-annotation is utilized. To build the gold standard, 1400 clinical trial announcements from the clinicaltrials.gov website were randomly selected and double annotated for diagnoses, signs, symptoms, Unified Medical Language System (UMLS) Concept Unique Identifiers, and SNOMED CT codes. We used two dictionary-based methods to pre-annotate the text. We evaluated the annotation time and potential bias through F-measures and ANOVA tests and implemented Bonferroni correction. Time savings ranged from 13.85% to 21.5% per entity. Inter-annotator agreement (IAA) ranged from 93.4% to 95.5%. There was no statistically significant difference for IAA and annotator performance in pre-annotations. On every experiment pair, the annotator with the pre-annotated text needed less time to annotate than the annotator with non-labeled text. The time savings were statistically significant. Moreover, the pre-annotation did not reduce the IAA or annotator performance. Dictionary-based pre-annotation is a feasible and practical method to reduce the cost of annotation of clinical named entity recognition in the eligibility sections of clinical trial announcements without introducing bias in the annotation process.
登录
查看更多内容
DOI：
10.1136/amiajnl-2012-001487
发表时间：
2013-09
期刊：
Journal of the American Medical Informatics Association : JAMIA
影响因子：
--
作者：
Li Q;Zhai H;Deleger L;Lingren T;Kaiser M;Stoutenborough L;Solti I
通讯作者：
Solti I
DOI：
10.1136/amiajnl-2012-001317
发表时间：
2013-09
期刊：
Journal of the American Medical Informatics Association : JAMIA
影响因子：
--
作者：
Albright D;Lanfranchi A;Fredriksen A;Styler WF 4th;Warner C;Hwang JD;Choi JD;Dligach D;Nielsen RD;Martin J;Ward W;Palmer M;Savova GK
通讯作者：
Savova GK
影响因子：
5.8
作者：
Salgado, David;Krallinger, Martin;Marcelle, Christophe
通讯作者：
Marcelle, Christophe
影响因子：
5.8
作者：
Settles, B
通讯作者：
Settles, B
影响因子：
--
作者：
BLAND, JM;ALTMAN, DG
通讯作者：
ALTMAN, DG