Clinical concept extraction using transformers
Clinical concept extraction using transformers
复制标题
DOI：
10.1093/jamia/ocaa189
复制
发表时间：
2020-12-01
影响因子：
6.4
通讯作者：
Wu, Yonghui
中科院分区：
文献类型：
--
作者：
Yang, Xi;Bian, Jiang;Wu, Yonghui
目的：本研究的目标是探索用于临床概念提取的基于 Transformer 的模型（例如 Transformers 的双向编码器表示 [BERT]），并开发一个包含预训练临床模型的开源包，以促进医学领域的概念提取和其他下游自然语言处理 (NLP) 任务。方法：我们系统地探索了 4 种广泛使用的基于 Transformer 的架构，包括 BERT、RoBERTa、 ALBERT 和 ELECTRA，使用来自 2010 年和 2012 年 i2b2 挑战赛以及 2018 年 n2c2 挑战赛的 3 个公共数据集提取各种类型的临床概念。我们检查了使用通用英语语料库预训练的通用 Transformer 模型以及使用临床语料库预训练的临床 Transformer 模型，并将它们与作为基线的长短期记忆条件随机场 (LSTM-CRF) 模式进行比较。此外，我们将 4 个基于临床 Transformer 的模型集成到一个开源包中。 结果和结论：RoBERTa-MIMIC 模型在 3 个公共临床概念提取数据集上实现了最先进的性能，F1 分数分别为 0.8994、0.8053 和 0.8907。与基线 LSTM-CRF 模型相比，RoBERTa-MIMIC 在 2010 年和 2012 年 i2b2 数据集上的 F1 分数显着提高了约 4% 和 6%。这项研究证明了基于变压器的临床概念提取模型的效率。我们的方法和系统可以应用于其他临床任务。包含 4 个预训练临床模型的临床 Transformer 包可在 https://github.com/uf-hobi-informatics-lab/ClinicalTransformerNER 上公开获取。我们相信该软件包将改善当前临床概念提取和医学领域其他任务的实践。
Objective: The goal of this study is to explore transformer-based models (eg, Bidirectional Encoder Representations from Transformers [BERT]) for clinical concept extraction and develop an open-source package with pretrained clinical models to facilitate concept extraction and other downstream natural language processing (NLP) tasks in the medical domain.Methods: We systematically explored 4 widely used transformer-based architectures, including BERT, RoBERTa, ALBERT, and ELECTRA, for extracting various types of clinical concepts using 3 public datasets from the 2010 and 2012 i2b2 challenges and the 2018 n2c2 challenge. We examined general transformer models pretrained using general English corpora as well as clinical transformer models pretrained using a clinical corpus and compared them with a long short-term memory conditional random fields (LSTM-CRFs) mode as a baseline. Furthermore, we integrated the 4 clinical transformer-based models into an open-source package.Results and Conclusion: The RoBERTa-MIMIC model achieved state-of-the-art performance on 3 public clinical concept extraction datasets with F1-scores of 0.8994, 0.8053, and 0.8907, respectively. Compared to the baseline LSTM-CRFs model, RoBERTa-MIMIC remarkably improved the F1-score by approximately 4% and 6% on the 2010 and 2012 i2b2 datasets. This study demonstrated the efficiency of transformer-based models for clinical concept extraction. Our methods and systems can be applied to other clinical tasks. The clinical transformer package with 4 pretrained clinical models is publicly available at https://github.com/uf-hobi-informatics-lab/ClinicalTransformerNER. We believe this package will improve current practice on clinical concept extraction and other tasks in the medical domain.