Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)-Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study
Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)-Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study
复制标题
DOI：
10.2196/14830
复制
发表时间：
2019-07-01
影响因子：
3.2
通讯作者：
Yu, Hong
中科院分区：
文献类型：
--
作者：
Li, Fei;Jin, Yonghao;Yu, Hong
背景资料：BERT模型在命名实体识别、问答等自然语言处理领域取得了巨大的成功。然而，很少有以前的工作已经探讨了这个模型被用于一个重要的任务，在生物医学和临床domains，即实体normalization.Objective：我们的目的是研究基于BERT模型的生物医学或临床实体规范化的有效性。此外，我们的第二个目标是调查是否域的训练数据的BERT为基础的models.Methods的性能的影响以及程度：我们的数据是由150万未标记的电子健康记录（EHR）的笔记。我们首先对这个大量未标记的EHR笔记集合进行了BioBERT微调。这生成了我们基于BERT的模型，该模型使用150万个电子健康记录笔记（EhrBERT）进行训练。然后，我们在三个注释语料库上进一步微调EhrBERT，BioBERT和BERT用于生物医学和临床实体标准化：药物，适应症和不良药物事件（MADE）1.0语料库，国家生物技术信息中心（NCBI）疾病语料库和化学疾病关系（CDR）语料库。我们将我们的模型与两个最先进的规范化系统，即MetaMap和疾病名称规范化（DNorm）进行了比较。结果：EhrBERT在MADE 1.0语料库中将命名实体映射到约380，000个术语的医学词典和医学临床术语系统化命名法（SNOMED-CT）中，达到了40.95%的F1。在该语料库中，EhrBERT在F1中的表现优于MetaMap 2.36%。对于NCBI疾病语料库和CDR语料库，EhrBERT也优于DNorm，将F1评分分别从88.37%和89.92%提高到90.35%和93.82%。与BioBERT和BERT相比，EhrBERT在MADE 1.0语料库和CDR corpus.Conclusions上的表现优于它们：我们的工作表明，基于BERT的模型在生物医学和临床实体规范化方面已经达到了最先进的性能。基于BERT的模型可以很容易地进行微调，以规范化任何类型的命名实体。
Background: The bidirectional encoder representations from transformers (BERT) model has achieved great success in many natural language processing (NLP) tasks, such as named entity recognition and question answering. However, little prior work has explored this model to be used for an important task in the biomedical and clinical domains, namely entity normalization.Objective: We aim to investigate the effectiveness of BERT-based models for biomedical or clinical entity normalization. In addition, our second objective is to investigate whether the domains of training data influence the performances of BERT-based models as well as the degree of influence.Methods: Our data was comprised of 1.5 million unlabeled electronic health record (EHR) notes. We first fine-tuned BioBERT on this large collection of unlabeled EHR notes. This generated our BERT-based model trained using 1.5 million electronic health record notes (EhrBERT). We then further fine-tuned EhrBERT, BioBERT, and BERT on three annotated corpora for biomedical and clinical entity normalization: the Medication, Indication, and Adverse Drug Events (MADE) 1.0 corpus, the National Center for Biotechnology Information (NCBI) disease corpus, and the Chemical-Disease Relations (CDR) corpus. We compared our models with two state-of-the-art normalization systems, namely MetaMap and disease name normalization (DNorm).Results: EhrBERT achieved 40.95% F1 in the MADE 1.0 corpus for mapping named entities to the Medical Dictionary for Regulatory Activities and the Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT), which have about 380,000 terms. In this corpus, EhrBERT outperformed MetaMap by 2.36% in F1. For the NCBI disease corpus and CDR corpus, EhrBERT also outperformed DNorm by improving the F1 scores from 88.37% and 89.92% to 90.35% and 93.82%, respectively. Compared with BioBERT and BERT, EhrBERT outperformed them on the MADE 1.0 corpus and the CDR corpus.Conclusions: Our work shows that BERT-based models have achieved state-of-the-art performance for biomedical and clinical entity normalization. BERT-based models can be readily fine-tuned to normalize any kind of named entities.