Self-Alignment Pretraining for Biomedical Entity Representations
Self-Alignment Pretraining for Biomedical Entity Representations
复制标题
DOI：
10.18653/v1/2021.naacl-main.334
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Fangyu Liu;Ehsan Shareghi;Zaiqiao Meng;Marco Basaldella;Nigel Collier
中科院分区：
文献类型：
--
作者：
Fangyu Liu;Ehsan Shareghi;Zaiqiao Meng;Marco Basaldella;Nigel Collier
尽管通过掩蔽语言模型（MLM）的自监督学习取得了广泛的成功，但在生物医学领域准确捕获细粒度语义关系仍然是一个挑战。这对于实体级任务（如实体链接）至关重要，在实体级任务中，建模实体关系（尤其是同义词）的能力至关重要。为了解决这一挑战，我们提出了SapBERT，一种自对齐生物医学实体表示空间的预训练方案。我们设计了一个可扩展的度量学习框架，可以利用UMLS，这是一个具有4M+概念的生物医学本体的大量集合。与之前基于管道的混合系统相比，SapBERT为医疗实体链接（MEL）问题提供了一个优雅的单一模型解决方案，在六个MEL基准数据集上实现了最先进的新技术（SOTA）。在科学领域，即使没有特定任务的监督，我们也能实现SOTA。与BioBERT、sciberand和PubMedBERT等不同领域的预训练传销相比，我们的预训练方案有了很大的改进，证明了它的有效性和鲁棒性。
Despite the widespread success of self-supervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we propose SapBERT, a pretraining scheme that self-aligns the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with 4M+ concepts. In contrast with previous pipeline-based hybrid systems, SapBERT offers an elegant one-model-for-all solution to the problem of medical entity linking (MEL), achieving a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific domain, we achieve SOTA even without task-specific supervision. With substantial improvement over various domain-specific pretrained MLMs such as BioBERT, SciBERTand and PubMedBERT, our pretraining scheme proves to be both effective and robust.