BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks-猫眼课题宝

DOI：

10.18653/v1/2021.findings-emnlp.140

发表时间：

2021-09

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

T. Lai;Heng Ji;ChengXiang Zhai

生物医学实体链接是将生物医学文档中提及的实体链接到知识库中的参考实体的任务。最近，许多基于BERT的模型被引入到这项任务中。尽管这些模型在许多数据集上取得了具有竞争力的结果，但它们的计算成本很高，包含约1.1亿个参数。关于它们令人印象深刻的表现的因素以及是否需要过度参数化，人们知之甚少。在这项工作中，我们阐明了这些基于BERT的大型模型的内部工作机制。通过一组探测实验，我们发现，当输入的词序被打乱或注意范围被限制在固定的窗口大小时，实体链接性能只会发生微小的变化。根据这些观察结果，我们提出了一种用于生物医学实体链接的具有剩余连接的高效卷积神经网络。由于稀疏连通性和权重共享的性质，我们的模型参数少，效率高。在五个公共数据集上，我们的模型达到了与最先进的基于BERT的模型相当甚至更好的链接精度，而拥有的参数大约少了60倍。

Biomedical entity linking is the task of linking entity mentions in a biomedical document to referent entities in a knowledge base. Recently, many BERT-based models have been introduced for the task. While these models have achieved competitive results on many datasets, they are computationally expensive and contain about 110M parameters. Little is known about the factors contributing to their impressive performance and whether the over-parameterization is needed. In this work, we shed some light on the inner working mechanisms of these large BERT-based models. Through a set of probing experiments, we have found that the entity linking performance only changes slightly when the input word order is shuffled or when the attention scope is limited to a fixed window size. From these observations, we propose an efficient convolutional neural network with residual connections for biomedical entity linking. Because of the sparse connectivity and weight sharing properties, our model has a small number of parameters and is highly efficient. On five public datasets, our model achieves comparable or even better linking accuracy than the state-of-the-art BERT-based models while having about 60 times fewer parameters.

BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks

文献摘要

相似文献