Knowledge Enhanced Contextual Word Representations
Knowledge Enhanced Contextual Word Representations
复制标题
DOI：
10.18653/v1/d19-1005
复制
发表时间：
2019-09
期刊：
影响因子：
--
通讯作者：
Matthew E. Peters;Mark Neumann;IV RobertL.Logan;Roy Schwartz;Vidur Joshi;Sameer Singh;Noah A. Smith-
中科院分区：
文献类型：
--
作者：
Matthew E. Peters;Mark Neumann;IV RobertL.Logan;Roy Schwartz;Vidur Joshi;Sameer Singh;Noah A. Smith-
通常在非结构化、未标记的文本上训练的上下文单词表示不包含对真实的世界实体的任何明确的基础，并且通常无法记住关于这些实体的事实。我们提出了一种通用的方法，将多个知识库（KBs）嵌入到大规模模型中，从而增强它们的表示与结构化，人工策划的知识。对于每个知识库，我们首先使用一个集成的实体链接器来检索相关的实体嵌入，然后通过一种形式的词到实体的注意力更新上下文单词表示。与以前的方法相比，实体链接器和自监督语言建模目标在多任务设置中进行端到端的联合训练，该设置将少量实体链接监督与大量原始文本相结合。在将WordNet和维基百科的一个子集集成到BERT中之后，知识增强BERT（KnowBert）展示了改进的困惑度、在探测任务中测量的回忆事实的能力以及在关系提取、实体键入和词义消歧上的下游性能。KnowBert的运行时间与BERT相当，并且可以扩展到大KB。
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update contextual word representations via a form of word-to-entity attention. In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that combines a small amount of entity linking supervision with a large amount of raw text. After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert) demonstrates improved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, entity typing, and word sense disambiguation. KnowBert’s runtime is comparable to BERT’s and it scales to large KBs.