PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion
PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion
复制标题
DOI：
10.48550/arxiv.2210.13715
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Jianhao Shen;Chenguang Wang;Ye Yuan;Jiawei Han;Heng Ji;Koushik Sen;Ming Zhang;Dawn Song
中科院分区：
文献类型：
--
作者：
Jianhao Shen;Chenguang Wang;Ye Yuan;Jiawei Han;Heng Ji;Koushik Sen;Ming Zhang;Dawn Song
本文提出了一种用于知识图（KG）完成的预训练语言模型（LM）的参数化迁移学习方法。而不是修改所有LM参数的微调，我们只调整了一些新参数，同时保持原始LM参数不变。我们通过将KG完成重新定义为“填空”任务，并在原始LM之上引入参数精简编码器来建立这一点。我们表明，通过调整比微调少得多的参数，LM转移到大多数任务，并与现有的最先进的方法达到竞争力。例如，我们在KG完成基准上的表现优于完全微调方法，只调整了1%的参数。代码和数据集可在\url{https：//github.com/yuanyehome/PALT}获得。
This paper presents a parameter-lite transfer learning approach of pretrained language models (LM) for knowledge graph (KG) completion. Instead of finetuning, which modifies all LM parameters, we only tune a few new parameters while keeping the original LM parameters fixed. We establish this via reformulating KG completion as a"fill-in-the-blank"task, and introducing a parameter-lite encoder on top of the original LMs. We show that, by tuning far fewer parameters than finetuning, LMs transfer non-trivially to most tasks and reach competitiveness with prior state-of-the-art approaches. For instance, we outperform the fully finetuning approaches on a KG completion benchmark by tuning only 1% of the parameters. The code and datasets are available at \url{https://github.com/yuanyehome/PALT}.