基于定义性数据的预训练语言模型词汇语义增强技术

大规模预训练语言模型技术触发了一系列突破性进展。然而，在实际场景中，这类模型常暴露出基础词汇语义理解方面的盲区，极大程度上限制了其更进一步的应用。有效增强预训练模型词汇语义能力是当前研究中的重要挑战。为从根源上规避现有基于通用数据训练的方法在数据覆盖、统计学习和汇报偏差方面的根本缺陷，本项目拟在保持端到端式框架的同时,利用定义性数据来增强预训练语言模型词汇语义理解能力。主要研究内容如下：(1)建立全新的使用定义性数据的预训练模型词汇语义增强范式，摆脱通用语料的统计性限制和知识库的质效困境；(2)构建简明端到端式的词汇语义增强的预训练模型，克服传统基于知识表示学习的词义增强中错误累积问题；(3)强化预训练语言模型多层次、多模态的词汇语义理解能力，从简练精确的词典释义、综合全面的百科释义、有形具象的多模态释义三个层次出发强化预训练语言模型的词汇语义理解能力。

Large-scale pre-trained language models have enabled a series of breakthroughs. However, in real-world scenarios, such models often expose deficiencies in basic understanding of lexical semantics, limiting its full potential in application. Enhancing lexical semantics in pre-trained language models has become a vital challenge in recent research. To overcome the fundamental weaknesses in general data in terms of data coverage, statistical learning, and reporting bias, the applicant proposes to make use of the definitional data to enhance the lexical semantics in pre-trained language models, maintaining the end-to-end framework. The main areas of research include: (1) Establish a novel paradigm of enhancement of lexical semantics in pre-trained language models with definitional data, breaking away from the frequence-constraints in general data and the quality-quantity dilemma in knowledge-bases; (2) Construct concise, end-to-end, lexical semantics-enhanced pre-trained language models and, overcoming the error propagation in conventional lexical semantics-enhancement with knowledge representation learning; and (3) Comprehensively promote lexical semantics in pre-trained language models from the multi-layer and the multi-modal perspective, from compact dictionary definitions to rich encyclopedia definitions to tangible visual definitions.