KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding
KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding
复制标题
DOI：
10.48550/arxiv.2210.04105
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Shangbin Feng;Zhaoxuan Tan;Wenqian Zhang;Zhenyu Lei;Yulia Tsvetkov
中科院分区：
文献类型：
--
作者：
Shangbin Feng;Zhaoxuan Tan;Wenqian Zhang;Zhenyu Lei;Yulia Tsvetkov
随着预训练的语言模型（LMS）的发展，不断增加的研究工作集中在注入常识性和特定领域的知识上，以准备LMS以实现下游任务。 ，以及预先训练的LMS，在现有方法中利用外部知识，仍然是一个悬而未决的问题不同的上下文 - 从本地（例如句子），文档级别到全球知识，以启用跨环境的知识丰富和可解释的交流，增加的内容尤其可以使长期的文档理解的任务受益，通常是利用预培训的LMS（通常）根据这些挑战，我们提出了kalm，该语言模型在本地，文档级别和全球长篇文档的上下文理解。得出总体文档表示。在6个数据集/设置中了解的文档了解。进一步分析表明，三个知识感上下文已完成，它们都有助于模型性能，而不同上下文的重要性和信息交换模式在不同的任务和数据集上有所不同。
With the advent of pre-trained language models (LMs), increasing research efforts have been focusing on infusing commonsense and domain-specific knowledge to prepare LMs for downstream tasks. These works attempt to leverage knowledge graphs, the de facto standard of symbolic knowledge representation, along with pre-trained LMs. While existing approaches leverage external knowledge, it remains an open question how to jointly incorporate knowledge graphs represented in varying contexts — from local (e.g., sentence), document-level, to global knowledge, to enable knowledge-rich and interpretable exchange across contexts. In addition, incorporating varying contexts can especially benefit long document understanding tasks that leverage pre-trained LMs, typically bounded by the input sequence length. In light of these challenges, we propose KALM, a language model that jointly leverages knowledge in local, document-level, and global contexts for long document understanding. KALM firstly encodes long documents and knowledge graphs into the three knowledge-aware context representations. KALM then processes each context with context-specific layers. These context-specific layers are followed by a ContextFusion layer that facilitates knowledge exchange to derive an overarching document representation. Extensive experiments demonstrate that KALM achieves state-of-the-art performance on three long document understanding tasks across 6 datasets/settings. Further analyses reveal that the three knowledge-aware contexts are complementary and they all contribute to model performance, while the importance and information exchange patterns of different contexts vary on different tasks and datasets.