ReasonBERT: Pre-trained to Reason with Distant Supervision
ReasonBERT: Pre-trained to Reason with Distant Supervision
复制标题
DOI：
10.18653/v1/2021.emnlp-main.494
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Xiang Deng;Yu Su;Alyssa Lees;You Wu;Cong Yu;Huan Sun
中科院分区：
文献类型：
--
作者：
Xiang Deng;Yu Su;Alyssa Lees;You Wu;Cong Yu;Huan Sun
我们提出了ReasonBert，这是一种预训练方法，它增强了语言模型在长距离关系和多种可能的混合上下文中推理的能力。与现有的预训练方法不同，这些方法只从自然发生的文本的本地上下文中获取学习信号，我们提出了一种广义的远程监督概念，可以自动连接多个文本和表格，以创建需要远程推理的预训练示例。模拟不同类型的推理，包括交叉多个证据，从一个证据到另一个证据的桥接，以及检测无法回答的案例。我们对各种提取问题回答数据集进行了全面评估，从单跳到多跳，从纯文本到纯表格到混合，需要各种推理能力，并表明ReasonBert在一系列强基线上取得了显着的改进。少量实验进一步证明，我们的预训练方法大大提高了样本效率。
We present ReasonBert, a pre-training method that augments language models with the ability to reason over long-range relations and multiple, possibly hybrid contexts. Unlike existing pre-training methods that only harvest learning signals from local contexts of naturally occurring texts, we propose a generalized notion of distant supervision to automatically connect multiple pieces of text and tables to create pre-training examples that require long-range reasoning. Different types of reasoning are simulated, including intersecting multiple pieces of evidence, bridging from one piece of evidence to another, and detecting unanswerable cases. We conduct a comprehensive evaluation on a variety of extractive question answering datasets ranging from single-hop to multi-hop and from text-only to table-only to hybrid that require various reasoning capabilities and show that ReasonBert achieves remarkable improvement over an array of strong baselines. Few-shot experiments further demonstrate that our pre-training method substantially improves sample efficiency.