Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance
Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance
复制标题
DOI：
10.18653/v1/2020.acl-main.770
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
Prasetya Ajie Utama;N. Moosavi;Iryna Gurevych
中科院分区：
文献类型：
--
作者：
Prasetya Ajie Utama;N. Moosavi;Iryna Gurevych
自然语言理解（NLU）任务的模型通常依赖于数据集的特殊偏差，这使得它们在训练分布之外的测试用例中变得脆弱。最近，提出的几种去偏置方法被证明在改善分布外性能方面非常有效。然而，它们的改进是以性能下降为代价的，当模型在分布数据上进行评估时，其中包含具有更高多样性的示例。这种看似不可避免的权衡可能无法告诉我们，除了分布外数据中所代表的小子集之外，结果模型在更广泛类型的示例上的推理和理解能力的变化。在本文中，我们通过引入一种新的去偏置方法（称为置信正则化）来解决这种权衡，该方法阻止模型利用偏差，同时使它们能够获得足够的激励来从所有训练示例中学习。我们在三个NLU任务上评估了我们的方法，并表明与其前辈相比，它提高了分布外数据集的性能（例如，HANS数据集上的7pp增益），同时保持原始分布准确度。
Models for natural language understanding (NLU) tasks often rely on the idiosyncratic biases of the dataset, which make them brittle against test cases outside the training distribution. Recently, several proposed debiasing methods are shown to be very effective in improving out-of-distribution performance. However, their improvements come at the expense of performance drop when models are evaluated on the in-distribution data, which contain examples with higher diversity. This seemingly inevitable trade-off may not tell us much about the changes in the reasoning and understanding capabilities of the resulting models on broader types of examples beyond the small subset represented in the out-of-distribution data. In this paper, we address this trade-off by introducing a novel debiasing method, called confidence regularization, which discourage models from exploiting biases while enabling them to receive enough incentive to learn from all the training examples. We evaluate our method on three NLU tasks and show that, in contrast to its predecessors, it improves the performance on out-of-distribution datasets (e.g., 7pp gain on HANS dataset) while maintaining the original in-distribution accuracy.