What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
复制标题
DOI：
10.1162/tacl_a_00298
复制
发表时间：
2020-01-01
影响因子：
10.9
通讯作者：
Ettinger, Allyson
中科院分区：
文献类型：
--
作者：
Ettinger, Allyson
通过语言建模进行预训练已成为NLP任务的一种流行而成功的方法，但是我们尚未确切了解这些前训练过程赋予模型的语言能力。在本文中，我们介绍了一套从人类语言实验中提取的诊断套件，这使我们能够询问有关语言模型在上下文中生成预测的信息的有针对性问题。作为一个案例研究，我们将这些诊断应用应用于流行的BERT模型，发现它通常可以将好的区别于涉及共享类别或角色逆转的不良完成，尽管比人类的敏感性较小，并且可以牢固地检索名词超声词，但是它与名词超声词相比，它的努力却很努力。具有挑战性的推论和基于角色的事件预测和尤其是对否定的上下文影响的明显不敏感。
Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event predictionand, in particular, it shows clear insensitivity to the contextual impacts of negation.