Investigating Reasons for Disagreement in Natural Language Inference
Investigating Reasons for Disagreement in Natural Language Inference
复制标题
DOI：
10.1162/tacl_a_00523
复制
发表时间：
2022-09
影响因子：
10.9
通讯作者：
Nan Jiang;M. Marneffe
中科院分区：
文献类型：
--
作者：
Nan Jiang;M. Marneffe
摘要：我们研究了自然语言推理（NLI）注释中的分歧是如何产生的。我们制定了分歧来源分类法，其中包含 10 个类别，涵盖 3 个高级类别。我们发现，一些分歧是由于句子含义的不确定性，另一些是由于注释者偏差和任务工件，导致对标签分布的不同解释。我们探索了两种用于检测具有潜在分歧的项目的建模方法：除了三个标准 NLI 标签之外还带有“复杂”标签的 4 路分类，以及多标签分类方法。我们发现多标签分类更具表现力，并且可以更好地回忆数据中可能的解释。
Abstract We investigate how disagreement in natural language inference (NLI) annotation arises. We developed a taxonomy of disagreement sources with 10 categories spanning 3 high- level classes. We found that some disagreements are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts, leading to different interpretations of the label distribution. We explore two modeling approaches for detecting items with potential disagreement: a 4-way classification with a “Complicated” label in addition to the three standard NLI labels, and a multilabel classification approach. We found that the multilabel classification is more expressive and gives better recall of the possible interpretations in the data.