Natural Language Inference with Mixed Effects
Natural Language Inference with Mixed Effects
复制标题
DOI：
--
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
William Gantt;Benjamin Kane;A. White
中科院分区：
文献类型：
--
作者：
William Gantt;Benjamin Kane;A. White
越来越多的证据表明，用于构建自然语言推理数据集的原始注释中普遍存在分歧，这使得将这些注释聚合到单个标签的常见做法存在问题。我们提出了一种通用方法，允许跳过聚合步骤，直接在原始注释上进行训练，而不会使模型受到注释器响应偏差可能产生的不必要的噪声。我们证明，这种方法通过将注释器随机效应纳入任何现有的神经模型来推广混合效应模型的概念，比不包含此类效应的模型提高了性能。
There is growing evidence that the prevalence of disagreement in the raw annotations used to construct natural language inference datasets makes the common practice of aggregating those annotations to a single label problematic. We propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise that can arise from annotator response biases. We demonstrate that this method, which generalizes the notion of a mixed effects model by incorporating annotator random effects into any existing neural model, improves performance over models that do not incorporate such effects.