Evaluating and Characterizing Human Rationales
Evaluating and Characterizing Human Rationales
复制标题
DOI：
10.18653/v1/2020.emnlp-main.747
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Samuel Carton;Anirudh Rathore;Chenhao Tan
中科院分区：
文献类型：
--
作者：
Samuel Carton;Anirudh Rathore;Chenhao Tan
评估机器生成的基本原理的质量的两种主要方法是：1）使用人类基本原理作为金标准; 2）基于基本原理如何影响模型行为的自动度量。然而，一个悬而未决的问题是，人类的理论如何与这些自动度量相结合。通过分析各种数据集和模型，我们发现人类的理性并不一定在这些指标上表现良好。为了解释这一发现，我们提出了改进的指标来解释依赖于模型的基线性能。然后，我们提出了两种方法来进一步表征合理的质量，一个基于模型再训练和一个使用“保真度曲线”，以揭示属性，如无关性和冗余。我们的工作导致可操作的建议，评估和表征的理由。
Two main approaches for evaluating the quality of machine-generated rationales are: 1) using human rationales as a gold standard; and 2) automated metrics based on how rationales affect model behavior. An open question, however, is how human rationales fare with these automatic metrics. Analyzing a variety of datasets and models, we find that human rationales do not necessarily perform well on these metrics. To unpack this finding, we propose improved metrics to account for model-dependent baseline performance. We then propose two methods to further characterize rationale quality, one based on model retraining and one on using "fidelity curves" to reveal properties such as irrelevance and redundancy. Our work leads to actionable suggestions for evaluating and characterizing rationales.