Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
复制标题
文本分类局部解释的自动评估和人工评估的比较
DOI：
10.18653/v1/n18-1097
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Dong Nguyen
中科院分区：
文献类型：
--
作者：
Dong Nguyen
文本分类模型变得越来越复杂和不透明，但是对于许多应用来说，模型的可解释性至关重要。最近，人们提出了多种方法来生成局部解释。虽然需要强有力的评估来推动进一步的进展，但到目前为止还不清楚哪种评估方法是合适的。本文是对当地解释进行更稳健评估的第一步。我们使用基于单词删除的自动措施来评估各种本地解释方法。此外，我们表明，使用众包实验进行的评估与这些自动测量有一定的相关性，并且各种其他因素也会影响人类的判断。
Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable. Recently, a variety of approaches have been proposed for generating local explanations. While robust evaluations are needed to drive further progress, so far it is unclear which evaluation approaches are suitable. This paper is a first step towards more robust evaluations of local explanations. We evaluate a variety of local explanation approaches using automatic measures based on word deletion. Furthermore, we show that an evaluation using a crowdsourcing experiment correlates moderately with these automatic measures and that a variety of other factors also impact the human judgements.