Why Cohen's Kappa should be avoided as performance measure in classification
Why Cohen's Kappa should be avoided as performance measure in classification
复制标题
DOI：
10.1371/journal.pone.0222916
复制
发表时间：
2019-09-26
期刊：
影响因子：
3.7
通讯作者：
Tibau, Xavier-Andoni
中科院分区：
文献类型：
--
作者：
Delgado, Rosario;Tibau, Xavier-Andoni
我们发现，科恩的Kappa和马修斯相关系数（MCC），在多类分类的性能的扩展和对比措施，在大多数情况下是相关的，虽然可以在其他不同。事实上，虽然在对称的情况下，两者都匹配，但我们考虑不同的不平衡情况，其中Kappa表现出不期望的行为，即较差的分类器获得更高的Kappa分数，与MCC的性质不同。关于卡帕行为不一致的争论围绕着使用相对度量的方便与否，这使得对其值的解释变得困难。我们通过展示其陷阱可以走得更远来扩展这些担忧。通过实验，我们提出了一种新的方法来解决这个问题。我们进行了一项全面的研究，确定了一个方案，其中MCC和卡帕之间的矛盾行为出现。具体来说，我们发现，当有一个减少到零的熵的元素的对角线上的混淆矩阵相关联的分类器，Kappa和MCC之间的差异上升，指向一个异常的性能前者。我们认为，这一发现禁用Kappa一般作为一个性能指标来比较分类器。
We show that Cohen's Kappa and Matthews Correlation Coefficient (MCC), both extended and contrasted measures of performance in multi-class classification, are correlated in most situations, albeit can differ in others. Indeed, although in the symmetric case both match, we consider different unbalanced situations in which Kappa exhibits an undesired behaviour, i.e. a worse classifier gets higher Kappa score, differing qualitatively from that of MCC. The debate about the incoherence in the behaviour of Kappa revolves around the convenience, or not, of using a relative metric, which makes the interpretation of its values difficult. We extend these concerns by showing that its pitfalls can go even further. Through experimentation, we present a novel approach to this topic. We carry on a comprehensive study that identifies an scenario in which the contradictory behaviour among MCC and Kappa emerges. Specifically, we find out that when there is a decrease to zero of the entropy of the elements out of the diagonal of the confusion matrix associated to a classifier, the discrepancy between Kappa and MCC rise, pointing to an anomalous performance of the former. We believe that this finding disables Kappa to be used in general as a performance measure to compare classifiers.