Information-Theoretic Measures for Objective Evaluation of Classifications
Information-Theoretic Measures for Objective Evaluation of Classifications
复制标题
DOI：
10.3724/sp.j.1004.2012.01169
复制
发表时间：
2011-07
期刊：
影响因子：
--
通讯作者：
Bao-Gang Hu;R. He;Xiao-Tong Yuan
中科院分区：
文献类型：
--
作者：
Bao-Gang Hu;R. He;Xiao-Tong Yuan
摘要本文采用信息论方法(ITMS)对弃权分类的客观评价进行了系统研究。首先，我们将客观度量定义为不依赖于任何自由参数的度量。根据这一定义，直接为分类评价提供了检查“客观性”或“主观性”的技术简单性。其次，我们提出了24个归一化的ITMS用于研究，它们要么来自互信息，要么来自散度，要么来自于交叉熵。与传统的基于用户直觉或偏好应用经验公式的绩效衡量相反，ITMS在实现分类的客观评估方面在理论上更具一般性。它们能够区分二进制分类中的“错误类型”和“拒绝类型”，而不需要输入成本项数据。第三，为了更好地理解和选择ITMS，我们为分类评估措施提出了三个可取的特征，从分类应用的角度来看，这三个特征显得更关键和更有吸引力。以这些特征作为“元度量”，我们可以从更高层次的评价知识中揭示ITMS的优势和局限性。给出了数值例子来验证我们的主张，并比较了所提出的措施之间的差异。根据元度量选择最佳度量，并解析地推导出其关于错误类型和拒绝类型的具体性质。
Abstract This work presents a systematic study of objective evaluations of abstaining classifications using information-theoretic measures (ITMs). First, we define objective measures as the ones which do not depend on any free parameter. According to this definition, technical simplicity for examining “objectivity” or “subjectivity” is directly provided for classification evaluations. Second, we propose 24 normalized ITMs for investigation, which are derived from either mutual information, divergence, or cross-entropy. Contrary to conventional performance measures that apply empirical formulas based on users' intuitions or preferences, the ITMs are theoretically more general for realizing objective evaluations of classifications. They are able to distinguish “error types” and “reject types” in binary classifications without the need to inputting data of cost terms. Third, to better understand and select the ITMs, we suggest three desirable features for classification assessment measures, which appear more crucial and appealing from the viewpoint of classification applications. Using these features as “meta-measures”, we can reveal the advantages and limitations of ITMs from a higher level of evaluation knowledge. Numerical examples are given to demonstrate our claims and compare the differences among the proposed measures. The best measure is selected in terms of the meta-measures, and its specific properties regarding error types and reject types are analytically derived.