Rashomon Capacity: A Metric for Predictive Multiplicity in Classification-猫眼课题宝

DOI：

发表时间：

2022-06

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Hsiang Hsu;F. Calmon

当具有统计上不可区分的性能的分类模型将相互冲突的预测分配给单个样本时，会出现预测多重性。当用于后果应用中的决策时（例如，贷款、教育、刑事司法），在不考虑预测多重性的情况下开发的模型可能导致对特定个人作出不公正和任意的决定。我们引入了一个新的度量，称为罗生门容量，以衡量预测的概率分类的多重性。用于预测多重性的先前度量集中于输出阈值的分类器（即，0-1)预测类相比之下，罗生门容量适用于概率分类器，捕获单个样本的更细微的分数变化。我们提供了一个严格的推导罗生门容量，论证其直观的吸引力，并演示如何在实践中估计它。我们表明，罗生门产能产生原则性的战略，披露冲突的模型，利益相关者。我们的数值实验说明了罗生门容量如何在各种数据集和学习模型（包括神经网络）中捕获预测多重性。本文介绍的工具可以帮助数据科学家在模型部署之前测量和报告预测多样性。

Predictive multiplicity occurs when classification models with statistically indistinguishable performances assign conflicting predictions to individual samples. When used for decision-making in applications of consequence (e.g., lending, education, criminal justice), models developed without regard for predictive multiplicity may result in unjustified and arbitrary decisions for specific individuals. We introduce a new metric, called Rashomon Capacity, to measure predictive multiplicity in probabilistic classification. Prior metrics for predictive multiplicity focus on classifiers that output thresholded (i.e., 0-1) predicted classes. In contrast, Rashomon Capacity applies to probabilistic classifiers, capturing more nuanced score variations for individual samples. We provide a rigorous derivation for Rashomon Capacity, argue its intuitive appeal, and demonstrate how to estimate it in practice. We show that Rashomon Capacity yields principled strategies for disclosing conflicting models to stakeholders. Our numerical experiments illustrate how Rashomon Capacity captures predictive multiplicity in various datasets and learning models, including neural networks. The tools introduced in this paper can help data scientists measure and report predictive multiplicity prior to model deployment.

Rashomon Capacity: A Metric for Predictive Multiplicity in Classification

文献摘要

相似文献