A Lazy Man's Approach to Benchmarking: Semisupervised Classifier Evaluation and Recalibration
A Lazy Man's Approach to Benchmarking: Semisupervised Classifier Evaluation and Recalibration
复制标题
懒人的基准测试方法：半监督分类器评估和重新校准
DOI：
--
复制
发表时间：
2013
期刊：
影响因子：
--
通讯作者：
P. Perona
中科院分区：
文献类型：
--
作者：
P. Welinder;M. Welling;P. Perona
需要多少标记的例子来估计分类器在新数据集上的性能？我们研究的情况下，数据丰富，但标签是昂贵的。我们表明，通过对数据的结构进行一些合理的假设，可以使用少量的地面真值标签来估计性能曲线和置信区间。我们的方法，我们称之为半监督性能评估（SPE），是基于分类器的置信度得分的生成模型。除了估计分类器在新数据集上的性能外，SPE还可以通过重新估计类条件置信度分布来重新校准分类器。
How many labeled examples are needed to estimate a classifier's performance on a new dataset? We study the case where data is plentiful, but labels are expensive. We show that by making a few reasonable assumptions on the structure of the data, it is possible to estimate performance curves, with confidence bounds, using a small number of ground truth labels. Our approach, which we call Semi supervised Performance Evaluation (SPE), is based on a generative model for the classifier's confidence scores. In addition to estimating the performance of classifiers on new datasets, SPE can be used to recalibrate a classifier by re-estimating the class-conditional confidence distributions.