Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories
Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories
复制标题
DOI：
10.1109/iccv48922.2021.01053
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Fait Poms;Vishnu Sarukkai;Ravi Teja Mullapudi;N. Sohoni;W. Mark;Deva Ramanan;Kayvon Fatahalian
中科院分区：
文献类型：
--
作者：
Fait Poms;Vishnu Sarukkai;Ravi Teja Mullapudi;N. Sohoni;W. Mark;Deva Ramanan;Kayvon Fatahalian
对于使用有限的标记训练数据训练的机器学习模型，验证将成为降低整体注释成本的主要瓶颈。我们提出了一种统计验证算法，可以准确估计稀有类别的二进制分类器的F分数，其中找到相关的例子进行评估是特别具有挑战性的。我们的关键见解是，即使在低样本状态（< 300个样本）下，同步校准和重要性采样也可以实现准确的估计。重要的是，我们还推导出了我们方法方差的准确单次试验估计量，并证明了该估计量在低样本数下凭经验是准确的，使从业者能够知道他们可以信任给定的低样本估计值。在ImageNet和iNatural-ist 2017上验证最先进的半监督模型时，我们的方法实现了相同的模型性能估计，标签数量比竞争方法少10倍。特别是，我们可以使用少至100个标签来估计方差为0.005的模型F1得分。
For machine learning models trained with limited labeled training data, validation stands to become the main bottleneck to reducing overall annotation costs. We propose a statistical validation algorithm that accurately estimates the F-score of binary classifiers for rare categories, where finding relevant examples to evaluate on is particularly challenging. Our key insight is that simultaneous calibration and importance sampling enables accurate estimates even in the low-sample regime (< 300 samples). Critically, we also derive an accurate single-trial estimator of the variance of our method and demonstrate that this estimator is empirically accurate at low sample counts, enabling a practitioner to know how well they can trust a given low-sample estimate. When validating state-of-the-art semi-supervised models on ImageNet and iNatural-ist2017, our method achieves the same estimates of model performance with up to 10× fewer labels than competing approaches. In particular, we can estimate model F1 scores with a variance of 0.005 using as few as 100 labels.