Limitations of Assessing Active Learning Performance at Runtime
Limitations of Assessing Active Learning Performance at Runtime
复制标题
在运行时评估主动学习表现的局限性
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
B. Sick
中科院分区：
文献类型：
--
作者：
D. Kottke;Jim Schellinger;Denis Huseljic;B. Sick
分类算法旨在预测新实例（例如产品）的未知标签（例如质量等级）。因此，训练样本（实例和标签）用于推导分类假设。通常，捕获实例相对容易，但获取相应的标签仍然很困难或昂贵。主动学习算法选择最有益的实例进行标记以降低成本。在研究中，这个标记过程是模拟的，因此可以得到基本事实。但在部署过程中，主动学习是一个一次性问题，并且没有评估集。因此，不可能在学习过程中可靠地估计分类系统的性能，并且很难确定系统何时满足质量要求（停止标准）。在本文中，我们将任务形式化并回顾现有策略，以评估主动训练的分类器在训练期间的性能。此外，我们确定了三个主要挑战：1）~导出性能分布，2）~保持标记子集的代表性，3）纠正智能选择策略引起的采样偏差。在定性分析中，我们评估了不同的现有方法，并表明它们都没有可靠地估计主动学习性能，这对此类系统的未来研究构成了重大挑战。所有图表和实验都在 Jupyter 笔记本中提供，可供下载。
Classification algorithms aim to predict an unknown label (e.g., a quality class) for a new instance (e.g., a product). Therefore, training samples (instances and labels) are used to deduct classification hypotheses. Often, it is relatively easy to capture instances but the acquisition of the corresponding labels remain difficult or expensive. Active learning algorithms select the most beneficial instances to be labeled to reduce cost. In research, this labeling procedure is simulated and therefore a ground truth is available. But during deployment, active learning is a one-shot problem and an evaluation set is not available. Hence, it is not possible to reliably estimate the performance of the classification system during learning and it is difficult to decide when the system fulfills the quality requirements (stopping criteria). In this article, we formalize the task and review existing strategies to assess the performance of an actively trained classifier during training. Furthermore, we identified three major challenges: 1)~to derive a performance distribution, 2)~to preserve representativeness of the labeled subset, and 3) to correct against sampling bias induced by an intelligent selection strategy. In a qualitative analysis, we evaluate different existing approaches and show that none of them reliably estimates active learning performance stating a major challenge for future research for such systems. All plots and experiments are provided in a Jupyter notebook that is available for download.