RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning
RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning
复制标题
DOI：
--
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Krishnateja Killamsetty;Xujiang Zhao;F. Chen;Rishabh K. Iyer
中科院分区：
文献类型：
--
作者：
Krishnateja Killamsetty;Xujiang Zhao;F. Chen;Rishabh K. Iyer
近年来，半监督学习（SSL）算法在有限的标记数据领域取得了巨大成功。然而，当前最先进的 SSL 算法的计算成本很高，并且需要大量的计算时间和能量需求。对于许多小公司和学术团体来说，这可能是一个巨大的限制。我们的主要见解是，对未标记数据的子集而不是整个未标记数据进行训练使当前的 SSL 算法能够更快地收敛，从而显着降低计算成本。在这项工作中，我们提出了 RETRIEVE，一种用于高效、鲁棒半监督学习的核心集选择框架。 RETRIEVE 通过解决混合离散连续双层优化问题来选择核心集，使得所选核心集最小化标记集损失。我们使用一步梯度近似，并表明离散优化问题近似为子模问题，使得简单的贪婪算法能够获得核心集。我们在几个真实数据集上凭经验证明，现有的 SSL 算法（如 VAT、Mean-Teacher、FixMatch）与 RETRIEVE 一起使用时，可以实现 a) 更快的训练时间，b) 当未标记数据包含分布外 (OOD) 数据和不平衡时，可以实现更好的性能。更具体地说，我们表明，在精度下降最小的情况下，RETRIEVE 在传统 SSL 设置中实现了约 $3\times$ 的加速，并且在不平衡和 OOD 数据的情况下，与最先进 (SOTA) 稳健的 SSL 算法相比，实现了 $5\times$ 的加速。 RETRIEVE 作为 CORDS 工具包的一部分提供：https://github.com/decile-team/cords。
Semi-supervised learning (SSL) algorithms have had great success in recent years in limited labeled data regimes. However, the current state-of-the-art SSL algorithms are computationally expensive and entail significant compute time and energy requirements. This can prove to be a huge limitation for many smaller companies and academic groups. Our main insight is that training on a subset of unlabeled data instead of entire unlabeled data enables the current SSL algorithms to converge faster, significantly reducing computational costs. In this work, we propose RETRIEVE, a coreset selection framework for efficient and robust semi-supervised learning. RETRIEVE selects the coreset by solving a mixed discrete-continuous bi-level optimization problem such that the selected coreset minimizes the labeled set loss. We use a one-step gradient approximation and show that the discrete optimization problem is approximately submodular, enabling simple greedy algorithms to obtain the coreset. We empirically demonstrate on several real-world datasets that existing SSL algorithms like VAT, Mean-Teacher, FixMatch, when used with RETRIEVE, achieve a) faster training times, b) better performance when unlabeled data consists of Out-of-Distribution (OOD) data and imbalance. More specifically, we show that with minimal accuracy degradation, RETRIEVE achieves a speedup of around $3\times$ in the traditional SSL setting and achieves a speedup of $5\times$ compared to state-of-the-art (SOTA) robust SSL algorithms in the case of imbalance and OOD data. RETRIEVE is available as a part of the CORDS toolkit: https://github.com/decile-team/cords.