Active Learning with Neural Networks: Insights from Nonparametric Statistics

Active Learning with Neural Networks: Insights from Nonparametric Statistics
复制标题

DOI：
10.48550/arxiv.2210.08367
复制
发表时间：
2022-10
期刊：
ArXiv
影响因子：
--
通讯作者：
Yinglun Zhu;R. Nowak
Yinglun Zhu;R. Nowak
中科院分区：
其他
文献类型：
--
作者：
Yinglun Zhu;R. Nowak

文献摘要

相似文献

深度神经网络具有很强的表示能力，但通常需要大量的训练样本。这激发了深度主动学习方法，可以显着减少标记的训练数据量。最近在文献中报道了深度主动学习的经验性成功，然而，深度主动学习的严格标签复杂性保证仍然难以捉摸。这是理论与实践之间的巨大差距。本文通过为深度主动学习提供第一个接近最优的标签复杂度保证来解决这一差距。关键的见解是从非参数分类的角度研究深度主动学习。在标准的低噪声条件下，我们证明了神经网络的主动学习可以证明达到极大极小标签复杂度，直到不一致系数和其他对数项。当配备了排除选项时，我们进一步开发了一种高效的深度主动学习算法，该算法可以实现$\mathsf{polylog}（\frac{1}{\displaystyle {\frac {1}）$ label复杂度，而无需任何低噪声假设。我们还提供了我们的结果超出了通常研究的Sobolev/H\“old空间的扩展，并开发了Radon $\mathsf{BV}^2 $空间中学习的标签复杂性保证，这些空间最近被提出作为与神经网络相关的自然函数空间。
Deep neural networks have great representation power, but typically require large numbers of training examples. This motivates deep active learning methods that can significantly reduce the amount of labeled training data. Empirical successes of deep active learning have been recently reported in the literature, however, rigorous label complexity guarantees of deep active learning have remained elusive. This constitutes a significant gap between theory and practice. This paper tackles this gap by providing the first near-optimal label complexity guarantees for deep active learning. The key insight is to study deep active learning from the nonparametric classification perspective. Under standard low noise conditions, we show that active learning with neural networks can provably achieve the minimax label complexity, up to disagreement coefficient and other logarithmic terms. When equipped with an abstention option, we further develop an efficient deep active learning algorithm that achieves $\mathsf{polylog}(\frac{1}{\epsilon})$ label complexity, without any low noise assumptions. We also provide extensions of our results beyond the commonly studied Sobolev/H\"older spaces and develop label complexity guarantees for learning in Radon $\mathsf{BV}^2$ spaces, which have recently been proposed as natural function spaces associated with neural networks.