A Theoretical Characterization of Semi-supervised Learning with Self-training for Gaussian Mixture Models
A Theoretical Characterization of Semi-supervised Learning with Self-training for Gaussian Mixture Models
复制标题
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Samet Oymak;Talha Cihad Gulcu
中科院分区：
文献类型：
--
作者：
Samet Oymak;Talha Cihad Gulcu
自训练是半监督学习的经典方法，已成功应用于各种机器学习问题。自训练算法为未标记的示例生成伪标签，并逐步细化这些伪标签，希望与实际标签一致。这项工作提供了对自训练算法的理论见解，重点是线性分类器。首先，我们提供了具有两个分量的高斯混合模型的示例复杂性分析。这是通过自训练迭代的尖锐非渐近特征建立的，它捕获了定点迭代方面模型精度的演变。我们的分析揭示了拒绝低置信度样本的可证明的好处，并演示了自我训练迭代如何优雅地提高模型的准确性。其次，我们研究了广义 GMM，其中分量均值服从分布。我们证明，岭正则化和类边距（即分量均值之间的分离）对于成功至关重要，缺乏正则化可能会阻止自我训练识别数据中的核心特征。
Self-training is a classical approach in semi-supervised learning which is successfully applied to a variety of machine learning problems. Self-training algorithms generate pseudo-labels for the unlabeled examples and progressively refine these pseudo-labels which hopefully coincides with the actual labels. This work provides theoretical insights into self-training algorithms with a focus on linear classifiers. First, we provide a sample complexity analysis for Gaussian mixture models with two components. This is established by sharp non-asymptotic characterization of the self-training iterations which captures the evolution of the model accuracy in terms of a fixed-point iteration. Our analysis reveals the provable benefits of rejecting samples with low confidence and demonstrates how self-training iterations can gracefully improve the model accuracy. Secondly, we study a generalized GMM where the component means follow a distribution. We demonstrate that ridge regularization and class margin (i.e. separation between the component means) is crucial for the success and lack of regularization may prevent self-training from identifying the core features in the data.