How Out-of-Distribution Data Hurts Semi-Supervised Learning
How Out-of-Distribution Data Hurts Semi-Supervised Learning
复制标题
DOI：
10.1109/icdm54844.2022.00087
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Xujiang Zhao;Killamsetty Krishnateja;Rishabh K. Iyer;Feng Chen
中科院分区：
文献类型：
--
作者：
Xujiang Zhao;Killamsetty Krishnateja;Rishabh K. Iyer;Feng Chen
最近的半监督学习算法由于使用了更好的未标记数据表示而表现出更高的整体性能。尽管如此，最近的研究表明，SSL算法的性能可能会下降时，未标记的集合包含分布外的例子（OOD）。这项工作解决了以下研究问题：如何外的分布（OOD）数据产生不利影响的半监督学习算法？为了回答这个问题，我们调查的关键原因OOD的SSL算法的负面影响。特别是，我们发现1）某些类型的OOD数据实例接近决策边界比那些更远的对性能的影响更显着，2）批量归一化（BN），一个流行的模块，可能会降低而不是提高性能时，未标记的集合包含OOD。为了解决这些挑战，我们开发了一个统一的加权强大的SSL框架，可以很容易地扩展到许多现有的SSL算法，并提高其对OODs的鲁棒性。在确定了双层优化中低阶近似的局限性之后，我们开发了一种有效的双层优化算法，该算法可以适应目标的高阶近似，并且可以扩展到大量的内部优化步骤以学习大量的权重参数。此外，我们进行了理论分析的影响，遥远的OODs在BN步骤，并提出了一个加权批量归一化（WBN）的过程中，使用的权重估计的双层优化问题在BN步骤。此外，我们讨论了我们的方法和低阶近似技术之间的连接。我们对合成和真实世界数据集的广泛实验表明，我们提出的方法显着提高了四个代表性的SSL算法对OODs的鲁棒性相比，四个国家的最先进的强大的SSL策略。
Recent semi-supervised learning algorithms have demonstrated greater success with higher overall performance due to the use of better-unlabeled data representations. Nonetheless, recent research suggests that the performance of the SSL algorithm can be degraded when the unlabeled set contains out-of-distribution examples (OODs). This work addresses the following research question: How do out-of-distribution (OOD) data adversely affect semi-supervised learning algorithms? To answer this question, we investigate the critical causes of OOD’s negative effect on SSL algorithms. In particular, we found that 1) certain kinds of OOD data instances that are close to the decision boundary have a more significant impact on performance than those that are further away, and 2) Batch Normalization (BN), a popular module, may degrade rather than improve performance when the unlabeled set contains OODs. To address these challenges, we developed a unified weighted robust SSL framework that can be easily extended to many existing SSL algorithms and improve their robustness against OODs. Having identified the limitations of low-order approximations in bi-level optimization, we developed an efficient bi-level optimization algorithm that could accommodate high-order approximations of the objective and could scale to a large number of inner optimization steps to learn a massive number of weight parameters. Furthermore, we conduct a theoretical analysis of the impact of faraway OODs in the BN step and propose a weighted batch normalization (WBN) procedure that uses the weights estimated by the bi-level optimization problem in the BN step. Additionally, we discuss the connection between our approach and low-order approximation techniques. Our extensive experiments on synthetic and real-world datasets demonstrate that our proposed approach significantly enhances the robustness of four representative SSL algorithms against OODs compared to four state-of-the-art robust SSL strategies.