Understanding self-supervised Learning Dynamics without Contrastive Pairs
Understanding self-supervised Learning Dynamics without Contrastive Pairs
复制标题
DOI：
--
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
Yuandong Tian;Xinlei Chen;S. Ganguli
中科院分区：
文献类型：
--
作者：
Yuandong Tian;Xinlei Chen;S. Ganguli
虽然自监督学习（SSL）的对比方法通过最小化同一数据点的两个增强视图（正对）之间的距离以及最大化来自不同数据点的视图（负对）来学习表征，但最近的\emph{非对比} SSL（例如，BYOL和SimSiam）在\emph{没有}负对的情况下表现出显著的性能，借助一个额外的可学习预测器和一个停止梯度操作。一个基本问题出现了：为什么这些方法不会坍缩为平凡的表征呢？我们通过一个简单的理论研究回答了这个问题，并提出了一种新的方法，DirectPred，它\emph{直接}根据其输入的统计信息设置线性预测器，无需梯度训练。在ImageNet上，它与采用BatchNorm的更复杂的两层非线性预测器性能相当，并且在300个epoch的训练中比线性预测器性能高出2.5\%（在60个epoch中高出5\%）。DirectPred是由我们对简单线性网络中非对比SSL的非线性学习动态的理论研究所激发的。我们的研究在概念上深入了解了非对比SSL方法如何学习、它们如何避免表征坍缩，以及多个因素（如预测器网络、停止梯度、指数移动平均和权重衰减）是如何起作用的。我们的简单理论概括了在STL - 10和ImageNet中实际消融研究的结果。代码发布在https://github.com/facebookresearch/luckmatters/tree/master/ssl。
While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs), recent \emph{non-contrastive} SSL (e.g., BYOL and SimSiam) show remarkable performance {\it without} negative pairs, with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and propose a novel approach, DirectPred, that \emph{directly} sets the linear predictor based on the statistics of its inputs, without gradient training. On ImageNet, it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm and outperforms a linear predictor by $2.5\%$ in 300-epoch training (and $5\%$ in 60-epoch). DirectPred is motivated by our theoretical study of the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our study yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Code is released https://github.com/facebookresearch/luckmatters/tree/master/ssl.