Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
复制标题
DOI：
--
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
Kento Nozawa;Issei Sato
中科院分区：
文献类型：
--
作者：
Kento Nozawa;Issei Sato
实例判别自监督表示学习由于其无监督性质和对下游任务的信息特征表示而受到人们的关注。在实践中，它通常使用的负样本数量大于监督类的数量。然而，在现有的分析中存在着不一致的地方；从理论上讲，大量的负样本会降低下游监督任务的分类性能，而从经验上讲，它们会提高分类性能。我们提供了一个新的框架来分析这一实证结果关于负样本使用优惠券收集器的问题。通过增加负样本的数量，我们的边界可以隐式地将下游任务的监督损失合并到自监督损失中。我们确认我们提出的分析适用于真实世界的基准数据集。
Instance discriminative self-supervised representation learning has been attracted attention thanks to its unsupervised nature and informative feature representation for downstream tasks. In practice, it commonly uses a larger number of negative samples than the number of supervised classes. However, there is an inconsistency in the existing analysis; theoretically, a large number of negative samples degrade classification performance on a downstream supervised task, while empirically, they improve the performance. We provide a novel framework to analyze this empirical result regarding negative samples using the coupon collector's problem. Our bound can implicitly incorporate the supervised loss of the downstream task in the self-supervised loss by increasing the number of negative samples. We confirm that our proposed analysis holds on real-world benchmark datasets.