Learning Decorrelated Representations Efficiently Using Fast Fourier Transform
Learning Decorrelated Representations Efficiently Using Fast Fourier Transform
复制标题
DOI：
10.1109/cvpr52729.2023.00204
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Yutaro Shigeto;M. Shimbo;Yuya Yoshikawa;A. Takeuchi
中科院分区：
文献类型：
--
作者：
Yutaro Shigeto;M. Shimbo;Yuya Yoshikawa;A. Takeuchi
Barlow、Twin和VICReg是自监督表示学习模型，它们使用正则化来解除特征的相关性。虽然这些模型与传统的表示学习模型一样有效，但如果投影嵌入的维度$d$很高，则它们的训练可能需要计算。由于正则化子是根据互相关或协方差矩阵的单个元素定义的，因此计算$n$样本的损失需要$O(nd^{2})$时间。本文提出了一种松弛去相关正则化方法，它可以在$O(nd\logd)$时间内通过快速傅立叶变换来计算。我们还提出了一种廉价的技术来减少随着松弛而产生的不希望看到的局部最小值。所提出的正则化方法在下游任务中表现出与现有正则化方法相当的精度，而它们的训练需要更少的内存，并且对于大$d$更快。源代码现已发布。11https://github.com/yutaro-s/scalable-decorrelation-ssl.git
Barlow Twins and VICReg are self-supervised representation learning models that use regularizers to decorrelate features. Although these models are as effective as conventional representation learning models, their training can be computationally demanding if the dimension $d$ of the projected embeddings is high. As the regularizers are defined in terms of individual elements of a cross-correlation or covariance matrix, computing the loss for $n$ samples takes $O(nd^{2})$ time. In this paper, we propose a relaxed decorre-lating regularizer that can be computed in $O(nd\log d)$ time by Fast Fourier Transform. We also propose an inexpensive technique to mitigate undesirable local minima that develop with the relaxation. The proposed regularizer exhibits accuracy comparable to that of existing regularizers in down-stream tasks, whereas their training requires less memory and is faster for large $d$. The source code is available. 11https://github.com/yutaro-s/scalable-decorrelation-ssl.git