Hiding Data Helps: On the Benefits of Masking for Sparse Coding
Hiding Data Helps: On the Benefits of Masking for Sparse Coding
复制标题
DOI：
10.48550/arxiv.2302.12715
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
Muthuraman Chidambaram;Chenwei Wu;Yu Cheng;Rong Ge
中科院分区：
文献类型：
--
作者：
Muthuraman Chidambaram;Chenwei Wu;Yu Cheng;Rong Ge
稀疏编码是指将信号建模为学习字典的元素的稀疏线性组合，已被证明是信号处理，计算机视觉和医学成像等应用中的成功（和可解释）方法。虽然这一成功促使人们在学习词典与地面实况词典大小相同时对词典恢复的可证明保证进行了大量工作，但在学习词典相对于地面实况更大（或过度实现）的设置上的工作相对较新。在这种情况下，现有的理论结果已被限制在无噪声数据的情况下。我们在这项工作中表明，在存在噪声的情况下，最小化标准字典学习目标可能无法在过度实现的机制中恢复地面实况字典的元素，无论数据生成过程中信号的大小如何。此外，从越来越多的自我监督学习的工作中，我们提出了一个新的掩蔽目标，对于一个大类的数据生成过程，随着信号的增加，恢复地面实况字典实际上是最佳的。我们证实了我们的理论结果与实验在几个参数制度表明，我们提出的目标也享有更好的经验性能比标准的重建目标。
Sparse coding, which refers to modeling a signal as sparse linear combinations of the elements of a learned dictionary, has proven to be a successful (and interpretable) approach in applications such as signal processing, computer vision, and medical imaging. While this success has spurred much work on provable guarantees for dictionary recovery when the learned dictionary is the same size as the ground-truth dictionary, work on the setting where the learned dictionary is larger (or over-realized) with respect to the ground truth is comparatively nascent. Existing theoretical results in this setting have been constrained to the case of noise-less data. We show in this work that, in the presence of noise, minimizing the standard dictionary learning objective can fail to recover the elements of the ground-truth dictionary in the over-realized regime, regardless of the magnitude of the signal in the data-generating process. Furthermore, drawing from the growing body of work on self-supervised learning, we propose a novel masking objective for which recovering the ground-truth dictionary is in fact optimal as the signal increases for a large class of data-generating processes. We corroborate our theoretical results with experiments across several parameter regimes showing that our proposed objective also enjoys better empirical performance than the standard reconstruction objective.