Learning Overcomplete, Low Coherence Dictionaries with Linear Inference
Learning Overcomplete, Low Coherence Dictionaries with Linear Inference
复制标题
DOI：
--
复制
发表时间：
2016-06
期刊：
影响因子：
--
通讯作者：
J. Livezey;Alejandro F. Bujan;F. Sommer
中科院分区：
文献类型：
--
作者：
J. Livezey;Alejandro F. Bujan;F. Sommer
寻找数据的过完备潜在表示在数据分析、信号处理、机器学习、理论神经科学和许多其他领域都有应用。在过完备表示中，潜在特征的数量超过数据维度，这在数据被测量欠采样（压缩感知，神经系统中的信息瓶颈）或由多个完整的线性特征集组成时非常有用，每个线性特征集都跨越数据空间。独立成分分析（伊卡）是一种用于学习稀疏潜在表示的线性技术，其计算成本通常低于稀疏编码，其非线性，递归对应物。虽然非常适合寻找完整的表示，我们表明，overcompleteness构成了挑战，现有的伊卡算法。具体而言，现有伊卡算法中的相干控制，必要的，以防止形成重复的字典功能，是不适合在过完备的情况下。我们表明，在这种情况下，现有的伊卡算法有不理想的全球最小值，最大限度地提高一致性。此外，通过比较伊卡算法对合成数据和自然图像的计算更昂贵的稀疏编码解决方案，我们表明，一致性控制偏置的数据流形的探索，有时产生次优的解决方案。我们提供了一个理论解释，这些故障，并在理论的基础上，提出了改进的过完备伊卡算法。总而言之，这项研究为线性伊卡的相干控制提供了新的见解和方法，其中一些适用于许多其他潜在的非线性无监督学习方法。
Finding overcomplete latent representations of data has applications in data analysis, signal processing, machine learning, theoretical neuroscience and many other fields. In an overcomplete representation, the number of latent features exceeds the data dimensionality, which is useful when the data is undersampled by the measurements (compressed sensing, information bottlenecks in neural systems) or composed from multiple complete sets of linear features, each spanning the data space. Independent Components Analysis (ICA) is a linear technique for learning sparse latent representations, which typically has a lower computational cost than sparse coding, its nonlinear, recurrent counterpart. While well suited for finding complete representations, we show that overcompleteness poses a challenge to existing ICA algorithms. Specifically, the coherence control in existing ICA algorithms, necessary to prevent the formation of duplicate dictionary features, is ill-suited in the overcomplete case. We show that in this case several existing ICA algorithms have undesirable global minima that maximize coherence. Further, by comparing ICA algorithms on synthetic data and natural images to the computationally more expensive sparse coding solution, we show that the coherence control biases the exploration of the data manifold, sometimes yielding suboptimal solutions. We provide a theoretical explanation of these failures and, based on the theory, propose improved overcomplete ICA algorithms. All told, this study contributes new insights into and methods for coherence control for linear ICA, some of which are applicable to many other, potentially nonlinear, unsupervised learning methods.