On Finite-Sample Identifiability of Contrastive Learning-Based Nonlinear Independent Component Analysis
On Finite-Sample Identifiability of Contrastive Learning-Based Nonlinear Independent Component Analysis
复制标题
DOI：
10.48550/arxiv.2206.06593
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Qi Lyu;Xiao Fu
中科院分区：
文献类型：
--
作者：
Qi Lyu;Xiao Fu
非线性独立分量分析（nICA）的目的是恢复由未知非线性函数混合的统计独立的潜在分量。nICA的核心是潜伏成分的可识别性，这一直难以捉摸，直到最近。具体来说，Hyv\ arinen等人已经表明，在广义对比学习（GCL）公式下，非线性混合潜在成分是可识别的（通常是无关两可的），因为潜在成分是独立于某个辅助变量的。基于gcl的nICA可识别性是优雅的，并且在nICA和流行的无监督/自监督学习范式之间建立了有趣的联系，这些范式包括表征学习、因果学习和因素解纠缠。然而，现有的nICA可识别性分析都是建立在无限样本假设和使用理想的通用函数学习器的基础上的，这在理论和实践之间造成了不可忽略的差距。缩小这一差距是一项非同小可的挑战，因为缺乏针对此类无监督问题的有限样本分析的既定“教科书”程序。本文提出了基于gcl的nICA的有限样本可识别性分析。我们的分析框架明智地结合了GCL损失函数、统计泛化分析和数值微分的性质。我们的框架还考虑了学习函数的近似误差，并揭示了所使用的函数学习器的复杂性和表达性之间的直观权衡。用数值实验验证了这些定理。
Nonlinear independent component analysis (nICA) aims at recovering statistically independent latent components that are mixed by unknown nonlinear functions. Central to nICA is the identifiability of the latent components, which had been elusive until very recently. Specifically, Hyv\"arinen et al. have shown that the nonlinearly mixed latent components are identifiable (up to often inconsequential ambiguities) under a generalized contrastive learning (GCL) formulation, given that the latent components are independent conditioned on a certain auxiliary variable. The GCL-based identifiability of nICA is elegant, and establishes interesting connections between nICA and popular unsupervised/self-supervised learning paradigms in representation learning, causal learning, and factor disentanglement. However, existing identifiability analyses of nICA all build upon an unlimited sample assumption and the use of ideal universal function learners -- which creates a non-negligible gap between theory and practice. Closing the gap is a nontrivial challenge, as there is a lack of established ``textbook'' routine for finite sample analysis of such unsupervised problems. This work puts forth a finite-sample identifiability analysis of GCL-based nICA. Our analytical framework judiciously combines the properties of the GCL loss function, statistical generalization analysis, and numerical differentiation. Our framework also takes the learning function's approximation error into consideration, and reveals an intuitive trade-off between the complexity and expressiveness of the employed function learner. Numerical experiments are used to validate the theorems.