k-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension
k-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension
复制标题
DOI：
10.48550/arxiv.2206.08526
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Ziv Goldfeld;K. Greenewald;Theshani Nuradha;Galen Reeves
中科院分区：
文献类型：
--
作者：
Ziv Goldfeld;K. Greenewald;Theshani Nuradha;Galen Reeves
切片互信息（SMI）定义为随机变量的一维随机投影之间互信息（MI）项的平均值。它作为对经典MI依赖的替代度量，保留了它的许多属性，但在高维上更具可伸缩性。然而，SMI本身及其估计率如何依赖于环境维度的定量表征（这对理解可伸缩性至关重要）仍然模糊不清。这项工作提供了SMI对维度依赖的多方面解释，在一个更广泛的框架下，称为$k$-SMI，它考虑到$k$维子空间的投影。利用2-Wasserstein度量中微分熵连续性的新结果，我们推导了基于蒙特卡罗（MC）估计的$k$-SMI的误差的尖锐界限，明确地依赖于$k$和环境维度，揭示了它们与样本数量的相互作用。然后，我们将MC积分器与神经估计框架相结合，提供了一个端到端的$k$-SMI估计器，并建立了最优收敛率。我们还探索了随着维数增长的总体k -SMI的渐近性，提供了高斯近似结果，其残差在适当的矩限下衰减。通过设置$k=1$，我们的所有结果都可以应用于SMI。我们的理论通过数值实验得到了验证，并应用于切片的InfoGAN，它们共同提供了对$k$-SMI的可伸缩性问题的全面定量说明，包括作为$k=1$时的特殊情况的SMI。
Sliced mutual information (SMI) is defined as an average of mutual information (MI) terms between one-dimensional random projections of the random variables. It serves as a surrogate measure of dependence to classic MI that preserves many of its properties but is more scalable to high dimensions. However, a quantitative characterization of how SMI itself and estimation rates thereof depend on the ambient dimension, which is crucial to the understanding of scalability, remain obscure. This work provides a multifaceted account of the dependence of SMI on dimension, under a broader framework termed $k$-SMI, which considers projections to $k$-dimensional subspaces. Using a new result on the continuity of differential entropy in the 2-Wasserstein metric, we derive sharp bounds on the error of Monte Carlo (MC)-based estimates of $k$-SMI, with explicit dependence on $k$ and the ambient dimension, revealing their interplay with the number of samples. We then combine the MC integrator with the neural estimation framework to provide an end-to-end $k$-SMI estimator, for which optimal convergence rates are established. We also explore asymptotics of the population $k$-SMI as dimension grows, providing Gaussian approximation results with a residual that decays under appropriate moment bounds. All our results trivially apply to SMI by setting $k=1$. Our theory is validated with numerical experiments and is applied to sliced InfoGAN, which altogether provide a comprehensive quantitative account of the scalability question of $k$-SMI, including SMI as a special case when $k=1$.