Nonlinear Information Bottleneck
Nonlinear Information Bottleneck
复制标题
DOI：
10.3390/e21121181
复制
发表时间：
2019-11-30
期刊：
影响因子：
2.7
通讯作者：
Wolpert DH
中科院分区：
文献类型：
--
作者：
Kolchinsky A;Tracey BD;Wolpert DH
信息瓶颈（IB）是一种从一个随机变量X中提取与预测另一个随机变量Y相关的信息的技术。IB通过将X编码在压缩的“瓶颈”随机变量M中来工作，Y可以从该压缩的“瓶颈”随机变量M中准确地解码。然而，找到最佳的瓶颈变量涉及到一个困难的优化问题，直到最近才被认为是只有两个有限的情况下：离散的X和Y与小的状态空间，和连续的X和Y与高斯联合分布（在这种情况下，最佳的编码和解码映射是线性的）。我们提出了一种方法，用于在任意分布的离散和/或连续X和Y上执行IB，同时允许非线性编码和解码映射。我们的方法依赖于一个新的非参数的互信息上限。我们描述了如何使用神经网络实现我们的方法。然后，我们表明，它实现了更好的性能比最近提出的“变分IB”的方法在几个真实世界的数据集。
Information bottleneck (IB) is a technique for extracting information in one random variable X that is relevant for predicting another random variable Y. IB works by encoding X in a compressed “bottleneck” random variable M from which Y can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete X and Y with small state spaces, and continuous X and Y with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous X and Y, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed “variational IB” method on several real-world datasets.
影响因子：
2.5
作者：
AHLSWEDE, RF;KORNER, J
通讯作者：
KORNER, J