Estimating mutual information -: art. no. 066138
Estimating mutual information -: art. no. 066138
复制标题
DOI：
10.1103/physreve.69.066138
复制
发表时间：
2004-06-01
影响因子：
2.4
通讯作者：
Grassberger, P
中科院分区：
文献类型：
--
作者：
Kraskov, A;Stögbauer, H;Grassberger, P
我们提出了两类改进的互信息M（X，Y）的估计，根据一些联合概率密度μ（x，y）分布的随机点的样本。与传统的基于合并的估计相比，它们基于k-最近邻距离的熵估计。这意味着它们是数据高效的（k=1时，我们将结构分解到最小的可能尺度），自适应的（数据越多，分辨率越高），并且具有最小的偏差。事实上，基本熵估计的偏差主要是由于在最小分辨率尺度上密度的不均匀性，通常给出作为N个点的k/N的函数的系统误差。在数值上，我们发现这两个家族对于独立分布都是精确的，即如果mu（x，y）=mu（x）mu（y），则cap（X，Y）上的估计量（M）为零（直到统计波动）。这适用于所有测试的边缘分布和x和y的所有维度。此外，我们还给出了两个以上的随机变量之间的冗余估计。我们比较我们的算法与现有的算法详细。最后，我们证明了我们的估计的有用性，用于评估从独立分量分析（伊卡）获得的组件的实际独立性，用于改进伊卡，以及用于估计盲源分离的可靠性。
We present two classes of improved estimators for mutual information M(X,Y), from samples of random points distributed according to some joint probability density mu(x,y). In contrast to conventional estimators based on binnings, they are based on entropy estimates from k-nearest neighbor distances. This means that they are data efficient (with k=1 we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to nonuniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of k/N for N points. Numerically, we find that both families become exact for independent distributions, i.e. the estimator (M) over cap (X,Y) vanishes (up to statistical fluctuations) if mu(x,y)=mu(x)mu(y). This holds for all tested marginal distributions and for all dimensions of x and y. In addition, we give estimators for redundancies between more than two random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.