Estimation of entropy and mutual information
Estimation of entropy and mutual information
复制标题
DOI：
10.1162/089976603321780272
复制
发表时间：
2003-06-01
影响因子：
2.9
通讯作者：
Paninski, L
中科院分区：
文献类型：
--
作者：
Paninski, L
我们提供了一些有关熵和相互信息的非参数估计的新结果。首先，我们使用熵函数的精确局部扩展来证明三个最常用的离散信息估计器的一致性和中心限制定理。该设置与Grenander的Sieves方法有关，并且对生成数据的基本概率度量没有任何假设。其次，我们证明了与这些一致性定理的交谈，表明对最常见的估计技术的错误应用也会导致对真实信息的任意估计，即使给定无限数据也是如此。这种“不一致”定理导致偏差的分析近似，在出乎意料的小样本状态下有效，并且比Miller和Madow的通常的1/N公式在很大的参数空间区域上有效。这些结果的两个最实际含义是负面的：（1）某种数据制度中的信息估计可能会受到偏见的污染，即使使用了“偏见校正”的估计器，（2）通过标准技术计算出的置信区间大大低估了最常见的估计方法的误差。在本文中，我们注意到熵估计器的偏置与某些多项式近似问题之间的一个非常有用的联系。通过在这个近似理论框架中施放偏置计算问题，我们获得了已知的渐近偏置结果的最佳概括。更有趣的是，该框架导致具有一些不错的属性的估计器：估算器在所有可能的基本概率分布上配备了严格的界限，最大误差的最大误差界限，并且此最大误差事实非常小。我们演示了该新估计器在真实和模拟数据上的应用。
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This "inconsistency" theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual 1/N formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if "bias-corrected" estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods.Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.