Estimating Mixture Entropy with Pairwise Distances
Estimating Mixture Entropy with Pairwise Distances
复制标题
DOI：
10.3390/e19070361
复制
发表时间：
2017-07-01
期刊：
影响因子：
2.7
通讯作者：
Tracey, Brendan D.
中科院分区：
文献类型：
--
作者：
Kolchinsky, Artemy;Tracey, Brendan D.
混合分布出现在许多参数和非参数设置中，例如，在高斯混合模型和非参数估计中。通常需要计算混合物的熵，但是，在大多数情况下，这个量没有封闭形式的表达式，因此需要某种形式的近似。我们提出了一组基于混合成分间的成对距离函数的估计量，并证明了这类估计量具有许多吸引人的性质。对于许多感兴趣的分布，所提出的估计量计算效率高，在混合参数中可微，并且在混合成分聚类时变得精确。我们证明了这个族包括混合熵的下界和上界。当选择Chernoff散度作为距离函数时，给出了一个下界，而Bhattacharyaa距离为对称分量和位置族成员提供了最紧的下界。当用作距离函数时，Kullback-Leibler散度给出了一个上界。我们给出了这些高斯混合界的封闭表达式，并讨论了它们在互信息估计中的应用。然后，我们用数值模拟证明了我们的边界比已知的现有边界要严格得多。这类估计器在涉及熵和互信息的最大化/最小化的优化问题中非常有用，例如MaxEnt和速率失真问题。
Mixture distributions arise in many parametric and non-parametric settings-for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in most cases, this quantity has no closed-form expression, making some form of approximation necessary. We propose a family of estimators based on a pairwise distance function between mixture components, and show that this estimator class has many attractive properties. For many distributions of interest, the proposed estimators are efficient to compute, differentiable in the mixture parameters, and become exact when the mixture components are clustered. We prove this family includes lower and upper bounds on the mixture entropy. The Chernoff alpha-divergence gives a lower bound when chosen as the distance function, with the Bhattacharyaa distance providing the tightest lower bound for components that are symmetric and members of a location family. The Kullback-Leibler divergence gives an upper bound when used as the distance function. We provide closed-form expressions of these bounds for mixtures of Gaussians, and discuss their applications to the estimation of mutual information. We then demonstrate that our bounds are significantly tighter than well-known existing bounds using numeric simulations. This estimator class is very useful in optimization problems involving maximization/minimization of entropy and mutual information, such as MaxEnt and rate distortion problems.