Estimation of KL divergence between large-alphabet distributions
Estimation of KL divergence between large-alphabet distributions
复制标题
大字母分布之间 KL 散度的估计
DOI：
10.1109/isit.2016.7541473
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
V. Veeravalli
中科院分区：
文献类型：
--
作者：
Yuheng Bu;Shaofeng Zou;Yingbin Liang;V. Veeravalli
研究了两个未知分布之间KL散度的估计问题。分布的字母大小k可以扩展到无穷大。估计是基于m和n个独立的样本分别从两个分布。首先证明了在所有分布对的集合上，不存在任何一致的估计量来保证渐近小的最坏情况二次风险。进一步考虑了包含具有有界比f（k）的分布对的限制集。提出了一种增广插入估计，证明了它是相容的当且仅当m = ω（k <$log2（f（k）），n = ω（kf（k））.进一步证明了当f（k）≥ log 2k且log 2（f（k））= o（k）时，相容估计必须满足必要条件：m = ω（k/logk <$log2（f（k）），n = ω（kf（k）/logk）.
The problem of estimating the KL divergence between two unknown distributions is studied. The alphabet size k of the distributions can scale to infinity. The estimation is based on m and n independent samples respectively drawn from the two distributions. It is first shown that there does not exist any consistent estimator to guarantee asymptotic small worst-case quadratic risk over the set of all pairs of distributions. A restricted set that contains pairs of distributions with bounded ratio f(k) is further considered. An augmented plug-in estimator is proposed, and is shown to be consistent if and only if m = ω(k ⋁ log2(f(k)) and n = ω(k f(k)). Furthermore, if f(k) ≥ log2k and log2(f(k)) = o(k), it is shown that any consistent estimator must satisfy the necessary conditions: m = ω( k/log k ⋁ log2(f(k)) and n = ω( k f(k)/log k).