Performance guarantees for regularized maximum entropy density estimation
Performance guarantees for regularized maximum entropy density estimation
复制标题
DOI：
10.1007/978-3-540-27819-1_33
复制
发表时间：
2004-01-01
期刊：
影响因子：
--
通讯作者：
Schapire, RE
中科院分区：
文献类型：
--
作者：
Dudík, M;Phillips, SJ;Schapire, RE
我们考虑的问题，估计一个未知的概率分布的样本使用最大熵（maxent）的原则。为了减轻过度拟合与大量的功能，我们建议应用maxent原则与放松约束的期望的功能。通过凸对偶，这相当于找到吉布斯分布最小化经验对数损失的正则化版本。我们证明非渐近界显示，相对于真正的底层分布，这种放松版本的maxent产生密度估计，几乎是尽可能好。这些界限是根据特征经验平均值相对于其真实期望的偏差，可以使用标准一致收敛技术来限制该数字。特别是，这会导致边界随着样本的数量迅速下降，并且非常适度地依赖于特征的数量或复杂性。我们还推导和证明收敛的顺序更新和并行更新算法。最后，我们简要介绍了实验的数据相关的物种地理分布的建模。
We consider the problem of estimating an unknown probability distribution from samples using the principle of maximum entropy (maxent). To alleviate overfitting with a very large number of features, we propose applying the maxent principle with relaxed constraints on the expectations of the features. By convex duality, this turns out to be equivalent to finding the Gibbs distribution minimizing a regularized version of the empirical log loss. We prove non-asymptotic bounds showing that, with respect to the true underlying distribution, this relaxed version of maxent produces density estimates that are almost as good as the best possible. These bounds are in terms of the deviation of the feature empirical averages relative to their true expectations, a number that can be bounded using standard uniform-convergence techniques. In particular, this leads to bounds that drop quickly with the number of samples, and that depend very moderately on the number or complexity of the features. We also derive and prove convergence for both sequential-update and parallel-update algorithms. Finally, we briefly describe experiments on data relevant to the modeling of species geographical distributions.