Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
复制标题
DOI：
10.5555/2503308.2188396
复制
发表时间：
2012
期刊：
影响因子：
--
通讯作者：
Michael U Gutmann;Aapo Hyvärinen
中科院分区：
文献类型：
--
作者：
Michael U Gutmann;Aapo Hyvärinen
我们考虑根据观测数据估计由有限数量的参数参数化的概率模型的任务。特别是，我们正在考虑模型概率密度函数未归一化的情况。也就是说，模型仅指定到配分函数。配分函数对模型进行归一化，以便对于任何参数选择都可以积分为一个模型。然而，通常不可能以封闭形式获得它。吉布斯分布、马尔可夫和多层网络是分析归一化通常不可能的模型示例。如果不求助于通常计算成本高昂的数值近似，则不能使用最大似然估计。我们在这里提出了一个新的目标函数来估计标准化和非标准化模型。基本思想是执行非线性逻辑回归来区分观察到的数据和一些人为生成的噪声。通过这种方法，可以像任何其他参数一样估计归一化配分函数。我们证明新的估计方法可以产生一致（收敛）的参数估计器。对于大噪声样本量，新的估计器的行为还类似于最大似然估计器。在非标准化模型的估计中，统计性能和计算性能之间存在权衡。我们表明，与非标准化模型的其他估计方法相比，新方法实现了竞争性权衡。作为实际数据的应用，我们估计了具有样条非线性的自然图像统计的新颖两层模型。
We consider the task of estimating, from observed data, a probabilistic model that is parameterized by a finite number of parameters. In particular, we are considering the situation where the model probability density function is unnormalized. That is, the model is only specified up to the partition function. The partition function normalizes a model so that it integrates to one for any choice of the parameters. However, it is often impossible to obtain it in closed form. Gibbs distributions, Markov and multi-layer networks are examples of models where analytical normalization is often impossible. Maximum likelihood estimation can then not be used without resorting to numerical approximations which are often computationally expensive. We propose here a new objective function for the estimation of both normalized and unnormalized models. The basic idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise. With this approach, the normalizing partition function can be estimated like any other parameter. We prove that the new estimation method leads to a consistent (convergent) estimator of the parameters. For large noise sample sizes, the new estimator is furthermore shown to behave like the maximum likelihood estimator. In the estimation of unnormalized models, there is a trade-off between statistical and computational performance. We show that the new method strikes a competitive trade-off in comparison to other estimation methods for unnormalized models. As an application to real data, we estimate novel two-layer models of natural image statistics with spline nonlinearities.