Estimation of Non-Normalized Statistical Models by Score Matching
Estimation of Non-Normalized Statistical Models by Score Matching
复制标题
DOI：
--
复制
发表时间：
2005-12
期刊：
影响因子：
--
通讯作者：
Aapo Hyvärinen
中科院分区：
文献类型：
--
作者：
Aapo Hyvärinen
人们经常想要估计统计模型，其中概率密度函数直到乘性归一化常数才是已知的。通常，人们不得不求助于马尔科夫链蒙特卡罗方法，或归一化常数的近似。在这里，我们提出这样的模型可以通过最小化模型给出的对数密度梯度与观测数据的对数密度梯度之间的期望平方距离来估计。虽然对数密度函数的梯度估计原则上是一个非常困难的非参数问题，但我们证明了一个令人惊讶的结果，它给出了这个目标函数的一个简单公式。该公式中没有出现观测数据的密度函数，简化为模型给出的对数密度的一些导数之和的样本平均值。在多变量高斯模型和独立分量分析模型上，并通过估计自然图像数据的过完备滤波器集，验证了该方法的有效性。
One often wants to estimate statistical models where the probability density function is known only up to a multiplicative normalization constant. Typically, one then has to resort to Markov Chain Monte Carlo methods, or approximations of the normalization constant. Here, we propose that such models can be estimated by minimizing the expected squared distance between the gradient of the log-density given by the model and the gradient of the log-density of the observed data. While the estimation of the gradient of log-density function is, in principle, a very difficult non-parametric problem, we prove a surprising result that gives a simple formula for this objective function. The density function of the observed data does not appear in this formula, which simplifies to a sample average of a sum of some derivatives of the log-density given by the model. The validity of the method is demonstrated on multivariate Gaussian and independent component analysis models, and by estimating an overcomplete filter set for natural image data.