Fisher Efficient Inference of Intractable Models
Fisher Efficient Inference of Intractable Models
复制标题
DOI：
--
复制
发表时间：
2018-05
期刊：
影响因子：
--
通讯作者：
Song Liu;T. Kanamori;Wittawat Jitkrittum;Yu Chen
中科院分区：
文献类型：
--
作者：
Song Liu;T. Kanamori;Wittawat Jitkrittum;Yu Chen
最大似然估计(MLE)具有许多良好的性质。例如，最大似然估计解的渐近方差等于渐近Cram{e}r-Rao下界(有效界)，它是无偏估计的最小可能方差。然而，要获得这种最大似然估计解，需要计算似然函数，但由于密度模型的归一化项，似然函数可能不容易处理。本文从密度比估计和Stein算子实现的Kullback-Leibler发散最小化准则出发，导出了判别似然估计(DLE)。研究了基于DLE的模型推理问题。我们证明了它的相合性，并证明了在温和的正则性条件下，其解的渐近方差可以达到有效界的等价性。我们还提出了一种易于优化的DLE的对偶公式。数值研究验证了我们的渐近定理，并给出了一个例子，其中DLE成功地估计了一个使用预训练的深度神经网络构造的难解模型。
Maximum Likelihood Estimators (MLE) has many good properties. For example, the asymptotic variance of MLE solution attains equality of the asymptotic Cram{e}r-Rao lower bound (efficiency bound), which is the minimum possible variance for an unbiased estimator. However, obtaining such MLE solution requires calculating the likelihood function which may not be tractable due to the normalization term of the density model. In this paper, we derive a Discriminative Likelihood Estimator (DLE) from the Kullback-Leibler divergence minimization criterion implemented via density ratio estimation and a Stein operator. We study the problem of model inference using DLE. We prove its consistency and show that the asymptotic variance of its solution can attain the equality of the efficiency bound under mild regularity conditions. We also propose a dual formulation of DLE which can be easily optimized. Numerical studies validate our asymptotic theorems and we give an example where DLE successfully estimates an intractable model constructed using a pre-trained deep neural network.