A Simple Stochastic Gradient Variational Bayes for Latent Dirichlet Allocation
A Simple Stochastic Gradient Variational Bayes for Latent Dirichlet Allocation
复制标题
潜在狄利克雷分配的简单随机梯度变分贝叶斯
DOI：
10.1007/978-3-319-42089-9_17
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
Atsuhiro Takasu
中科院分区：
文献类型：
--
作者：
Tomonari Masada;Atsuhiro Takasu
本文对潜在狄利克雷分配(LDA)[4]提出了一种新的推论。我们的建议是随机梯度变分贝叶斯(SGVB)的一个实例[9，13]。SGVB是为贝叶斯概率模型设计后验推断的通用框架。我们的目的是通过给出一个针对文本挖掘中最著名的贝叶斯模型LDA的SGVB类型推理的例子来说明SGVB的有效性。本文提出的推理方法易于从头开始实现。该推论的一个特点是用Logistic正态分布来近似真实的后验分布。这是违反直觉的，因为在应用平均场近似后，当我们降低LDA的对数证据的界限时，我们通过取泛函导数来获得狄里克莱特分布。然而，我们的实验表明，在测试集困惑方面，所提出的推理比使用Dirichlet分布进行后验逼近的推理具有更好的预测性能。虽然Logistic正态分布比Dirichlet模型更复杂，但SGVB使后验期望的处理相对容易。对于我们实验中参考的许多设置，建议的推论甚至比折叠的Gibbs抽样[6]更好。如何在SGVB的基础上为其他贝叶斯模型设计一种新的推论，这是值得进一步研究的工作。
This paper proposes a new inference for the latent Dirichlet allocation (LDA) [4]. Our proposal is an instance of the stochastic gradient variational Bayes (SGVB) [9, 13]. SGVB is a general framework for devising posterior inferences for Bayesian probabilistic models. Our aim is to show the effectiveness of SGVB by presenting an example of SGVB-type inference for LDA, the best-known Bayesian model in text mining. The inference proposed in this paper is easy to implement from scratch. A special feature of the proposed inference is that the logistic normal distribution is used to approximate the true posterior. This is counterintuitive, because we obtain the Dirichlet distribution by taking the functional derivative when we lower bound the log evidence of LDA after applying a mean field approximation. However, our experiment showed that the proposed inference gave a better predictive performance in terms of test set perplexity than the inference using the Dirichlet distribution for posterior approximation. While the logistic normal is more complicated than the Dirichlet, SGVB makes the manipulation of the expectations with respect to the posterior relatively easy. The proposed inference was better even than the collapsed Gibbs sampling [6] for not all but many settings consulted in our experiment. It must be worthwhile future work to devise a new inference based on SGVB also for other Bayesian models.